Perceval runs in error - GitHub rate limit and exiting

[TheReal1604]

Calling Perceval via p2o.py:

python3 p2o.py --debug --index github-data --github-token <my-token> github grimoirelab perceval

perceval doesn't wait for the rate limit and exit with the following error:

Error feeding ocean from github (https://github.com/grimoirelab/perceval): GitHub rate limit exhausted.; 356 seconds to rate reset
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/arthur.py", line 196, in feed_backend
    ocean_backend.feed()
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/ocean/elastic.py", line 201, in feed
    self.feed_items(items)
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/ocean/elastic.py", line 211, in feed_items
    for item in items:
  File "/home/user/arthur/src/perceval/perceval/backend.py", line 360, in decorator
    for data in func(self, *args, **kwargs):
  File "/home/user/arthur/src/perceval/perceval/backends/core/github.py", line 168, in fetch
    issue[field + '_data'] = self.__get_user(issue[field]['login'])
  File "/home/user/arthur/src/perceval/perceval/backends/core/github.py", line 338, in __get_user
    user_raw = self.client.user(login)
  File "/home/user/arthur/src/perceval/perceval/backends/core/github.py", line 513, in user
    r = self.__send_request(url_user, headers=self.__build_headers())
  File "/home/user/arthur/src/perceval/perceval/backends/core/github.py", line 593, in __send_request
    raise RateLimitError(cause=cause, seconds_to_reset=seconds_to_reset)
perceval.errors.RateLimitError: GitHub rate limit exhausted.; 356 seconds to rate reset

Any suggestions? :)

[valeriocos]

Hi @TheReal1604, you should add the parameter --sleep-for-rate (which allows the fetch process to sleep until the rate limit is reset) in the GitHub backend.

Let me know if this solves your issue :)

[TheReal1604]

hi @valeriocos
seems that this options is not supported anymore? Used the actual version of p2o.py from GrimoireELK master repository.

p2o.py: error: unrecognized arguments: --sleep-for-rate

[valeriocos]

--sleep-for-rate should go together with the backend params, otherwise you will get the error just above

try this:

--debug --index github-data github grimoirelab perceval --api-token <my-token> --sleep-for-rate

[TheReal1604]

@valeriocos this seems to be working now. Thanks!

But got another issue, seems that the bulk indexing is broken? :/

406 Client Error: Not Acceptable for url: http://127.0.0.1:9200/github-analysis/items/_bulk
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/arthur.py", line 196, in feed_backend
    ocean_backend.feed()
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/ocean/elastic.py", line 201, in feed
    self.feed_items(items)
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/ocean/elastic.py", line 219, in feed_items
    self._items_to_es(items_pack)
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/ocean/elastic.py", line 247, in _items_to_es
    self.elastic.bulk_upload_sync(json_items, field_id)
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/elk/elastic.py", line 195, in bulk_upload_sync
    total += self.bulk_upload(items_pack, field_id)
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/elk/elastic.py", line 137, in bulk_upload
    self._safe_put_bulk(url, bulk_json)
  File "/home/user/.local/lib/python3.5/site-packages/grimoire_elk/elk/elastic.py", line 101, in _safe_put_bulk
    res.raise_for_status()
  File "/home/user/.local/lib/python3.5/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 406 Client Error: Not Acceptable for url: http://127.0.0.1:9200/github-analysis/items/_bulk

I use the p2o.py script against an ES 6 instance.

[valeriocos]

You are welcome!
Try to use elasticsearch 5.x (https://www.elastic.co/downloads/past-releases). I tried with the version 5.6.3 and the bug didn't appear (but it did with 6.1.1).

[valeriocos]

@TheReal1604 I would close this issue if you didn't experience any problem when using Elasticsearch 5.x