[github] requests.exceptions.HTTPError: 502 Server Error: Bad Gateway

[acs]

Trying to get the github data from Apache Spark project I have found the same exception in two different executions. The command used was:

(venv) acastillo@acastillo:~$ perceval -v
perceval 0.12.3
(venv) acastillo@acastillo:~$ p2o.py -g  --index spark_github github --sleep-for-rate -t 33fcb030b3007cb9f1735b7cc6f4887021XXXXXX apache spark

The first one:

2019-01-14 17:26:42,127 Error feeding ocean from github (https://github.com/apache/spark): 502 Server Error: Bad Gateway for url: https://api.github.com/repositories/17165658/issues?state=all&per_page=100&direction=asc&sort=updated&requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repositories/17165658/issues?state=all&per_page=100&direction=asc&sort=updated&since=1970-01-01T00%3A00%3A00%2B00%3A00&page=5&state=all&per_page=100&direction=asc&sort=updated&since=1970-01-01T00%3A00%3A00%2B00%3A00since=1970-01-01T00%3A00%3A00%2B00%3A00&page=7&state=all&per_page=100&direction=asc&sort=updated&since=1970-01-01T00%3A00%3A00%2B00%3A00

And the second one:

requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repositories/17165658/issues?state=all&per_page=100&direction=asc&sort=updated&since=1970-01-01T00%3A00%3A00%2B00%3A00&page=5&state=all&per_page=100&direction=asc&sort=updated&since=1970-01-01T00%3A00%3A00%2B00%3A00

Both of them are getting the list issues. It is an error internal to GitHub (https://tools.ietf.org/html/rfc7231#section-6.6.3) so probably the only solution is to retry the request until it works or until it is clear it won't work (3 requests failed for example).

I have retried exactly the same queries from the web browser and both of them work. So it is an intermittent issue.

[acs]

I have retried it this morning with a good fiber connection and no issue until the end of the rate limit (5000 requests). So it seems that the Bad Gateway error could appear when the GItHub servers are overloaded. My yesterday testing was done using a mobile connection.

[acs]

In this morning, just 2 hours later, and using a good connection also, the same error appeared.

[valeriocos]

Hi @acs thank you for reporting this bug. I'm currently performing some tests but it seems that the error is gone if you replace the code at: https://github.com/chaoss/grimoirelab-perceval/blob/master/perceval/backends/core/github.py#L485 with

super().__init__(base_url, sleep_time=sleep_time, max_retries=max_retries,
    extra_headers=self._set_extra_headers(), extra_status_forcelist=[403, 502, 503],
    archive=archive, from_archive=from_archive)

[acs]

Great news! I will try to check it asap. Thanks @valeriocos Would this code be included in the next release of perceval?

[sduenas]

I'm going to close this since seems to be a problem with GitHub. I think we cannot do more at this moment.