TaskCluster sometimes fails due to network issues

[stephenmcgruer]

There is an ongoing Mozilla infrastructure issue that is causing network failures in TaskCluster runs when attempting to fetch Firefox testing profiles:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='hg.mozilla.org', port=443): Max retries exceeded with url: /mozilla-central/archive/tip.zip/testing/profiles/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5d88c94f50>: Failed to establish a new connection: [Errno 110] Connection timed out',))

This is just a tracking issue to link to from blocked PRs; I will attempt to post updates here, but see https://status.mozilla.org/ for the latest information on the outage.

[stephenmcgruer]

https://status.mozilla.org/ reports that everything is green now. If your PR failed due to this issue, you should be able to close and re-open the PR to trigger the checks to re-run.

[Hexcles]

I also saw some 5xx errors from this URL earlier this week. This has been happening relatively often.

I think the testing profile needs to be treated in the same way as #21194 (which is also more technically correct), with retry mechanism. @jgraham

[Hexcles]

I think we have a more general problem regarding network. Anecdotally, I'm seeing more failures due to network errors recently (though I don't have any real stats). I think it makes sense to implement a generic retry mechanism for all the downloads.