[MRG+1] Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError

[starrify]

It's a small change. Please just check the diff.

[redapple]

hi @starrify , what kind of error triggered TunnelError in your tests?
did you see retrying them succeed eventually?

[starrify]

Hi @redapple , here is the error message:

[scrapy.core.scraper] Error downloading <GET XXXX>: Could not open CONNECT tunnel.

For details, please check https://staging.scrapinghub.com/p/6327/job/9/17618/#/log?filterAndHigher&filterType=error (private access only). From the job stats we know there's no retrying on such failures.

Here's another sample job after applying the same fix in this PR: https://staging.scrapinghub.com/p/6327/job/9/17622/#details (private access only) where the same error has almost disapeared. Also the job stats suggest that there are many retries.

[redapple]

@starrify , could this be configured via a new downloader setting?

[starrify]

I'm afraid not, since this EXCEPTIONS_TO_RETRY thing is initialized here in this file and never updated later.

However I think it might be a good idea to make it configurable through settings. :)

[redapple]

@starrify , I'm not sure I understand your comment.
one could still update an instance attribute with exceptions to retry, defaulting to EXCEPTIONS_TO_RETRY, adding TunnelError to the tuple if some new RETRY_TUNNEL_ERROR is True or something.
maybe I'm missing something?

[codecov-io]

Current coverage is 83.27%

Merging #1974 into master will increase coverage by <.01%

Powered by Codecov. Last updated by ed4b9af...81ed956

[starrify]

Hi, seems I'm the one who misunderstood. :( I thought you asked could this be configured now via a downloader setting?

I have updated this PR by introducing a new setting RETRY_EXCEPTIONS_EXTRA and removed TunnelError from the default ones.

But personally I still think TunnelError shall be retried by default like other errors.

[redapple]

@starrify , after some thought, I think your initial commit (adding TunnelError to EXCEPTIONS_TO_RETRY) makes more sense in the end. Sorry for making you update it.

[starrify]

np :)

I guess making TunnelError default and introducing a configurable EXCEPTIONS_TO_RETRY are both good things to do. But one thing at a time, and I've just discarded the 2nd commit in this PR.

[redapple]

Thanks @starrify