Make Exceptions Handled by Retry Middleware Configurable

[tad3j]

I was having troubles finding which exceptions are handled by Retry Middleware, so I asked on freenode scrapy irc channel (should obviously check the source code). Paul from scrapinhub helped me there and pointed me to the source code.

Beside that he also asked me to open an issue here.
He gave a great suggestion to make exceptions handled by RetryMiddleware configurable, so if we don't like defaults we can configure our own.

Beside that he said that the handled exceptions in question are not documented well; I'm just mentioning it here, didn't want to open another issue (I guess by adding them to settings will also require some documentation).

[sanjucta]

Hi,
Since this is tagged as an as 'help-wanted' and 'enhancement', I think the help wanted here is to make the exceptions handled configurable. I'd like to work on this.
I think the way the exceptions should be made configurable is to allow the user to exclude certain exceptions rather than add exceptions, since we only want to re-try in the case of exceptions in which the issue could have been resolved when we retry . Does that sound reasonable?

[doprdele]

^^ would also like to see this. how exactly should we retry on exceptions now?

[serhii73]

I'm on it.