[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list

[jdemaeyer]

Inspired by #1334

I have decided not to include the HTTPERROR_ALLOWED_CODES setting, as, unlike the handle_httpstatus_list Spider attribute, the name of that setting implies that it is directly tied to the HttpErrorMiddleware.

[nramirezuy]

is it really necessary to use network for this test ? You can create a Response object with status=301 and a location header.

[jdemaeyer]

is it really necessary to use network for this test ? You can create a Response object with status=301 and a location header.

Hm, but that is what I'm doing?

[jdemaeyer]

Whoops but I just realized that I kept the dont_redirect meta in when I copied the request from another test

[nramirezuy]

Sorry I'm late to the party. But we missed request.meta [handle_httpstatus_list]; which is basically the same as the spider argument. http://doc.scrapy.org/en/master/topics/spider-middleware.html#module-scrapy.spidermiddlewares.httperror

[jdemaeyer]

Sorry I'm late to the party. But we missed request.meta [handle_httpstatus_list]; which is basically the same as the spider argument. http://doc.scrapy.org/en/master/topics/spider-middleware.html#module-scrapy.spidermiddlewares.httperror

Damnit. Proposed fix in #1447