Attribute `handle_httpstatus_list` not working for codes 301 and 302 #1334
Comments
The
|
maybe the |
I think I tried making |
You need to set Here's a Spider that has # ~/playground/spidy.py
import scrapy
class Spidy(scrapy.Spider):
name = "Spidy the Spider"
start_urls = ["https://jigsaw.w3.org/HTTP/300/301.html"]
custom_settings = {'REDIRECT_ENABLED': False}
handle_httpstatus_list = [301]
def parse(self, response):
print "Got this:", response.status
|
In the code above, |
Hm, don't you mean |
Yeah, sorry. So in your example above, |
It was right jdemaeyer, all worked as expected it was an issue from some pages I was crawling. If nobody else is interested in RedirectMiddleware follow handle_httpstatus_list rule, we can close this issue. |
I think that's a good idea for RedirectMiddleware to respect |
+1 |
fixed by #1364 |
When set
handle_httpstatus_list = [301, 302]
, the spider doesn't execute parse. Howeber, it executes for other codes like 404.The text was updated successfully, but these errors were encountered: