Crawl spider crawls previously visited URL on redirect

[shaneaevans]

Previously reported by michaelvmata on Trac http://dev.scrapy.org/ticket/299

If a crawl spider is redirected to an already visited page, it will still crawl it.

From the mailing list http://groups.google.com/group/scrapy-users/browse_thread/thread/ee9ad68f5dbacc6d:

"...the dupe filter only catches requests after they leave the spider, so redirected pages are ignored by the dupe filter.

Since the dupefilter and the redirect middleware components are decoupled now, it would be awkward to implement what you suggest, but nevertheless I think it would be useful"

[dangra]

Dupe filter is part of scheduler now, and any redirected request goes trough it too and will be discarded if already seen.