Arbitrary Redirection

[zapbot]

What steps will reproduce the problem?
1. Spider a site with redirection
2. crawl redirected msg
3. Request is from orginal msg, Response is from redirected msg

What is the expected output? What do you see instead?
When this happens not only the HttpMessage is dishonest, saved  and shown on the site
tree that way, but all the crawled anchors of this message will have a base url of
the original message. Which is unusable!

What version of the product are you using? On what operating system?
Version 1.1.1

Please provide any additional information below.
I suggest, for the spider at least, To not follow redirects in the HttpSender, but
keep the redirect response intact, and handle it in the spider thread, by for e.g.
adding the redirect url to the spider queue.

Original issue reported on code.google.com by amjad.masad on 2011-01-08 08:41:39

[zapbot]

(No text was entered with this change)

Original issue reported on code.google.com by psiinon on 2011-01-08 12:09:39

[zapbot]

(No text was entered with this change)

Original issue reported on code.google.com by THC202 on 2012-01-09 15:49:15

[zapbot]

r1126

Original issue reported on code.google.com by THC202 on 2012-01-09 16:51:55

[zapbot]

(No text was entered with this change)

Original issue reported on code.google.com by psiinon on 2012-04-08 13:19:51

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.