Redirect fails silently - need debug messages

[perplexes]

Howdy. We just had a big debugging session centered around redirects, and it turned out that they were redirecting from non-www to a www.domain URL, so spidr silently failed, finding that it was a different host, which is true, but not intuitive.

May we suggest that there be an option to include a logger and logger level, then you could log "page (url) prevented by host filter". That would be super duper nice.

[watzon]

Giving an option to follow redirects as suggested in #56 would also solve this. I agree a logger is in order though.

[postmodern]

Logging is a good idea. What about also adding a callback for URLs that get filtered out?

[watzon]

Not a bad idea @postmodern

[postmodern]

@perplexes I'm just curious how you were invoking spidr? If you used Spidr.host then it would only spider that hostname, none of the other sub-domains. Coming in 0.7.0, the Spidr.domain method will allow one to spider a domain (starting at http://#{domain}/) along with any of it's sub-domains.