Handling of redirections

[jnioche]

The FetcherBolt nor the ParserBolt currently do anything specific with the HTTP status returned for a given page and as a result redirections are currently not handled.

One option would be to let whichever component is in charge of the persistence of the URL status i.e. post parsing. Another approach would be to have the parser (and maybe also the fetcher) declare a non-default output (e.g. 'status') and plug that as input to the persistence bolt.

any thoughts on this?

[jakekdodd]

A couple of thoughts. First, we should give the option to handle redirects within the fetcher itself, and the ability to specify a max number of redirect attempts.

Second, I think that for many status codes, some applications built with SC will want to skip parsing altogether (403, 404, etc.). However I think that this should be an option, and not hardwired behavior.

Let's think about how we can use streams to implement this. Handling redirection within the Fetcher itself is the easiest part to implement--maybe we should do that first?

[jnioche]

redirects within the Fetcher

do you meant put the redirected URLs straight into the internal queues? Could do that as an optional behaviour, I think this is the case in Nutch.

skipping parsing for non OK codes

let's do that with the status stream within the Fetcher

[jakekdodd]

do you meant put the redirected URLs straight into the internal queues? Could do that as an optional behaviour, I think this is the case in Nutch.

Exactly

let's do that with the status stream within the Fetcher

In the spirit of making these bolts easily extendable, maybe we could implement several event-handling methods that could be overridden in subclasses? For example there might be methods for handleRedirect(), handleForbidden(), handleTimeout(), etc. This would make it easy to emit to a non-default stream, or none at all, upon specific events

[jnioche]

Part of it is implemented in #44. Still need to implement case where redirected URLs are sent straight into the internal queues

[jnioche]

Will close for now. Redirections can be handled within the protocol implementations if necessary. Better just to treat them with the default behaviour and go through the status stream anyway as the targets might already be known. Keeps the code simpler as well.