Protocol interface to take metadata into account

[jnioche]

Ideally we should be able to access the metadata for a URL from the protocol implementations. One motivation for this is that we could use different resources (e.g. scripts for processing AJAX) based on the value of a metadatum.

What would be the best way of doing this? add a separate method? if so would what would the FetcherBolt do? Replace the existing method with the one handling metadata? This way if a protocol implementation does not need the metadata it could simply ignore them.

Any thoughts on this?

[jakekdodd]

Maybe make make ProtocolFactory.getProtocol() a variadic method, so that in the fetcher bolt protocolFactory.getProtocol(u) would become protocolFactory.getProtocol(u, metadata) if metadata is present? Alternatively, the metadata could be added as an argument to protocol.getProtocolOutput(). I guess it depends on whether the metadata would be needed in setting up the protocol (in which case it would be needed by the factory), or just when generating the request.

[jnioche]

Implemented by [https://github.com/DigitalPebble/storm-crawler/commit/bade3806c2db8a0592cdd8c4acb4c5c027f80a68]