Check that the status gets modified for pages that return 304

[jnioche]

SimpleFetcherBolt does nothing about it so the pages are parsed as usual.

FetcherBolt skips the parsing but probably doesn't update the status either.

Need to check if/how we handle conditional requests as a server would return a 304 only in such cases, although in practice I have seen servers do so even with a non-conditional one.

[jnioche]

Maybe it would make sense to share the code that handles that between the *FetcherBolt implementations? a super class?

[jnioche]

[https://github.com/DigitalPebble/storm-crawler/blob/master/core/src/main/java/com/digitalpebble/storm/crawler/protocol/httpclient/HttpProtocol.java#L131] uses the values for the keys cachedLastModified and cachedEtag if present in the metadata.

At the moment these k/v are not stored automatically See #99 and #109

[jnioche]

Note to self : need unit test with an embedded web server.
[http://wiremock.org/]
or classes from HTTPClient itself [https://thecarlhall.wordpress.com/2010/03/25/unit-testing-with-httpclients-localtestserver/]

[jnioche]

Added fix for FetcherBolt + attempt at writing a Junit test with an embedded server in a separate branch (does not work yet but it's a start)