2.x branch sitemaps

[simonwibberley]

A package of 'improvements' to sitemap handling, including:

• 'from' heuristic to skip loc's which have a lower lastmod than the provided timestamp. Includes refactored sitemap parse to eval sitemap node at the end of element to allow heuristic on indexes.
• optional error escalation: exceptions propagated instead of logged
• more defensive content-type checking: null check
• accept '204 no content' as successful response
• XML stream filter to remove / ignore invalid XML chars. Currently: unescaped control chars inc. '&', illegal white space preamble

Also, somewhat accidentally, includes an option to skip references we know would be beyond the depth limit.

Apologies for the messy commit history! I attempted to cherry pick only relevant changes.

Regards,

Simon

[simonwibberley]

@essiembre Happy to fill in any gaps if you point me in the right direction.

[essiembre]

Woah! Thanks for your contribution. I will merge it and also check if some of it can make it to version 3 (master branch). I'll let you know if I encounter issues.

I only have only one question/comment so far. The HTTP Collector already supports filtering documents on dates (DateMetadataFilter, in Importer module) and also supports not recrawling documents that do not feed a certain elapsed time. These two features combined can give you the same end result as the "from" you added. Did you add the "from" because you were not familiar with other options, or because you find it more convinient to filter it right at the source.

I can see rejecting sitemap entries as the sitemap is being read could significantly reduce the number of sitemap entries being queued for processing. For that alone, I think it is a worthwhile addition, but I am curious to find out more about your use case.

[essiembre]

One feature I think is having no effect is the "quitAtDepth" on the LinkExtractorStage. The reason being, URLs that are too deep are already rejected before link extraction could take place. If you have evidence of the contrary, please let me know how to reproduce as it woiuld be a bug IMO.

[simonwibberley]

@essiembre great, glad to contribute! Indeed, I wanted to prevent unnecessary downloading of ancient sitemaps which can be large. One of our use cases is to frequently get the latest articles from news sites by utilising only sitemaps, we want to run this on a managed schedule instead of recrawling based on update frequency and such (I'm not very familiar with that mode of operation yet).

FYI, some more sitemap related patches to PR soon...