RobotsURLFilter

[jnioche]

The filtering of URLs based on the robots.txt directives is done within the Fetcher for an incoming URL. It would be efficient to be able to filter on the outlinks so that URLs do not get added to the queues (or any other form of persistence) if they can't be fetched anyway.

We should provide a RobotsURLFilter for this where we'd refetch the robots.txt file for a given URL and store it in a cache. The additional cost of pulling the robots.txt would be outweighed by the benefits of not adding unnecessary URLs to the queues.

[jakekdodd]

I'm working on this now; a few observations:

• We could potentially avoid fetching the rules a second time if they can be serialized and included in the metadata
• If we need to fetch and parse the rules a second time, it should happen outside of the filter itself. Making an HTTP request within the filter would lead to non-deterministic performance when filtering, which seems like a pain.

Have you started this? If not, I'll give it a go. It looks to me like the easiest approach, until we have more sophisticated metadata serialization, would be to include the raw text of the robots.txt file in the metadata and re-parse before filtering.

[jnioche]

I haven't started on this. Your approach makes complete sense, the way I was thinking about this was that we'd need re-fetching and parsing only once per hostname since the subsequent URLs would be in the cache so it would note be a big deal + there would be cases where we'd get links to external sites for which we wouldn't necessarily have fetched the robots already. I was also concerned about bloating the metadata too much or introducing changes into other components.

One way around the issue of fetching + parsing once when filtering would be to have the robots info stored in an external cache (Redis?) within a bespoke protocol implementation and connect to the same cache within the filter. Makes the whole thing more complex. I'd be tempted to stick to the simplest scenario for now and just refetch + parse, knowing that in practice it probably won't affect the performance very much and for cases where it does a more complex approach could be used.

[jakekdodd]

there would be cases where we'd get links to external sites

I couldn't think of a solution to that issue, we'll probably need to limit the robots filtering to outlinks with the same hostname as the source page.

I was also concerned about bloating the metadata too much

Me too--have you seen cases of huge robots.txt files? One way to limit the bloat would be to remove robots.txt from the metadata after filtering, so it's only carried in the tuple that travels between the fetcher and parser.

I like the idea of using an external cache, but that would be a burdensome requirement and barrier to entry for people wanting to use SC. What if we abstract the cache implementation within RobotRulesParser? This would enable users to plug in their own caching system. We could include a default cache that would have no outside dependencies.

[jnioche]

Me too--have you seen cases of huge robots.txt files?

Is that huge [http://www.amazon.com/robots.txt]? We could at least keep only the part relevant to the user-agent

One way to limit the bloat would be to remove robots.txt from the metadata after filtering, so it's only carried in the tuple that travels between the fetcher and parser.

hmm, that's more assumptions that we'd need to make explicit.

What if we abstract the cache implementation within RobotRulesParser? This would enable users to plug in their own caching system. We could include a default cache that would have no outside dependencies.

good idea

One thing though is that the filters are currently used within the parsing step but there is no reason why people could not use them somewhere else e.g. in a spout. In that case we wouldn't be able to leverage the metadata at all.

I'd be tempted to have a basic implementation doing an extra fetch+parse->cache and add an optimisation that would look in the metadata in case there is something interesting there.

This filter is just an idea I had and is not super high on my priority list, what I wouldn't want is to make other components more complex than necessary because of it. Of course if we can make other parts better / more generic then great!

[jakekdodd]

Check out #40

[jnioche]

Have implemented the simple version I originally had in mind. See #178
We can of course add a more sophisticated one based on the discussions we had here later on; for now this does exactly the job for small vertical crawls.