[Robots.txt] Clarify behavior when to close blocks of multiple user-agents

[sebastian-nagel]

The RFC reference parser includes a unit test to verify that sitemap and other directives not covered by the RFC are just ignored and do not close blocks of rules. See robots_test.cc, line 126 expecting that BarBot is disallowed given:

User-agent: BarBot
Sitemap: https://foo.bar/sitemap
User-agent: *
Disallow: /

However, this interpretation might contradict the actual usage of certain non-RFC directives, see for example https://www.skyrock.com/robots.txt:

User-agent: *
Disallow: /cache/
Disallow: /demo/
...

User-agent: Twitterbot
Allow: /img/

User-agent: bingbot
Crawl-delay: 30

User-agent: 007ac9
User-agent: AhrefsBot
User-agent: LinqiaBot
User-agent: LinqiaRSSBot
User-agent: SMTBot
User-agent: HaosouSpider
User-agent: OpenindexSpider
User-agent: PetalBot
User-agent: serpstatbot
Disallow: / 

...

It's likely not not intended to exclude bingbot entirely - in fact, it wouldn't make sense to define a crawl-delay if a user-agent is entirely excluded. See also the discussion in google/robotstxt#51.

[garyillyes]

Hi Sebastian,

The spirit of the line in the RFC was to treat rules undefined by the RFC as comments, which in your example, as you note, merges the bingbot group together with the group that's following it. This was introduced because more common RFC-undefined rules such as Sitemap were closing groups when it was plain obvious that that's not what the site owner meant to do (i.e. a block of disallow rules followed without a preceding user-agent). I think the current implementation is the best solution to that, but I'm open to suggestions if you have a better idea.

[sebastian-nagel]

Hi @garyillyes, I agree that the RFC needs to find a clear and easy definition how unsupported directives are treated. Ignoring them sounds reasonable. I have no better idea, at least, by now.

The spirit of crawler-commons and its robots.txt parser was always to find a compromise between the standard and common practice. Before we change the current implementation, I'll try to estimate the impact of a change. Eventually, we need treat directives differently: the crawl-delay directive addresses individual user-agents but the sitemap directive is in a global scope.

[sebastian-nagel]

See also #114.

[sebastian-nagel]

See also the comments in SimpleRobotRulesParser, line 629: it should be well defined which instructions close a block of rules.