core(is-crawlable): make sure that page is not blocked by robots.txt file

[kdzwinel]

First part of #4356

Extends is-crawlable audit, so that it also verifies robots.txt file (besides meta tag and header).

This change adds a new dependency - robots-parser. It's a lightweight, well tested robots.txt parser with no dependencies.

[kdzwinel]

Second part of #4356 is a new audit that checks if robots.txt file is valid. That audit should fail if fetch returns HTTP 500+ and that's why we collect it.

[paulirish]

I love the PR. just one question:

right now we don't really expose where the blocking directive came from... meta tag, resp header or robots.txt. feels like knowing that, in the case of a failure, would be really useful. wdyt?

[paulirish]

Also... looking at the robots-parser test suite, do you think there are any additional cases they should add?

i'm looking at https://github.com/ChristopherAkroyd/robots-txt-parser/tree/master/test/test-data https://github.com/python/cpython/blob/master/Lib/test/test_robotparser.py https://github.com/meanpath/robots/tree/master/test

[kdzwinel]

@paulirish we do show where the blocking directive comes from (header, meta, robots.txt) - check out the screenshot at the top. I'd be great to show which robots.txt directive is blocking, but robots-parser doesn't tell us that.

Regarding tests - I quickly scanned test suites that you linked and robots-parser test file and I don't see any prominent test cases missing. Are you concerned about some specific case?