Use protego as a default robots.txt parser

[whalebot-helmsman]

Implementation of proposal #3969

[codecov]

Codecov Report

Merging #4006 into master will increase coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4006      +/-   ##
==========================================
+ Coverage   85.43%   85.52%   +0.08%     
==========================================
  Files         167      167              
  Lines        9732     9741       +9     
  Branches     1456     1461       +5     
==========================================
+ Hits         8315     8331      +16     
+ Misses       1159     1153       -6     
+ Partials      258      257       -1

Impacted Files	Coverage Δ
scrapy/settings/default_settings.py	98.7% <100%> (ø)	⬆️
scrapy/logformatter.py	90.47% <0%> (ø)	⬆️
scrapy/core/engine.py	92.79% <0%> (+0.03%)	⬆️
scrapy/pipelines/files.py	66.16% <0%> (+0.78%)	⬆️
scrapy/core/scraper.py	89.33% <0%> (+2.84%)	⬆️
scrapy/extensions/corestats.py	100% <0%> (+10%)	⬆️

[elacuesta]

Thanks! Looks good to me, I just left a couple of minor comments.

[Gallaecio]

I see a few minor style issues with the documentation changes. Mainly, inconsistent use of capitalization (paragraphs that start with lowercase, lists where some entries are capitalized while others are not), inconsistent ending of list entries (lists where some entries end in a comma), etc.

I’m thinking it may be better to keep documentation changes here to the minimum here (e.g. just change which parser is the default), and propose a separate pull request which works on improving the documentation of the different parsers.

[kmike]

Hey! The PR looks fine to me.

I'm on fence on it, but maybe it can be nice to review protego API a bit before making it default, to avoid churn in future - e.g. in Python's default parser (in upcoming 3.8) there is .site_maps() method, while in protego there is .sitemaps property. On the other hand, it doesn't affect Scrapy directly, so maybe it is not needed.

[whalebot-helmsman]

review protego API a bit before making it default, to avoid churn in future - e.g. in Python's default parser (in upcoming 3.8) there is .site_maps() method, while in protego there is .sitemaps property.

I see no problem here. We don't directly use parsers' API in scrapy. Every parser should provide proper adapter conforming common interface. We tied to interface's API, not parser's API.

[kmike]

Ok, let's do it!