Scrapy documentation (at the end of this section) links to a blog that contains performance numbers for different robots.txt parser backends. It would be better to change the link to point to this issue as it is more lasting. Currently, we are linking to a personal blog website which might disappear if the blog owner decides to change the hosting or blog name.
Scrapy documentation (at the end of this section) links to a blog that contains performance numbers for different
robots.txtparser backends. It would be better to change the link to point to this issue as it is more lasting. Currently, we are linking to a personal blog website which might disappear if the blog owner decides to change the hosting or blog name.