Wikipedia robots.txt raises exceptions #2373
Closed
Comments
I can reproduce this with Python 2.7 on Linux but not with Python 3.
|
It does look like a bug in Python 2's
|
This line seems to be the issue. |
What do you suggest @redapple? Use a backport of Python 3.x robotparser, switch to reppy, create our own robots.txt parser, tell people to use Python 3, something else? |
I think we can move to |
reppy sounds fine, but I have a few small reservations about it:
|
oh right, reading some recent PRs. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm scraping a page which in turn links to wikipedia.
But the wikipedia robots.txt is creating some errors/exceptions as below.
Python 2.7.12
Scrapy 1.2.1
The text was updated successfully, but these errors were encountered: