New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wikipedia robots.txt raises exceptions #2373
Comments
I can reproduce this with Python 2.7 on Linux but not with Python 3.
|
It does look like a bug in Python 2's
|
This line seems to be the issue. |
What do you suggest @redapple? Use a backport of Python 3.x robotparser, switch to reppy, create our own robots.txt parser, tell people to use Python 3, something else? |
I think we can move to |
reppy sounds fine, but I have a few small reservations about it:
|
oh right, reading some recent PRs. |
I'm scraping a page which in turn links to wikipedia.
But the wikipedia robots.txt is creating some errors/exceptions as below.
Python 2.7.12
Scrapy 1.2.1
The text was updated successfully, but these errors were encountered: