New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
robotsparser deny all with some rules #80388
Comments
RobotsParser parse a "Disallow: ?" rule as a deny all, but this is a valid rule that should be interpreted as "Disallow: /?" or "Disallow: /?*" |
Can you provide a link to documentation showing that "Disallow: ?" shouldn't be the same as deny all? Thanks! |
I can't find a documentation about it, but all of the robots.txt checkers I find behave like this. You can test on this site: http://www.eskimoz.fr/robots.txt, I believe that this is how it's implemented now in most parsers ? |
I removed almost all messages of this issue since most of them looked list SPAM. I also blocked user accounts who posted SPAM. If it was a mistake, contact me. This is the Python bug tracker, not a forum to ask questions how to use Python, or to report bugs in your website. Multiple comments were written in French, whereas this bug tracker is in English. I even hesitate to close the issue since it got too many SPAM comments. |
I restored one non-spam message from the OP that was deleted. Changing to enhancement because this is not a bug (i.e., deviation from documentation). I don't know enough about this to have a view on whether this enhancement request should be accepted. |
I removed two comments: none of the mentioned URL contains a "Disallow: ?" rule and the comments didn't add any value to this issue. It looks like regular spam (SEO). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: