Skip to content

gh-141444: Replace dead URL in urllib.robotparser example #144443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
kovan wants to merge 1 commit into
base: main
Choose a base branch
from
Open

gh-141444: Replace dead URL in urllib.robotparser example #144443

kovan wants to merge 1 commit into from
+6 −6

Conversation

@kovan
Copy link

@kovan kovan commented Feb 3, 2026 •
edited by github-actions bot
Loading

Summary

  • Replace defunct musi-cal.com URL with python.org in the RobotFileParser documentation example
  • Simplify the example to show basic can_fetch() usage since the original output values were specific to musi-cal.com's robots.txt

Closes #141444

Test plan

  • Verified python.org/robots.txt is accessible and returns valid robots.txt
  • make check in Doc/ directory passed

🤖 Generated with Claude Code

📚 Documentation preview 📚: https://cpython-previews--144443.org.readthedocs.build/

@kovan kovan requested a review from berkerpeksag as a code owner February 3, 2026 19:53
@bedevere-app bedevere-app bot mentioned this pull request Feb 3, 2026
Open
@bedevere-app bedevere-app bot added docs Documentation in the Doc dir skip news labels Feb 3, 2026
@github-project-automation github-project-automation bot moved this to Todo in Docs PRs Feb 3, 2026
@picnixz picnixz closed this Feb 3, 2026
@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

We do not accept changes generated solely by AI. Please read https://devguide.python.org/getting-started/generative-ai/.

@kovan
Copy link
Author

kovan commented Feb 3, 2026

But the contribution is useful and/or productive; I don't think you are reading the document right. Please re-read.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

I do think we need to update this but we don't accept AI-generated PRs.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

In addition, the example is now smaller which isn't really good. I would also prefer that the comments in the #141461 are taken into account.

@kovan
Copy link
Author

kovan commented Feb 3, 2026

The document is quite clear: as long as the contributions are useful or productive, it is allowed, or, rather, left to the mantainers to decide.
Maybe it could be rephrased better to avoid this kind of confussions, but I will repeat: I think you are not reading it correctly.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

And as a maintainer, I've decided that (1) removing most of the example is not useful (2) not addressing previous PR's comment is not useful either. So, you need to address them. Simply generating this PR without acknowledging what has already been discussed in #141461 is not acceptable.

@kovan
Copy link
Author

kovan commented Feb 3, 2026 •
edited
Loading

OK, now we are in talking terms. Please reopen the PR, so I can work on it; and the same for the other PR you closed.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

The other PR is not useful. I decided not to update the documentation.

@picnixz picnixz reopened this Feb 3, 2026
@StanFromIreland
Copy link
Member

StanFromIreland commented Feb 3, 2026

As previously discussed, it would be better to find a non-empty example.

picnixz
picnixz requested changes Feb 3, 2026
View reviewed changes
Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a non-trivial example.

@bedevere-app
Copy link

bedevere-app bot commented Feb 3, 2026

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be put in the comfy chair!

@StanFromIreland StanFromIreland mentioned this pull request Feb 3, 2026
Open
@StanFromIreland
Copy link
Member

StanFromIreland commented Feb 3, 2026

We could do something like python/pythontestdotnet#7?

@kovan @claude
pythongh-141444: Replace dead URL in urllib.robotparser example
da1cb06 
Replace the defunct musi-cal.com URL with pythontest.net in the
RobotFileParser documentation example. This uses CPython-controlled
infrastructure and provides non-None values for request_rate and
crawl_delay, addressing feedback from previous PR python#141461.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@kovan kovan force-pushed the gh-141444-robotparser-url branch from cc57fff to da1cb06 Compare February 3, 2026 21:40
@kovan
Copy link
Author

kovan commented Feb 3, 2026

I have made the requested changes; please review again

@kovan kovan closed this Feb 3, 2026
@bedevere-app
Copy link

bedevere-app bot commented Feb 3, 2026

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

@bedevere-app bedevere-app bot requested a review from picnixz February 3, 2026 21:43
@kovan kovan reopened this Feb 3, 2026
picnixz
picnixz approved these changes Feb 4, 2026
View reviewed changes
Copy link
Member

@picnixz picnixz left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those changes look better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@picnixz picnixz picnixz approved these changes

@berkerpeksag berkerpeksag Awaiting requested review from berkerpeksag berkerpeksag is a code owner

Assignees

No one assigned

Labels

awaiting merge docs Documentation in the Doc dir skip news

Projects

Docs PRs
Status: Todo

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Dead example URL in urlib.robotparser documentation

3 participants

@kovan @picnixz @StanFromIreland