New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix linkcheck GitHub 429 errors #5396
Conversation
The nonstandard formatting and indentation makes fixing this annoying.
We hit URLs under https://github.com/freedomofpress/securedrop/issues more than GitHub likes, resulting in 429 Too Many Requests errors. These URLs aren't likely to change or go missing, so ignoring them seems a safe way to prevent spurious linkcheck failures. Also increase the number of worker threads checking URLs, to reduce the overall time linkcheck takes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm:
- Run make docs-linkcheck and confirm that it does not produce any 429 errors from any URL starting with https://github.com/freedomofpress/securedrop/issues/
"https://forum.securedrop.org/admin/users/list/active", | ||
"https://weblate.securedrop.org/projects/securedrop/securedrop/#repository", | ||
"https://github.com/freedomofpress/securedrop-debian-packages-lfs", | ||
r"https://github.com/freedomofpress/securedrop/issues/.*", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No 429s for issues tickets, but I did see one for a path on the develop branch:
Warning, treated as error:
/home/user/securedrop/docs/development/tips_and_tricks.rst.rst:134:broken link: https://github.com/freedomofpress/securedrop/tree/develop/docs/diagrams (429 Client Error: too many requests for url: https://github.com/freedomofpress/securedrop/tree/develop/docs/diagrams)
It might be worth adding https://github.com/freedomofpress/securedrop/tree/.*
to the list.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, passing for me locally now
Status
Ready for review
Description of Changes
Sometimes,
make docs-linkcheck
can fail because it hits too many GitHub URLs in a short timeframe and runs afoul of their rate limits. This PR tells the Sphinx linkcheck plugin to ignore anything at https://github.com/freedomofpress/securedrop/issues/ as those URLs are not likely to change or disappear.It also increases the number of linkcheck worker threads in an attempt to reduce the time this check takes.
Testing
Run
make docs-linkcheck
and confirm that it does not produce any 429 errors from any URL starting with https://github.com/freedomofpress/securedrop/issues/Deployment
dev only
Checklist
If you made non-trivial code changes: