Getting 429 Too Many Requests when GET-ting https://github.com/<org>/<repo>/(?:blob|tree)/.* links

[ppalaga]

Select Topic Area

Question

Body

Our project contains sources of Antora documentation site. We validate all links available in our docs on every pull request.
Among others, we have a few tens of links referring to various source files on GitHub. Because we maintain the documentation separately for all active LTS (long term support) branches, those links are pointing to source files in specific branches. Here are some examples:

• https://github.com/quarkiverse/quarkus-cxf/blob/main/integration-tests/ws-trust/src/main/resources/application.properties
• https://github.com/quarkiverse/quarkus-cxf/blob/3.15/integration-tests/ws-security-policy/generate-certs.sh

The validation consists in sending an anonymous GET request to those URLs - that's basically what readers of our documentation would do. We use jsoup without setting any special user agent or any other request headers.

It worked fine till one or two days ago. Now, there seem to be a Rate limit of 4 requests per minute set for https://github.com/<org>/<repo>/blob/... and https://github.com/<org>/<repo>/tree/... URLs.
That makes our testing slower by ~7 minutes, thus burning a lot of CPU time on GH Actions workers.

More observations:

• GET-ting other github.com links still works fine.
• I hit the limit of 4 requests per minute is also when browsing resources under some branch with a browser without being logged in.
• Sending HEAD requests instead of GET makes no difference. I wonder whether GitHub would be ready to loosen the limit for HEAD requests at least? Those are not useful for AI training.

Could you please suggest, how can we make our testing fast again?

[Marantral]

Use a Access Token. Also there is a header within the Github response that will tell you how many requests you have before they throttle your access. I tend to add sleeps until that value resets.

That amount is much less without a token.

Look at how I did it within my Orphaned Commit scanner tool:
https://github.com/Marantral/orphanedCommit/blob/main/orphane.py

I looked at the 'X-RateLimit-Remaining' and the 'x-ratelimit-reset' header values to manage my interactions.

Detailed Steps:
Navigate to Settings:

• Go to your GitHub profile, click on the profile picture, and then select "Settings".
• In the left sidebar, click on "Developer settings".
  Access Fine-grained Tokens:
• Under "Personal access tokens," click on "Fine-grained tokens".
  Generate a New Token:
• Click on "Generate new token".
  Configure the Token:
• Token Name: Enter a descriptive name for the token.
  Expiration:
• Set an expiration date for the token, allowing for infinite lifetimes if your organization doesn't have a maximum lifetime policy.
  Resource Owner:
• Choose whether the token will have access to your personal account or to an organization.
  Select Repository Access:
• Choose whether the token should have access to all repositories or only specific repositories.
• If you select only specific repositories, you can choose which ones from a dropdown.
  Specify Permissions:
• Choose the appropriate permissions for the token, which define what resources the token can access.
• Consider using the minimum permissions needed for your use case.
  Generate the Token:
• Click "Generate token" to create the token.
  Note: You will only be able to see the token once, so make sure to save it securely.
  Note: If they are just public repositories that you want to access, set the token access to public.

Thanks,

[ppalaga]

Thanks for your suggestions, @Marantral! I think your situation is a bit different than mine: you access the GitHub API via https://api.github.com/ but I access the WEB UI via https://github.com/ - I need to validate those links, because that's where I am sending my users from my documentation.
Unlike https://api.github.com/, https://github.com/ does not send X-RateLimit-Remaining and the x-ratelimit-reset. I also tried with a token but it did not change anything.

[Marantral]

I would assume that they will most likely be similar. You should be able to validate items in the front end through the API.

[ppalaga]

Well, yes, there would be some value in letting my test programm to transfrom the docs links (intended for humans) into API links before accessing them. But in that way, I would actually be testing the availability of other resources than my human users will access in reality. The accessibility of those may differ now in the future for reasons that are out of my control.

[Marantral]

It would be the same source data. You accessing a public repo via an API is doing basically the same thing that the front-end UI is doing for your users. Using the API is the correct way to programically access that data.

[ppalaga]

I think the main difference for my use case is the authentication. If I let my program test the links with an API token set, it may be able to access resources that are invisible to for an anonymous user, thus producing false positive test results. I guess there is no way to set up a token that has only access to public resources?

[Marantral]

That is the awesome thing about tokens: you can set them only to view public repositories.

That is a fine-grained token

You can do anything that you can do from the Front-end within the API. It is the same data source, and it will allow you to programmatically interact.

https://docs.github.com/en/rest?apiVersion=2022-11-28

Using the approved API will work for you and your use case and you will not have to worry as much about those 429 responses.

Tree interactions:
https://docs.github.com/en/rest/git/trees?apiVersion=2022-11-28

Blob Interactions:
https://docs.github.com/en/rest/git/blobs?apiVersion=2022-11-28

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[ppalaga]

I consider this still an issue. GitHub should provide a way how a script should validate non-GitHub API links. Transforming non-GitHub API links to their API counterparts cumbersome and error-prone. It would be great if my script could validate the non-API links using a token or whatever other kind of auth.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬

[ppalaga]

Thanks for the reminder. I consider this still an issue. GitHub should provide a way how a script should validate non-GitHub API links. Transforming non-GitHub API links to their API counterparts cumbersome and error-prone. It would be great if my script could validate the non-API links using a token or whatever other kind of auth.

[jfullstackdev]

not very sure of your situation, but here are the possible workarounds maybe :

1. authenticate the requests

   • Use a GitHub personal access token (PAT) or GitHub Actions GITHUB_TOKEN and call the GitHub REST API instead of anonymous scraping.

   • Example: Instead of fetching https://github.com/.../blob/..., use the Contents API

     GET https://api.github.com/repos/<org>/<repo>/contents/<path>?ref=<branch>
     

   • This bypasses the public web rate limits.

2. check raw content URLs

   • Use https://raw.githubusercontent.com/<org>/<repo>/<branch>/<path> instead of the blob URL.
   • These are served as static files and are not rate-limited the same way.

3. reduce redundant checks

   • Cache previously validated links per branch/version.
   • Skip re-validating unchanged links in CI.

4. custom User-Agent

   • Sometimes setting a proper User-Agent header avoids being treated as a bot, but in this case GitHub likely enforces limits regardless.

[ppalaga]

Thanks for your suggestions, @jfullstackdev.

authenticate the requests

• Use a GitHub personal access token (PAT) or GitHub Actions GITHUB_TOKEN and call the GitHub REST API instead of anonymous scraping.
• Example: Instead of fetching https://github.com/.../blob/..., use the Contents API

  GET https://api.github.com/repos/<org>/<repo>/contents/<path>?ref=<branch>
  

• This bypasses the public web rate limits.

Yes, I know this workaround and I can confirm it works. The main problem I have with it is that in this way I am going to validate a different link than my users would click on our documentation site. So this is still a workaround and it would be better if GitHub would start supporting auth on non-API links.

check raw content URLs

• Use https://raw.githubusercontent.com/<org>/<repo>/<branch>/<path> instead of the blob URL.
• These are served as static files and are not rate-limited the same way.

Thanks, interesting. Would be worth following if raw.githubusercontent.com really does not enforce limits or auth.

• reduce redundant checks

  • Cache previously validated links per branch/version.
  • Skip re-validating unchanged links in CI.

• custom User-Agent

Does not help. Github source files come and go and we need a way to make sure the ones we send our docs readers to still exist.

• Sometimes setting a proper User-Agent header avoids being treated as a bot, but in this case GitHub likely enforces limits regardless.

Could you please be more specific about what the User-Agent header should look like?

[jfullstackdev]

it's not totally the answer but part of best practice :

Java (jsoup) example:

Connection connection = Jsoup.connect(
    "https://raw.githubusercontent.com/quarkiverse/quarkus-cxf/3.15/integration-tests/ws-security-policy/generate-certs.sh"
).userAgent("MyDocsLinkChecker/1.0 (+https://github.com/my-org/my-repo)");

Document doc = connection.get();

Raw HTTP example:

GET /quarkiverse/quarkus-cxf/3.15/integration-tests/ws-security-policy/generate-certs.sh HTTP/1.1
Host: raw.githubusercontent.com
User-Agent: MyDocsLinkChecker/1.0 (+https://github.com/my-org/my-repo)
Accept: */*

but again this is just good practice

GitHub’s new 4-requests/minute cap on blob/tree URLs applies no matter what User-Agent you send. To really avoid 429 Too Many Requests, you’ll need to use either the REST API with authentication or raw.githubusercontent.com URLs.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬