Retry-After header missing?

[ewels]

If you hit the GitHub API with a lot of requests at once to make PRs etc, you may run in to the API Abuse rate limits.

The documentation tells us that:

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

However, when I run in to these problems I never seem to have that header set:

{
	"message": "You have triggered an abuse detection mechanism and have been temporarily blocked from content creation. Please retry your request again later.",
	"documentation_url": "https://docs.github.com/overview/resources-in-the-rest-api#abuse-rate-limits"
}

{
	'Server': 'GitHub.com',
	'Date': 'Tue, 11 May 2021 11:10:25 GMT',
	'Content-Type': 'application/json; charset=utf-8',
	'Transfer-Encoding': 'chunked',
	'X-OAuth-Scopes': 'repo, workflow',
	'X-Accepted-OAuth-Scopes': '',
	'X-GitHub-Media-Type': 'github.v3; format=json',
	'X-RateLimit-Limit': '5000',
	'X-RateLimit-Remaining': '4628',
	'X-RateLimit-Reset': '1620734914',
	'X-RateLimit-Used': '372',
	'X-RateLimit-Resource': 'core',
	'Access-Control-Expose-Headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
	'Access-Control-Allow-Origin': '*',
	'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload',
	'X-Frame-Options': 'deny',
	'X-Content-Type-Options': 'nosniff',
	'X-XSS-Protection': '0',
	'Referrer-Policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
	'Content-Security-Policy': "default-src 'none'",
	'Vary': 'Accept-Encoding, Accept, X-Requested-With',
	'Content-Encoding': 'gzip',
	'X-GitHub-Request-Id': '0780:11AC:3AA0BD0:6B7151F:609A6620'
}

The Access-Control-Expose-Headers lists Retry-After as an allowed header, but the Retry-After header itself is not there.

I’m calling the API using the requests library in Python, code here.

Am I missing something at my end, or is this a problem with the GitHub API?

Phil

[nethgato]

Hi there @ewels 👋

Thanks so much for joining us and for sharing your experience. There is additional documentation that speaks to the lack of Retry-After header found here:

docs.github.com

Best practices for integrators - GitHub Docs

//docs.github.com/en/rest/guides/best-practices-for-integrators

Specifically:

• Requests that create content which triggers notifications, such as issues, comments and pull requests, may be further limited and will not include a Retry-After header in the response. Please create this content at a reasonable pace to avoid further limiting.

The behavior you described in your OP:

If you hit the GitHub API with a lot of requests at once to make PRs etc

…is precisely what we try to protect against. As this is an abuse protection measure, it is based on multiple input variables, and there isn’t an X value of seconds to suggest.

The process to resolve this would be to decrease the frequency of the scripts calls dramatically, and increase as needed. It looks like the code already has logic to identify an updated frequency value, when that header is present; awesome!

Just gotta slow down to start, and everything should behave as you’re hoping.

[tyeth]

Hey, I'm now regularly not getting the rate limit info included in responses. Presumably more anti-hammering, I don't quite get it as I'm still able to make more requests? All the rate related headers are missing except token expiration:

-- Remaining defaulted to -1: Github responded with no rate limit info, possible problems, printing reponse (200):
Request (get) - URL: /repos/adafruit/Adafruit_DRV8830/releases
Response text: []
Response header Date=Tue, 01 Oct 2024 18:50:30 GMT
Response header Content-Type=application/json; charset=utf-8
Response header Content-Length=2
Response header Cache-Control=private, max-age=60, s-maxage=60
Response header Vary=Accept, Authorization, Cookie, X-GitHub-OTP,Accept-Encoding, Accept, X-Requested-With
Response header ETag="cbff93ad9ca68eafbd30c57f01c1049f36162ed46ab7a757a38674943a9e85e0"
Response header X-OAuth-Scopes=read:org, read:packages, read:public_key, read:repo_hook, read:user, repo, user:email, workflow
Response header X-Accepted-OAuth-Scopes=repo
Response header github-authentication-token-expiration=2024-10-01 23:00:00 UTC
Response header X-GitHub-Media-Type=github.v3; param=hellcat-preview; format=json
Response header x-github-api-version-selected=2022-11-28
Response header Access-Control-Expose-Headers=ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
Response header Access-Control-Allow-Origin=*
Response header Strict-Transport-Security=max-age=31536000; includeSubdomains; preload
Response header X-Frame-Options=deny
Response header X-Content-Type-Options=nosniff
Response header X-XSS-Protection=0
Response header Referrer-Policy=origin-when-cross-origin, strict-origin-when-cross-origin
Response header Content-Security-Policy=default-src 'none'
Response header Server=github.com
Response header X-GitHub-Request-Id=ADF4:1C3425:9115B8:99D8E8:66FC4476
-- Continuing as if nothings wrong 😇

[ewels]

Amazing, thank you for the comprehensive answer @nethgato! I’m not quite sure how I missed that, seeing as the bullet is the one below the one I pasted in my original post 🤦‍♂️ I think I must have missed the critical word not (include…).

Anyway, that makes sense and as you say I added a backup approach if the header isn’t there. The context is a sync that runs once every few months but does various repos in parallel so submits 40 odd PRs all at once. How synchronous that is depends on how fast GitHub Actions runs the jobs so the abuse mechanism varies a lot. In the last run it was just one PR, in the one before it was about half. Anyway, my random delay seemed to work on the most recent run so I’m happy 👍

The main thing was that I thought I was going crazy / doing something wrong! So thank you for putting my mind at rest :slight_smile:

[nethgato]

So glad to help! You’re definitely not going crazy and I’m happy I was able to help verify sanity.

🙇

[Deepakgthomas]

I have a slightly similar question. I would highly appreciate your feedback.