Fix API calls failing because of secondary api limits #1853
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue: After upgrading GitHub Enterprise Server (GHES) from version 3.9 to 3.11 and 3.12, we encountered issues with failing API calls during Git search (and other)operations. Further investigation revealed that the newer GHES versions consume more CPU resources, leading to secondary rate limits being reached more frequently. Below is a quote from GitHub Enterprise Support:
"Customers planning to upgrade to GitHub Enterprise Server (GHES) version 3.11 or higher should be aware of the potential for increased CPU utilization. If a system's peak CPU usage is already high, typically above 65-70%, it is strongly advisable to scale up the server resources before upgrading. This is to accommodate the additional load from new features and improvements that are part of the latest releases."
Moreover, we realized that GitHub does not guarantee the presence of the
Retry-After
header. However, thegh-limited-by
header is included in the response when the error is due to rate-limitingWe got ourselves unblocked after making this change. I see the issue is reported by another user here as well
#1805
Fix: The intention is not to change any existing functionality. So ended up adding an additional optional check, so that the API client can wait for another minute before retrying the API call if either of the headers are present. With the new change the client will execute this block https://github.com/hub4j/github-api/blob/main/src/main/java/org/kohsuke/github/AbuseLimitHandler.java#L82-L93 which is not happening right now