GitHub badges showing "Unable to select next Github token from pool"

[jcs090218]

Are you experiencing an issue with...

shields.io

🐞 Description

Similar to #8907.

🔗 Link to the badge

My repo https://github.com/emacs-eask/cli getting the error.

💡 Possible Solution

No response

[WSTxda]

same issue for me

repo: https://github.com/WSTxda/MicroG-RE

but it's happen randomly

[jcs090218]

Interesting... the repository https://github.com/emacs-eask/cli appears to no longer have the issue, yet now my other repository https://github.com/jcs090218/JCSUnity (which was previously functional) is encountering the same error. It appears to occur randomly. 🤔

[m4heshd]

This seems to be an issue related to GitHub's image caching system ("camo").

If you check https://github.com/m4heshd/ufc-ripper , the first badge is broken but if you visit the actual URL pointing to the image, it works.

[WSTxda]

This seems to be an issue related to GitHub's image caching system ("camo").

If you check https://github.com/m4heshd/ufc-ripper , the first badge is broken but if you visit the actual URL pointing to the image, it works.

Actually this doesn't work in the direct link either

[m4heshd]

Actually this doesn't work in the direct link either

Works for me. 🤔

Tried both with and without the browser cache.

[WSTxda]

desktop also

i think its just randomly

[m4heshd]

@WSTxda Hmm.. it might be request-based. Can you try with your browser's guest profile?

[jcs090218]

It's very random. I can see some of them work and some of them not... 😕

[007revad]

I just refreshed the readme page for one of my repos for a second it correctly showed the latest release version but then changed back to "Unable to select next GitHub token from pool"

I just tried a different browser and it correctly shows the latest release version.

I also tried an incognito Chrome window and it correctly shows the latest release version.

[pwbriggs]

I'm seeing this error intermittently, too. Could the token pool be exhausted? (@espadrine)

Whether or not that's the cause of these errors, if you haven't already given shields.io a token, you can do that here: https://img.shields.io/github-auth. not actually the issue, see below.

[chris48s]

Not exactly sure what happened here, but it looks like 3 of the servers in the cluster got themselves into a state where they hit a token that had maxed its rate limit and weren't able to recover and pull another token from the pool.

This is why you're seeing it intermittently: Depending on what server your request got sent to at the load balancer, you may or may not have seen the issue.

I've just restarted all of the VMs. This should clear up as everything drops out of cache.

We haven't actually run out of tokens. We have many more than we need. Thanks

[chris48s]

OK. This problem is back. Restarting cleared it for about an hour, but now it looks like most of the VMs in the cluster have hit the same state again. Not sure what is going on, but I'm aware of it.

[chris48s]

OK. We're still seeing this issue intermittently.
It first started about 8-9pm UK time yesterday. Up to that point, everything was fine.
We do a lot of traffic on the GH API, but there hasn't been a big increase and the number of tokens we have in the pool should easily cover our usage many times over.
There haven't been any recent changes to the code for managing the token pool.
I'm tempted to assume there has been a change upstream to the GitHub API. One thing we've seen before in the past is API responses intermittently missing x-ratelimit-* headers. However, I've tried pulling down some tokens from the pool and running thousands of requests against my local copy to see if I can get a local repro: nothing :(
For the moment, I am stumped.

[imfx77]

It's weird, on the same repo some of the badges work, and some don't.
Guess depends to which server goes the request, because when I force refresh the repo page the broken badges change randomly.

[chris48s]

This seems to have magically fixed itself. I didn't change anything, but about 9 hours ago, the Token pool is exhausted errors just.. stopped. The last one happened at Feb 19, 2024 10:04:02 PM UTC

Given this both happened and fixed itself with no intervention from us, I assume this was some kind of incident on the GitHub API side, but I'm still none the wiser what happened or how we could try and be more resilient to it. I never managed to reproduce it.

I'm going to leave this issue open for the moment and continue to monitor it.

[chris48s]

This hasn't recurred, so I'm going to close it.