Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API rate limit exceeded when editing pages #6410

Open
jeremyzilar opened this issue Apr 15, 2022 · 14 comments
Open

API rate limit exceeded when editing pages #6410

jeremyzilar opened this issue Apr 15, 2022 · 14 comments
Assignees
Labels
area: extensions/backends/github type: bug code to address defects in shipped code

Comments

@jeremyzilar
Copy link
Contributor

We are still having issues with our NetlifyCMS instance where our team of editors keep running into issues updating content in our site (they are not technical people). When looking into it, I see a series of API rate limit errors in their consoles:

Pausing requests for 3600 second due to fetch failures: API rate limit exceeded for user ID 395641.

image

In digging further, these errors seem to happen when they try to load content via a relation widget in Netlify CMS. (see the three dots)

image

There are times when this works well, but for many, they experience so many issues uploading content that it can often take many attempts through the week to upload their content. It has been incredibly hard to debug what is happening. So far, my best guess is that it has something to do with authenticating with GitHub and the auth going stale.

Our config & setup

A theory

It could be that they log in at one point in the day they are logging in to our Netlify CMS, making a few edits, then leaving that browser window open. Then later in the day, they continue editing and at this point the CMS no longer authenticated with GitHub.

These are not people who will likely log in to GitHub during the day, nor are they checking out the code.

Any advice you can provide would be amazing 🎉

@wing5wong
Copy link
Contributor

wing5wong commented Jun 29, 2022

still experiencing related issue #4097

@ryangittings
Copy link
Contributor

@jeremyzilar did you manage to resolve this? Any advice/workarounds/fixed?

Cheers!

@willkuerlich
Copy link

It is very likely that you run into these https://docs.github.com/en/rest/rate-limit in scenarios of heavy usage of the github API

@wing5wong
Copy link
Contributor

It is very likely that you run into these https://docs.github.com/en/rest/rate-limit in scenarios of heavy usage of the github API

The usage is supposed to be cached, and when this issue occurs it does not appear to be.

Of course if there is no local caching this would be expected

@willkuerlich
Copy link

YOu can easily verify that issue at least by either checking the http headers or via "_token" (I believe this is the name) endpoint of github API

@wing5wong
Copy link
Contributor

YOu can easily verify that issue at least by either checking the http headers or via "_token" (I believe this is the name) endpoint of github API

Verifying the rate limit being hit is not the issue though. The issue is exactly that - the rate limit being hit unexpectedly

@samo4
Copy link
Collaborator

samo4 commented Jan 17, 2023

The crux of the problem is that CMS is firing the API requests more often than necessary (on relation widget see #5920, but also elsewhere). GIT backends on the other hand have strict rate limits. Github for example currently has 5k/h but that can be increased to 15k/h if you go with one of their paid plans.

I'm aware of various efforts to optimize this by reducing the number of requests, but ultimately there will always be a hard limit.

@jeremyzilar
Copy link
Contributor Author

Thank you @samo4 -- I will look at our use of the Relation Widget and maybe I can remove that. This is still happening, and our editors only have a few windows in the day when they can make edits, otherwise they are locked out. :/

@mohab-sameh
Copy link

@martinjagodic @demshy is there any progress on this issue? Quite troublesome for our workflow.

@demshy
Copy link
Member

demshy commented Nov 28, 2023

Back then, we fixed it on our fork and it seems to have helped. Since then, the fixes were already merged into Decap: #6410
#6902

Have you already tried using Decap 3.0? If you're still hitting rate limits, please inspect the network and maybe try to identify which sections are doing a ton of requests (possibly duplicate). Sharing your config would also help (to see if path regex fix applies to your case).

@mohab-sameh
Copy link

@demshy Oh that's great. I haven't tried Decap 3.0 yet. Looks like there shouldn't be breaking changes going from 2.10.79. I'll test it and report back. I'm deploying the CMS locally for now till I make sure the new version doesn't introduce breaking changes. Thanks a lot for pointing me to this.

@wing5wong
Copy link
Contributor

Just updated and still receiving this.
Going to allow a few reload to see if it only occurs on the initial application load since upgrading to decap 3.0

@mohab-sameh
Copy link

@demshy Oh that's great. I haven't tried Decap 3.0 yet. Looks like there shouldn't be breaking changes going from 2.10.79. I'll test it and report back. I'm deploying the CMS locally for now till I make sure the new version doesn't introduce breaking changes. Thanks a lot for pointing me to this.

Forgot to report back. After upgrading, this issue happens much less than before. However, #7218 is now the new blocker.

@wing5wong
Copy link
Contributor

Still experiencing rate limits.
Only 3 changes were made to pages before being limited.
image

I am not certain if this is related to relations on the pages, or the user utilising the Search field to find pages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: extensions/backends/github type: bug code to address defects in shipped code
Projects
Status: In Progress
Development

No branches or pull requests

8 participants