Batch Changes - monitoring rolloutWindow - github "secondary" rateLimit

[DaedalusG]

Batch Changes - monitoring rolloutWindow - github "secondary" rateLimit

Is your feature request related to a problem? If so, please describe.

Users often encounter rateLimits from GitHub when implementing their first batch changes. This is discovered when github issues blocks requests from the user token for around an hour.

Currently Sourcegraph displays the following error message in the UI under a carrot dropdown in the batchs UI

reached GitHub's internal creation limit: see https://docs.sourcegraph.com/admin/config/batch_changes#avoiding-hitting-rate-limits

(PR where this was added)

Unfortunately if an admin runs into this error it can be difficult to determine what rate to set here -- effectively the troubleshooting process is too decrement the rate intermittently until requests are no longer being limited, waiting an hour between each decrement until a stable rate is reached.

This is a known issue relating in part to Githubs lack of response signal about the remaining requests before hitting the secondary limit.

It would be ideal if we could --

1. document more information about how batch changes consume the graphQL UI so that admins may make a more informed decision about what rate to set.
2. provide some monitoring about the number of requests consuming this API in grafana

Describe alternatives you've considered.

a long shot would be to automate this whole process so that Sourcegraph could asses and configure a good rateLimit automatically, while leaving the ability to manually configure this limit.

Additional context

This issues was observed in customer instances twice in a week, so it is assumed to be affecting a lot of batch changes users.

[DaedalusG]

This issue follows up on #39847

[chrispine]

How frequently does this happen? Is it basically a think that only happens once for new accounts? What percentage of accounts see this?

[malomarrec]

#44980

[malomarrec]

Quick fix was done by Warren, moving to backlog for a potential longer term fix.

[DaedalusG]

@chrispine apologies just seeing your questions --

How frequently does this happen? Is it basically a think that only happens once for new accounts? What percentage of accounts see this?

I only have anecdotal experience here, but I've run into primarily with admins in trial or who are just getting started. I've run into it several times in a fairly short period. It essentially shuts down batch changes when it comes up though, and ends up breaking developers flow as they wait out the hour long block from github. I'll see if we can track for this specifically in support.