Rate limit errors

[danielpops]

TL;DR

When legitify analyze'ing across large number of repos (in my organization, there are ~290), rate limit errors are encountered, and there is no way to slow the rate and mitigate the errors (other than reducing the number of repos in scope for analyze)

Expected behavior

Exponential backoff by default and/or allow the caller to specify a value for wait between requests

Observed behavior

Errors like this:

2023/01/12 11:09:25 2023/01/12 11:09:25 error getting branch protection info for the_repo: GET https://api.github.com/repos/TheOrg/the_repo/branches/master/protection: 403 You have exceeded a secondary rate limit. Please wait a few minutes before you try again.

which results in incomplete results in output.

Version

legitify version 0.2.1 commit a5a45e1

On which operating system are you using legitify?

Linux

Relevant log output

2023/01/12 11:09:25 2023/01/12 11:09:25 error getting branch protection info for the_repo: GET https://api.github.com/repos/TheOrg/the_repo/branches/master/protection: 403 You have exceeded a secondary rate limit. Please wait a few minutes before you try again.

### Additional information

_No response_

[gofri]

Sounds legit.
While we work it out, I'd suggest using list-repos to get the list of repos, then use the output to run analyze with the --repository flag and a subset of the repos each time (let me know if you need help with that)

[gal-legit]

@danielpops can you please checkout gofri/transport, run make build and run legitify to see if that fixes the issue for you?

[gal-legit]

hey @danielpops,
The secondary rate limit fix is now released: https://github.com/Legit-Labs/legitify/releases/tag/v0.2.2
so feel free to pick legitify off the release and try it out.
I'm closing this issue for now; If you still tackle rate limit issues running v0.2.2 please comment here and we'll reopen it.

[iamvolvo]

@gofri @gal-legit I believe I've just run into this.

My org has ~7k repositories.
And I'm running legitify 1.0.7 Commit Homebrew

Thanks, and let me know if there's anything else I could help with.
Alternatively, would it be possible to consider implementing support for specifying your own rate limit?

2024/05/21 12:33:57 2024/05/21 12:33:57 error getting rules set for repo1: GET https://api.github.com/repos/my-org/repo1/rules/branches/master: 403 You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 1660:26EC0F:31DC5A3:323B67D:664C7894. []
2024/05/21 12:33:57 2024/05/21 12:33:57 error getting rules set for repo2: GET https://api.github.com/repos/my-org/repo2/rules/branches/master: 403 You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 1660:26EC0F:31DC62D:323B710:664C7895. []
2024/05/21 12:33:57 2024/05/21 12:33:57 attempt 3/5 failed: collect repositories for my-org with err: non-200 OK status code: 403 Forbidden body: "{\n "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 1660:26EC0F:31DBACD:323AB9C:664C7893."\n}"
2024/05/21 12:33:57 2024/05/21 12:33:57 all 5 attempts failed (collect repositories for volvo-cars) with err: non-200 OK status code: 403 Forbidden body: "{\n "documentation_url": "https://docs.github.com/free-pro-team@latest/rest/overview/rate-limits-for-the-rest-api#about-secondary-rate-limits\",\n "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 1660:26EC0F:31DC8E9:323B9D1:664C7895."\n}"
2024/05/21 12:33:57 2024/05/21 12:33:57 BUG: closing bar repository although it is not completed. please report this issue to legitify repository.

Thanks!