Facing GitHub API quota issue in multi-branch pipelines.

[shailesh-vaidya]

We are using polling method to trigger Jenkins jobs on code commit. Seems we are exceeding GitHub API limit and that's causing delay in multi-branch pipelines execution.
Please refer - HA Multi-branch job.

Logs are as below,

Started by timer
[Mon Sep 14 05:25:00 UTC 2020] Starting branch indexing...
05:25:00 Connecting to https://api.github.com using cortx-admin/****** (RE Github Access)
05:25:00 GitHub API Usage: Current quota has 2936 remaining (483 over budget). Next quota of 5000 in 50 min. Sleeping for 8 min 8 sec.
05:28:00 GitHub API Usage: Still sleeping, now only 5 min 8 sec remaining.
05:31:00 GitHub API Usage: Still sleeping, now only 2 min 8 sec remaining.
05:33:09 GitHub API Usage: Current quota has 2320 remaining (590 over budget). Next quota of 5000 in 42 min. Sleeping for 9 min 51 sec.
05:36:09 GitHub API Usage: Still sleeping, now only 6 min 51 sec remaining.
05:39:09 GitHub API Usage: Still sleeping, now only 3 min 51 sec remaining.
05:42:09 GitHub API Usage: Still sleeping, now only 51 sec remaining.
05:43:00 GitHub API Usage: Current quota has 100 remaining (2194 over budget). Next quota of 5000 in 32 min. Sleeping until reset.

[shailesh-vaidya]

@mukul-seagate11 Can you please raise ticket with GitHub enterprise support for this.

CC: @priyankdal @gowthamchinna @swatiseagate

[gowthamchinna]

We are using cortx-admin cred for all Jenkins related github interaction please increase the API limit for this user

[gowthamchinna]

@mukul-seagate11 Can we consider this as priority I see lot of Jenkins jobs queued because of this issue

[shailesh-vaidya]

We are using cortx-admin cred for all Jenkins related github interaction please increase the API limit for this user

@gowthamchinna If issue is due to user limit can we distribute jobs in my and your's credentials. We can use this as temporary adjustment till quota is not increased.

[gowthamchinna]

distribute jobs in my and your's credentials. We can use this as tempo

Okay @shailesh-vaidya for PR VM deployment job I am using my credentials.

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya

[mukul-seagate11]

From: GitHub support@githubsupport.com
Sent: Sunday, September 27, 2020 11:38 PM
To: Mukul Malhotra mukul.malhotra@seagate.com
Subject: [GitHub Support] Re: Facing GitHub API quota issue in multi-branch pipelines

Stephen Mather(GitHub Support)Sep 27, 2020, 6:07 PM UTC

Hello Mukul,

The best way to avoid hitting our API rate limit in this scenario would be to switch from polling to https://docs.github.com/en/free-pro-team@latest/developers/webhooks-and-events/about-webhooks in order to trigger your Jenkins jobs on commit. If using webhooks is not an option for you, you could take advantage of our recent https://github.blog/changelog/2020-07-27-increase-to-api-limits-in-github-enterprise-subscriptions. When associated with an enterprise account, both GitHub Apps and OAuth Apps have an hourly limit of 15,000 API calls. Finally, you could consider splitting API calls between separate user accounts in order to avoid hitting rate limits for a single account. Let me know if you have any further questions.

Kindly
Stephen

[shailesh-vaidya]

From above update

• We can not use web-hooks as GitHub -> Jenkins connectivity for push is not configured. As Jenkins is not available outside.
• We tried using two separate users and seems issue frequency is reduced. @gowthamchinna can confirm.
• I feel we should increase rate limit as using two separate account is just temporary solution. We need create and maintain two accounts if we opt for using two accounts,

[gowthamchinna]

• We tried using two separate users and seems issue frequency is reduced. @gowthamchinna can confirm.
  Yes the issue frequency is reduced now.
• I feel we should increate rate limit as using two separate account is just temporary solution. We need create and maintain two accounts if we opt for using two accounts,
  Please consider user name jenkins-* or ci-* instead of cortx-admin

[gowthamchinna]

Also our repos are public now. I think we can remove credentials in the jobs where it only clones the repo.

[stale]

This issue/pull request has been automatically marked as stale because it has not had recent activity. Tagging @shailesh-vaidya

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @shailesh-vaidya for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[stale]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @shailesh-vaidya for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[shailesh-vaidya]

We are not facing issue now. We will re-open if issue occurs again.