x/build/cmd/coordinator: add health check and graph to track hourly remaining GitHub API rate limit

[dmitshur]

While investigating #44404, I noticed in GopherBot's logs that it was failing to take some actions due to exceeding GitHub API rate limit quota:

$ kubectl logs -f gopherbot-deployment-6c6d86d5b9-s88c8 | grep "API rate limit"
[...]
2021/02/19 00:19:47 cl2issue: GET https://api.github.com/repos/golang/go/issues/44295/comments?per_page=1000&since=2021-02-19T00%3A14%3A00Z: 403 API rate limit of 5000 still exceeded until 2021-02-19 00:47:55 +0000 UTC, not making remote request. [rate reset in 28m08s]
[...]

When rate limit is exceeded, GopherBot stops being reliable for its users, and regular maintenance tasks do not occur.

This may be related to heavy activity, or perhaps it's caused by increased deterioration of issue #28320. This issue is to keep an eye on big of a problem it is and what we need to do here.

CC @golang/release.

[dmitshur]

Based on golang/crypto#143 (comment), this issue might be affecting more services than just GopherBot.

[dmitshur]

This is happening today too, affecting the "close cherry pick issues" task.

2021/03/01 20:54:36 close cherry pick issues: GET https://api.github.com/repos/golang/go/issues/44464/comments?per_page=1000&since=2021-02-26T10%3A29%3A53Z: 403 API rate limit of 5000 still exceeded until 2021-03-01 20:56:13 +0000 UTC, not making remote request. [rate reset in 1m43s]

[dmitshur]

From GerritBot logs today:

2021/03/15 17:47:59 getFullPR(ctx, "golang", "website", 40): b.githubClient.Do: GET https://api.github.com/repos/golang/website/pulls/40: 403 API rate limit of 5000 still exceeded until 2021-03-15 18:05:45 +0000 UTC, not making remote request. [rate reset in 17m45s]

[jeremyfaller]

Ticket opened w/ Github, awaiting response.

[jeremyfaller]

Poked Github. No rate limit increase is in the cards. We'll be held to 5k/hour unless we upgrade to an enterprise account (being related to the Google {which is an enterprise account} doesn't seem to help us here). I think we'll need to fix the underlying issues.

edit: We'd get 15k/hour if we upgraded.

[dmitshur]

Thanks for the update.

I'm working on getting a graph of our rate limit usage. Having that should help get a sense of how much/often the rate limit is being exceeded, and how much we need to decrease its usage by.