HTTP 502 error querying private repo with GITHUB_TOKEN

[klauern]

I'm trying to get information about our pull requests from an internal repo and I'm getting a 502 error and I'm not sure how to debug what the issue is:

mergestat "SELECT count(*) from github_repo_prs('private-org/private-repo');" -v
Apr 19 12:57:06 INF starting GitHub repo_pull_requests iterator for private-org/private-repo name=private-repo owner=private-org per-page=100
Apr 19 12:57:06 INF fetching page of repo_pull_requests for private-org/private-repo cursor=null name=private-repo owner=private-org per-page=100
Apr 19 12:57:13 INF fetching page of repo_pull_requests for private-org/private-repo cursor=Y3Vyc29yOnYyOpHOGcCNUA== name=private-repo owner=private-org per-page=100
+----------+
| COUNT(*) |
+----------+
+----------+
Apr 19 12:57:28 ERR failed to output resultset: non-200 OK status code: 502 Bad Gateway body: "{\n   \"data\": null,\n   \"errors\":[\n      {\n         \"message\":\"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `51EA:050C:152958:3CEDAD:625EF7F3` when reporting this issue.\"\n      }\n   ]\n}\n"

[patrickdevivo]

Hi @klauern - it's hard to know exactly from the GitHub API error, but something to try would be to set the GITHUB_PER_PAGE env variable to a smaller number (it defaults to 100) - try 50, if that doesn't address it, try going lower.

My guess is the relevant bit of the message is This may be the result of a timeout. If you're requesting too much data at once (many PRs with lots of content in their body for instance), it could cause the GitHub API to take too long to respond and throw this error.

Let me know if that helps!

We may want to change the default from 100 to a smaller number, as it's not uncommon to hit this type of error

[klauern]

Thank you for the tip! I didn't know the GITHUB_PER_PAGE var was available. I dropped it to 50 and things seem to be working.

[klauern]

do you have any flags for wait periods between queries?

Apr 19 13:41:49 ERR failed to output resultset: non-200 OK status code: 403 Forbidden body: "{\n  \"documentation_url\": \"https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits\",\n  \"message\": \"You have exceeded a secondary rate limit. Please wait a few minutes before you try again.\"\n}\n"

[klauern]

ok, I am using GITHUB_RATE_LIMIT as well, so I'll fiddle with these two values to see if I can find a happy place that passes all the tests. 50 and 1/3 don't appear to be enough

[patrickdevivo]

Great - GITHUB_RATE_LIMIT would be what I'd suggest. So if you're triggering a "secondary" rate limit, slowing down the call frequency may not actually address it. You might need to reduce the GITHUB_PER_PAGE again, as I believe the secondary limit could be triggered based on the "complexity" of results. This page has some info as well. My suggestion would be to try reducing GITHUB_PER_PAGE again to see if that helps. It makes the total collection run slower...but may avoid some of these errors