How to handle rate limiting from GitHub?

[iandouglas]

Turns out that 75+ user stories with lots of interdependencies and labels triggers a rate limit on the GitHub API.

Any thoughts on how best to throttle this? Got any contacts at GitHub to find out what this rate limit actually is so I can calculate how best to circumvent this especially if I have a bunch of students trying to run this all at the same time? :)

[iandouglas]

Even with a full second pause between creating each story I'm only able to import about 37 stories before GitHub shuts me down. :)

[iandouglas]

5 seconds between each new story seems to be more than enough, and 1.5 seconds between edits for dependencies and such seems to work okay for my 70-ish user stories with lots of dependencies and child/parent relationships. (total time was about 15 minutes)

It'd be nice, though, if the code could see that it's being rate-limited, pause for a while and retry the request on its own (with exponential backoff) instead of shutting down completely. Not sure this is something that comes back in GitHub's headers or not, or how we'd even intercept that.

[adamzolyak]

@iandouglas you should be able to interactively check on rate limits. Seems like GitHub Apps are governed by the same rate limits with the potential to earn an increased rate limit for larger organizational users. I'll also ask my team to see if anyone knows about other rate limiting.

Please share if you learn more.

[adamzolyak]

I got some additional info from GitHub. There's also an abuse rate limit. See:

https://developer.github.com/v3/#abuse-rate-limits

https://developer.github.com/v3/guides/best-practices-for-integrators/#dealing-with-abuse-rate-limits

[iandouglas]

I like it. I just couldn't find any docs from Probot to look at headers. I posted an issue for them a while back as well at probot/probot#779 but haven't had any response yet.

[adamzolyak]

Created #9 as a proposed improvement. Sorry for moving things around. This card seemed like a spike (answer some questions and learn about options), so suggest closing it.