Use Github graphql api to avoid rate limiting #104
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adjusts the index regeneration so that it retrieves the list of all pack versions (repo tags) in as few API calls as possible. At 1 call per pack per pack deployment, multiple pack deployments can easily hit the rate limit. Using graphql reduces the number of calls that count toward the rate limit (today) to 2 per pack deployment. Tags are apparently a very inexpensive API request, so github is only counting 1 call per page. More complex queries can count as more than one call per page, so this is very fortunate for our use case. Graphql limits each page to 100 results, so with about 160 packs, that is only 2 pages. So, we won't hit a 3rd page until we get over 200 packs.
|
I've tested my changes manually. Both |
|
I've tested the |
cognifloyd
force-pushed
the
use-graphql-for-index
branch
2 times, most recently
from
4f9766f
to
9808cb2
Compare
Also fix git clone "fatal: Too many arguments." error
cognifloyd
force-pushed
the
use-graphql-for-index
branch
from
9808cb2
to
8c62bef
Compare
cognifloyd
commented
Feb 8, 2021
Using the dependencies job requires updating .circleci/config.yml in all packs, so we continue installing it in the dependencies script until it has been deployed everywhere and is thus safe to remove.
blag
suggested changes
Feb 8, 2021
cognifloyd
force-pushed
the
use-graphql-for-index
branch
2 times, most recently
from
1f1acf0
to
d36f7a3
Compare
Co-authored-by: blag <blag@users.noreply.github.com>
blag
approved these changes
Feb 8, 2021
|
The new CI error messages we're getting for Exchange Packs after this PR: @blag Is that because of expired creds? @cognifloyd I guess we need a bit better error handling there. |
|
Yes it is. And yes we do. |
|
Captured in #109. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adjusts the index regeneration so that it retrieves the list of all
pack versions (repo tags) in as few API calls as possible.
At 1 call per pack per pack deployment, multiple pack deployments can
easily hit the rate limit. Using graphql reduces the number of calls
that count toward the rate limit (today) to 2 per pack deployment.
Tags are apparently a very inexpensive API request, so github is only
counting 1 call per page. More complex queries can count as more than
one call per page, so this is very fortunate for our use case.
Graphql limits each page to 100 results, so with about 160 packs, that
is only 2 pages. So, we won't hit a 3rd page until we get over 200
packs.