Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement personal token rotation for GitHub when we hit rate limits #1625

Merged
merged 12 commits into from
Oct 9, 2023

Conversation

garrrikkotua
Copy link
Contributor

@garrrikkotua garrrikkotua commented Oct 4, 2023

Changes proposed ✍️

What

🤖 Generated by Copilot at b480ecd

This pull request adds token rotation functionality to the GitHub integration stream worker, which is a service that fetches data from GitHub using GraphQL queries. It introduces a new module tokenRotator.ts that handles the rotation of GitHub personal access tokens and their rate limit information in the Redis cache. It also updates the BaseQuery class, the processStream.ts file, the GithubPlatformSettings interface, and the ICache interface to support the token rotation feature. It adds a new configuration option for the GitHub integration stream worker to specify the list of personal access tokens.

🤖 Generated by Copilot at b480ecd

To avoid rate limits from GitHub
We added a tokenRotator hub
It uses the cache
To manage the stash
Of tokens that power our stream sub

Why

How

🤖 Generated by Copilot at b480ecd

  • Add a new configuration option for the GitHub integration stream worker, which is a comma-separated list of personal access tokens that can be used to rotate the API requests and avoid rate limits (link)
  • Define a new module that implements the logic for rotating the GitHub tokens and managing their rate limit information in the cache, using the ICache interface and the axios library (link)
  • Add a new field personalAccessTokens to the GithubPlatformSettings interface, which stores the list of tokens from the configuration (link)
  • Add three new methods to the ICache interface and the RedisCache class, which are wrappers for the Redis hash commands that allow storing and retrieving key-value pairs within a hash (link, link)
  • Import the GithubTokenRotator class from the tokenRotator.ts file in the baseQuery.ts and processStream.ts files, which use it to provide tokens for the GraphQL requests (link, link)
  • Add a new optional parameter to the getSinglePage method of the BaseQuery class, which is a GithubTokenRotator instance that can be used to provide a token for the GraphQL request (link)
  • Add a new process function that wraps the GraphQL request and the event data extraction, and modifies the getSinglePage method to use this function. It also adds a new error handling logic that checks if the error is related to the rate limit, and if so, tries to use the token rotation to retry the request. If the token rotation is not available or fails, it rethrows the error (link)
  • Declare a new variable tokenRotator that will hold a GithubTokenRotator instance for the current stream context in the processStream.ts file (link)

Checklist ✅

  • Label appropriately with Feature, Improvement, or Bug.
  • Add screehshots to the PR description for relevant FE changes
  • New backend functionality has been unit-tested.
  • API documentation has been updated (if necessary) (see docs on API documentation).
  • Quality standards are met.

@garrrikkotua garrrikkotua merged commit beed131 into main Oct 9, 2023
59 checks passed
@garrrikkotua garrrikkotua deleted the improve/github-token-rotation branch October 9, 2023 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants