Implement personal token rotation for GitHub when we hit rate limits #1625
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes proposed ✍️
What
🤖 Generated by Copilot at b480ecd
This pull request adds token rotation functionality to the GitHub integration stream worker, which is a service that fetches data from GitHub using GraphQL queries. It introduces a new module
tokenRotator.ts
that handles the rotation of GitHub personal access tokens and their rate limit information in the Redis cache. It also updates theBaseQuery
class, theprocessStream.ts
file, theGithubPlatformSettings
interface, and theICache
interface to support the token rotation feature. It adds a new configuration option for the GitHub integration stream worker to specify the list of personal access tokens.
🤖 Generated by Copilot at b480ecd
Why
How
🤖 Generated by Copilot at b480ecd
ICache
interface and theaxios
library (link)personalAccessTokens
to theGithubPlatformSettings
interface, which stores the list of tokens from the configuration (link)ICache
interface and theRedisCache
class, which are wrappers for the Redis hash commands that allow storing and retrieving key-value pairs within a hash (link, link)GithubTokenRotator
class from thetokenRotator.ts
file in thebaseQuery.ts
andprocessStream.ts
files, which use it to provide tokens for the GraphQL requests (link, link)getSinglePage
method of theBaseQuery
class, which is aGithubTokenRotator
instance that can be used to provide a token for the GraphQL request (link)process
function that wraps the GraphQL request and the event data extraction, and modifies thegetSinglePage
method to use this function. It also adds a new error handling logic that checks if the error is related to the rate limit, and if so, tries to use the token rotation to retry the request. If the token rotation is not available or fails, it rethrows the error (link)tokenRotator
that will hold aGithubTokenRotator
instance for the current stream context in theprocessStream.ts
file (link)Checklist ✅
Feature
,Improvement
, orBug
.