-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce parallelism when polling from AWS APIs #40775
Conversation
Signed-off-by: Tiago Silva <tiago.silva@goteleport.com>
// This is a temporary solution until we have a better way to limit the | ||
// number of concurrent requests. | ||
eG.SetLimit(10) | ||
eG.SetLimit(5) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Should we make this configurable?
- Does AWS has different rate limits for different accounts? If yes, this change will help some and make stuff slower for other.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- No
- Yes and it depends if you requested an API quota increase. Once we move to SDK V2, I can implement a proper retry and rate limiter but until then we are kind of limited on what we can do.
@tigrato IMO this should be include in the release notes. Something like "reduce the number of concurrent requests to AWS API to reduce the rate limiting errors". |
Reduce parallelism when polling AWS APIs and add a retry definition
Changelog: Reduce parallelism when polling AWS resources to prevent API throttling when exporting them to Teleport Access Graph