Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry when rate limit exception occurs for getStreams #342

Merged
merged 2 commits into from Dec 16, 2019
Merged

Retry when rate limit exception occurs for getStreams #342

merged 2 commits into from Dec 16, 2019

Conversation

@danotorrey
Copy link
Contributor

danotorrey commented Dec 12, 2019

Fixes #341

  • Retries if Kinesis LimitExceededException occurs.
  • Increases page size for streams from 30 to 400 to reduce the need for successive calls. Retrying successive requests will only be needed for extremely large numbers of streams.

Ref :
https://docs.aws.amazon.com/streams/latest/dev/fundamental-stream.html
https://docs.amazonaws.cn/en_us/kinesis/latest/APIReference/kinesis-api.pdf
https://docs.aws.amazon.com/cli/latest/reference/kinesis/create-stream.html

CLI Reference Guide:
https://docs.aws.amazon.com/cli/latest/reference/kinesis/describe-limits.html

CLI-Example for AWS services IAM(Dynamodb|Cloud Watch|Kinesis...):

helium@helium-Precision-5540:~/workspace/config/shell$ aws iam get-user
{
    "User": {
        "UserName": "xxxx", 
        "PasswordLastUsed": "2019-12-12T17:00:43Z", 
        "CreateDate": "2019-11-06T14:01:44Z", 
        "UserId": "AIDAWV25I6RLUN4BMLKXB", 
        "Path": "/", 
        "Arn": "arn:aws:iam::459220251735:user/xxx"
    }
}
```
@danotorrey danotorrey requested review from ceruleancee and lingpri Dec 12, 2019
@danotorrey

This comment has been minimized.

Copy link
Contributor Author

danotorrey commented Dec 13, 2019

I need to add some comments to the code before this can be merged. I might have a chance to do that later.

@danotorrey

This comment has been minimized.

Copy link
Contributor Author

danotorrey commented Dec 16, 2019

@lingpri @ceruleancee I have added the needed comments. This PR is ready for review. Once merged, I will create the 3.1 backport PR.

@danotorrey danotorrey requested a review from lingpri Dec 16, 2019
Copy link
Contributor

ceruleancee left a comment

LGTM 👍

Copy link
Contributor

lingpri left a comment

I will test this bugfix for LimitExceededException, in the aws_multi_auth branch.

@lingpri lingpri merged commit 8b917cd into master Dec 16, 2019
3 checks passed
3 checks passed
ci-web-linter Jenkins build graylog-plugin-pr-linter-check 630 has succeeded
Details
graylog-project/pr Jenkins build graylog-project-pr-snapshot 7390 has succeeded
Details
license/cla Contributor License Agreement is signed.
Details
@lingpri lingpri deleted the issue-341 branch Dec 16, 2019
@danotorrey

This comment has been minimized.

Copy link
Contributor Author

danotorrey commented Dec 17, 2019

I will test this bugfix for LimitExceededException, in the aws_multi_auth branch.

Thank you @lingpri. :) I have rebased aws_multi_auth #279 with master to include this fix too, so you can test it now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.