Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle AWS Cloudformation Throttling More Gracefully #48

Merged
merged 1 commit into from
Jul 29, 2019

Conversation

dissolved
Copy link
Contributor

We see frequently errors due to AWS throttling. Increasing the number of retries (the default was set to 3 retries) resolves the problem. The sdk does an exponential backoff, and 3 retries was not sufficient for our needs.

Considerations

This PR only increases the number of retries for cloudformation, but potentially, many of the AWS services may benefit from an increase in the retry limit. Additionally, I didn't see an established way for user configuration of such a constant (though imagine this could be set via the Staxfile. I chose to make it configurable via ENV var, but I can change that if there is a more preferred way. Lastly, I arbitrarily made the retry limit very high (100). The thought here was ctrl-c is your friend, but I honestly have no idea what a reasonable retry limit should be.

@rlister
Copy link
Owner

rlister commented Jul 29, 2019

This is great, thanks! Let's try it with 100, which may well be too high, but we'll drop it if necessary. We can also try a retry_backoff proc to bump up the amount of exponential backoff if necessary to avoid throttling.

@rlister rlister merged commit 2088fbb into rlister:master Jul 29, 2019
@dissolved
Copy link
Contributor Author

@rlister The docs also references a retry_jitter which apparently setting to :full might improve things, but for some reason it didn't recognize the parameter. I didn't dig deeper. Article on jitter here: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants