Retry intermittent S3 download failures #594
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This builds on the recent work in botocore (boto/botocore#210) to address two things: * ensure than any .read() call used from the streaming response of `GetObject` will never hang. We do this by applying a timeout to the underlying socket. * Catch and retry IncompleteReadError. There may be times when we don't receive all of the contents from a `GetObject` request such that amount_read != content-length header. Botocore now checks this and raises an exception, which the CLI now catches and retries.
| @@ -292,17 +298,30 @@ class DownloadPartTask(object): | |||
|
|
|||
| # Amount to read from response body at a time. | |||
| ITERATE_CHUNK_SIZE = 1024 * 1024 | |||
| READ_TIMEOUT = 60 | |||
| TOTAL_ATTEMPTS = 5 | |||
There was a problem hiding this comment.
Where does this come from? Given our typical number of retries is 6, I'm just curious.
There was a problem hiding this comment.
I believe botocore uses 5 by default (except for special cases like ddb).
There was a problem hiding this comment.
TIL: https://github.com/boto/botocore/blob/develop/botocore/data/aws/_retry.json#L46-L53 Good to know, thanks.
|
LGTM otherwise. |
|
LGTM |
thoward-godaddy
pushed a commit
to thoward-godaddy/aws-cli
that referenced
this pull request
Feb 12, 2022
thoward-godaddy
pushed a commit
to thoward-godaddy/aws-cli
that referenced
this pull request
Feb 12, 2022
* fix: Functional tests must run on localhost to work in Windows (aws#552) * fix: spacing typo in Log statement in start-lambda (aws#559) * docs: Fix syntax highlighting in README.md (aws#561) * docs: Change jest to mocha in Nodejs init README (aws#564) * docs: Fix @mhart link in README (aws#562) * docs(README): removed cloudtrail, added SNS to generate-event (aws#569) * docs: Update repo name references (aws#577) * feat(debugging): Fixing issues around debugging Golang functions. (aws#565) * fix(init): Improve current init samples around docs and fixes (aws#558) * docs(README): Update launch config to SAM CLI from SAM Local (aws#587) * docs(README): Update sample code for calling Local Lambda Invoke (aws#584) * refactor(init): renamed handler for camel case, moved callback call up (aws#586) * chore: aws-lambda-java-core 1.1.0 -> 1.2.0 for java sam init (aws#578) * feat(validate): Add profile and region options (aws#582) Currently, `sam validate` requires AWS Creds (due to the SAM Translator). This commits adds the ability to pass in the credientials through a profile that is configured through `aws configure`. * docs(README): Update README prerequisites to include awscli (aws#596) * fix(start-lambda): Remove Content-Type Header check (aws#594) * docs: Disambiguation "Amazon Kinesis" (aws#599) * docs: Adding instructions for how to add pyenv to your PATH for Windows (aws#600) * docs: Update README with small grammar fix (aws#601) * fix: Update link in NodeJS package.json (aws#603) * docs: Creating instructions for Windows users to install sam (aws#605) * docs: Adding a note directing Windows users to use pipenv (aws#606) * fix: Fix stringifying λ environment variables when using Python2 (aws#579) * feat(generate-event): Added support for 50+ events (aws#612) * feat(invoke): Add region parameter to all invoke related commands (aws#608) * docs: Breaking up README into separate files to make it easier to read (aws#607) * chore: Update JVM size params to match docker-lambda (aws#615) * feat(invoke): Invoke Function Without Parameters through --no-event (aws#604) * docs: Update advanced_usage.rst with clarification on --env-vars usage (aws#610) * docs: Remove an extra word in the sam packaging command (aws#618) * UX: Improves event names to reflect Lambda Event Sources (aws#619) * docs: Fix git clone typo in installation docs (aws#630) * docs(README): Callout go1.x runtime support (aws#631) * docs(installation): Update sam --version command (aws#634) * chore(0.6.0): SAM CLI Version bump (aws#635)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on the recent work in botocore
(boto/botocore#210) to address two things:
GetObjectwill never hang. We do this by applying a timeoutto the underlying socket.
don't receive all of the contents from a
GetObjectrequest suchthat amount_read != content-length header. Botocore now checks
this and raises an exception, which the CLI now catches and retries.