Increase tenacity of S3RetryingInputStream #87243
Labels
>bug
:Distributed/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
Team:Distributed
Meta label for distributed team
Comments
DaveCTurner
added
>bug
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Jun 24, 2022
S3 sometimes enters a state where blob downloads repeatedly fail but with nontrivial progress between failures. Often each attempt yields 10s or 100s of MBs of data. Today we abort a download after three (by default) such failures, but this may not be enough to completely retrieve a large blob during one of these flaky patches. With this commit we start to avoid counting download attempts that retrieved at least 1% of the configured `buffer_size` (typically 1MB) towards the maximum number of retries. Closes elastic#87243
elasticsearchmachine
pushed a commit
that referenced
this issue
Jun 30, 2022
S3 sometimes enters a state where blob downloads repeatedly fail but with nontrivial progress between failures. Often each attempt yields 10s or 100s of MBs of data. Today we abort a download after three (by default) such failures, but this may not be enough to completely retrieve a large blob during one of these flaky patches. With this commit we start to avoid counting download attempts that retrieved at least 1% of the configured `buffer_size` (typically 1MB) towards the maximum number of retries. Closes #87243
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
S3RetryingInputStreamhides cases where S3 closes a connection partway through downloading a blob. By default it retries 3 times before failing the download. However, the number of failures tends to increase on larger blobs and often 3 failures is not enough to complete a multi-GB download if S3 is suffering from a cluster of failures as sometimes happens. Typically we make 10s-to-100s of MBs of progress between each failure even in this state.I think we should increase the tenacity of
S3RetryingInputStreamwhen downloading larger blobs. For instance, we could not count a partial download towards the retry limit if it makes significant progress before failing.The text was updated successfully, but these errors were encountered: