Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resume partial download from S3 on connection drop #46589

Conversation

DaveCTurner
Copy link
Contributor

Today if the connection to S3 times out or drops after starting to download an
object then the SDK does not attempt to recover or resume the download, causing
the restore of the whole shard to fail and retry. This commit allows
Elasticsearch to detect such a mid-stream failure and to resume the download
from where it failed.

Today if the connection to S3 times out or drops after starting to download an
object then the SDK does not attempt to recover or resume the download, causing
the restore of the whole shard to fail and retry. This commit allows
Elasticsearch to detect such a mid-stream failure and to resume the download
from where it failed.
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@DaveCTurner
Copy link
Contributor Author

The retry behaviour is perhaps not ideal here - we only fail completely after maxRetries consecutive failures to make any progress at all. It also doesn't try and interact with the retry tracking within the SDK. Thus if we're seeing lots of different kinds of failures happening all at the same time we might retry a lot more than maxRetries times.

I'm also not sure about trying to handle the case where a retrying openStream() throws an exception, leaving S3RetryingInputStream#currentStream set to the old (failed and closed) stream. I don't think that's so bad, we shouldn't be using it after such an exception, but your thoughts are welcome.

Copy link
Member

@original-brownbear original-brownbear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @DaveCTurner good stuff :)

@original-brownbear
Copy link
Member

I'm also not sure about trying to handle the case where a retrying openStream() throws an exception, leaving S3RetryingInputStream#currentStream set to the old (failed and closed) stream. I don't think that's so bad, we shouldn't be using it after such an exception, but your thoughts are welcome.

I think this is fine. We're not using these streams in any tricky-async way anyway so if we run into an Exception it's always inside a try-with-resources I think, so no point in doing extra work here imo.

Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very nice, thanks @DaveCTurner. I left a comment on ensuring that a closed S3 stream will prevent more reads.

public void testReadNonexistentBlobThrowsNoSuchFileException() {
final BlobContainer blobContainer = createBlobContainer(between(1, 5), null, null, null);
final Exception exception = expectThrows(NoSuchFileException.class, () -> blobContainer.readBlob("read_nonexistent_blob"));
assertThat(exception.getMessage().toLowerCase(Locale.ROOT),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can fit on the same line

@DaveCTurner
Copy link
Contributor Author

Thanks @tlrx and @original-brownbear. I thought about this overnight and decided it'd be safer to limit the number of retries per blob rather than per read() call, see efb2422.

I also think it might be useful to collect the exceptions that result in retries and add them as suppressed exceptions in case we hit the limit and fail the download. WDYT?

@tlrx
Copy link
Member

tlrx commented Sep 12, 2019

I thought about this overnight and decided it'd be safer to limit the number of retries per blob rather than per read()

I agree

I also think it might be useful to collect the exceptions that result in retries and add them as suppressed exceptions in case we hit the limit and fail the download. WDYT?

I also thought about it when reviewing the PR. I think we can add them as suppressed exceptions but maybe just limit their numbers in case the max retries is set to a large value?

@DaveCTurner
Copy link
Contributor Author

Yes, a bound is a good idea. I added suppressed exceptions in 22f5703.

Copy link
Member

@tlrx tlrx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks David!

/**
* Wrapper around an S3 object that will retry the {@link GetObjectRequest} if the download fails part-way through, resuming from where
* the failure occurred. This should be handled by the SDK but it isn't today. This should be revisited in the future (e.g. before removing
* the {@link Version#V_7_0_0} version constant) and removed when the SDK handles retries itself.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, otherwise we'll adding retries over retries

@original-brownbear
Copy link
Member

original-brownbear commented Sep 12, 2019

@DaveCTurner @tlrx

I thought about this overnight and decided it'd be safer to limit the number of retries per blob rather than per read()

I'm not sure sure about this. This has the strange side effect that once again larger blobs are more likely to fail downloading which is exactly what motivated this in the first place.

What do you think is the safety issue here?
I don't think it's mathematically possible (likely enough) to run into something like thousands of retries here? The only case I could see would be some bug in S3 that forces retrying every chunk once (to fix some cache or whatever) exactly once. If it's not once per chunk but some larger number of retries with every chunk failing at a rate x, then you will quickly exceed your retry count on one chunk anyway? (tl;dr ... I don't see what distribution of exceptions could realistically lead to retrying tousands of times per blob but never exceed the limit of retries on a single chunk here).

But if you're still worried after that argument, could we make the retry count proportional to the size of the blob (could make it proportional to the blob's size divided by the upload chunk size we used for it ... it may have changed since the snapshot was taken but it's a good enough estimate IMO)?

@DaveCTurner
Copy link
Contributor Author

What do you think is the safety issue here?

I am thinking of something like a badly-configured security device that drops connections it believes to have downloaded an unreasonable amount of data like, say, 1MB. In the presence of such a device I don't think we should push on through and download each object in 1MB chunks - this would result in a rather surprising bill at the end of the month!

By default each blob has bounded size (1GB) which bounds the probability of it failing outright. I think the default of 3 retries for each chunk is ample.

Copy link
Member

@original-brownbear original-brownbear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am thinking of something like a badly-configured security device that drops connections it believes to have downloaded an unreasonable amount of data like, say, 1MB. In the presence of such a device I don't think we should push on through and download each object in 1MB chunks - this would result in a rather surprising bill at the end of the month!

Assuming we want to protect people from this LGTM :)

In practice this fixes our issues anyway I think, given the low rate of failure we were observing. Sorry for the noise :)

}

@Override
public synchronized void reset() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to make this synchronized

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, not sure where that came from. Fixed in 3f8c20e.


/**
* Wrapper around an S3 object that will retry the {@link GetObjectRequest} if the download fails part-way through, resuming from where
* the failure occurred. This should be handled by the SDK but it isn't today. This should be revisited in the future (e.g. before removing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you link to the corresponding open AWS SDK issue? i.e. aws/aws-sdk-java#856

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, done in 3f8c20e. I am not convinced that that's the whole issue, because the problem we were chasing was to do with S3 actively closing the connection rather than a network timeout, but there doesn't seem to be an issue for that.

@DaveCTurner DaveCTurner merged commit d391446 into elastic:master Sep 17, 2019
@DaveCTurner DaveCTurner deleted the 2019-09-11-retry-s3-download-on-partial-content branch September 17, 2019 12:10
DaveCTurner added a commit that referenced this pull request Sep 17, 2019
Today if the connection to S3 times out or drops after starting to download an
object then the SDK does not attempt to recover or resume the download, causing
the restore of the whole shard to fail and retry. This commit allows
Elasticsearch to detect such a mid-stream failure and to resume the download
from where it failed.
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Feb 18, 2020
Exactly as elastic#46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes elastic#52319
original-brownbear added a commit that referenced this pull request Feb 19, 2020
* Add Blob Download Retries to GCS Repository

Exactly as #46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes #52319
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Feb 19, 2020
* Add Blob Download Retries to GCS Repository

Exactly as elastic#46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes elastic#52319
sbourke pushed a commit to sbourke/elasticsearch that referenced this pull request Feb 19, 2020
* Add Blob Download Retries to GCS Repository

Exactly as elastic#46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes elastic#52319
original-brownbear added a commit that referenced this pull request Feb 19, 2020
* Add Blob Download Retries to GCS Repository

Exactly as #46589 (and kept as close to it as possible code wise so we can dry things up in a follow-up potentially) but for GCS.

Closes #52319
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement v7.5.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants