No retries on network timeouts S3 InputStream #856
Comments
|
This is probably not something we'd consider taking on until the next major version bump as it is a big departure from what we do today. The retry policy for all streaming operations do not apply while reading the content because we've passed control back to the caller already. Presumably we could retry transparently by capturing a reference to the client in a special input stream and on calls to read, catch the IO exception and make another ranged GET starting from the last successful byte. The transfer manager utility has some more robust retry and resume behavior, would that meet your needs for now? |
|
Hi @shorea,
Regards, |
|
Yeah I think it's definitely possible and makes a lot of sense to honor the retry policy even for streaming operations but I don't think we can add it to the SDK without a major version bump due to the performance implications. |
dagnir
added
feature-request
|
Using Using aws-java-sdk-s3:1.11.18 here. Introducing a new method that accepts a OutputStream would be so great, instead of File, as streaming is a much desired use-case -- Also, @shorea it'd be great if any retry examples, PR, existed before official rollout within SDK -- #893! I have all objects have stored using multi-part upload (5MB or greater partsize). |
|
@phraktle : Did you come up with a work-around? |
|
Is this lack of retries the cause of the error I have been getting very frequently while streaming data from S3 to an EC2 instance in a VPC? I really don't want to download these files (I don't want to deal with the disk at all -- and streaming seems like it ought to work). But the error rate when downloading files is increasing dramatically, and it's a big operational pain. The failure happens at random places in the files (when I retry, the same file will often fail again, but at a different place). Stack trace: |
|
Hi @OrigamiMarie, sorry to hear you're having issues. We do have #893 in our backlog, which is to allow downloading to a |
|
If anyone ever does add transparent retries to failures in input stream reads, can I, as a representative of the Hadoop team who maintain the S3A connector, have a way to turn this off? Because we do our own reconnect logic and think we've got it under control (now), and having something underneath trying to be helpful might be a regression. Happy to discuss what could be done here, including what exceptions should be treated as recoverable... |
|
There's a similar problem when the underlying s3 client fails the @dagnir Is there any workaround other than catching the exceptions from the TransferManager and retrying the whole download? |
|
V2 supports retrying streaming operations using the retry policy, if the proper API is used. We do not intend to make this change in 1.11.x. |
|
@millems Can you give an example or link to some documentation on how to this properly with V2? The |
|
@electrum In 2.x, you can use the response transformer abstraction to allow retrying failures that occur while reading the response: s3.getObject(r -> r.bucket("bucket").key("key"), (response, inputStream) -> {
try {
// Do something with the stream.
IoUtils.copy(inputStream, System.out);
return null;
} catch (IOException e) {
throw RetryableException.create("Failed to read from input stream.", e);
}
});Note that the response transformer can be called multiple times, once for each retry. It's a new input stream each time, so it will start back at the beginning of the object. |
It appears that there are no retries attempted when there's a network timeout on the underlying HTTP connection while reading the
InputStreamfromS3Object#getObjectContent. It should instead transparently reconnect (as per the retry policy) and continue from the last byte's position.Stack trace
The text was updated successfully, but these errors were encountered: