New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Streaming HTTP requests retried after sending data #2311
Comments
It seems as if this is caused by the resetting of the body here:
That is triggered before the retry logic, and thus the retry-logic, which is already actively checking whether the response body is truncatable, doesn't actually see the real body, and is therefore unable to prevent retries. This overriding of the response body seems to have been introduced in ba932c1, but there doesn't seem to be any motivation for the response body change in that commit, as the issue pertains to handling error responses. In #1877 the behavior was fixed for one corner-case, but this issue indicates it needs to be fixed for other cases as well. Perhaps it should be disabled altogether? The only other option I can think of would be to introduce separate replacement bodies depending on the type of body issued, or a generic wrapper, but neither option really seems reasonable. |
Hey @grddev thank-you for reaching out to us with your issue, I was able to reproduce it, I am not sure if this will be prioritized until next week to be investigated and hopefully fixed over the other issues. We would appreciate if you'd like to contribute and open a PR for us to review as well. |
I've been looking into this and reviewing (and updating) an existing PR that we have open: #1617 that may (mostly) address this. Our documentation for streaming blocks currently mentions: #1617 attempts to fix this by keeping track of the bytes that have been sent to the body IO already and not re-sending them. However, this does not work for the S3 Encryption::Client (as the number of unencrypted vs encrypted bytes will differ) and the block will still be called multiple times with the same data. |
Merged #1617 which addresses this. |
Thank you for looking into this @alextwoods. I was surprised that there was a PR given that I searched the issues and didn't find anything, but this isn't the first time I'm fooled by the separation of issues and PR:s. Sorry for not reaching out earlier, but the current solution looks reasonable to me. |
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug
If a networking error is raised during processing of a streaming response, such as an S3 download, the request is retried. This is problematic, as the block received the same first chunk of the file three times, without any indication that the download was effectively restarted. I fixed this behavior for aws-sdk-v1 back in the day, but it seems the fix wasn't carried over to aws-sdk-v3.
Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version
aws-sdk-core (3.82.0)
aws-sdk-s3 (1.57.0)
Version of Ruby, OS environment
-paste the output of ruby -v
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-darwin18]
To Reproduce (observed behavior)
In order to reproduce the error in IRB, I simulate a socket error by raising it from the processing callback, and in the debug output, you can see that the request was retried three times.
Technically, the fact that I can simulate this by raising from within the block is a bug in itself, as the net-http-component shouldn't pick up a socket error from within the callback and translate into retryable networking error, as this is a completely unrelated error.
Expected behavior
The download error should not be retried after the callback block has been invoked, or there needs to be a separate callback for errors, or the callback needs to be supplied another argument that would indicate a restart.
The text was updated successfully, but these errors were encountered: