Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Got "read: connection reset by peer" error when using version 1.0.0+ #294

Closed
kevin-wyx opened this issue Jun 7, 2021 · 6 comments · Fixed by #298
Closed

Got "read: connection reset by peer" error when using version 1.0.0+ #294

kevin-wyx opened this issue Jun 7, 2021 · 6 comments · Fixed by #298
Labels
non-aws Issue related to other S3-Compatible Services regression

Comments

@kevin-wyx
Copy link

Tested on OpenStack Swift using s5cmd version 1.0.0 and 1.2.1 with the following command:

s5cmd --endpoint-url http://objstor.testserver.com --no-verify-ssl  cp -n -s -u  aaaatest2 s3://aaaatest01/aaaajfg006/

And got errors like this:

ERROR "cp aaaatest2/2ms/2m_file_1 s3://aaaatest01/aaaajfg006/aaaatest2/2ms/2m_file_1": RequestError: send request failed caused by: Head "http://objstor.testserver.com/aaaatest01/aaaajfg006/aaaatest2/2ms/2m_file_1": read tcp xxx.xxx.xxx.xxx:38924->xxx.xxx.xxx.xxx:80: read: connection reset by peer

However, it's all good using s5cmd v0.7.0 with the same command. No error returned at all.

I aslo found an issue on aws-sdk-go repo (see below) complainting about the same error.

aws/aws-sdk-go#3027

Is it possible that it's the new version of aws-sdk-go causes the problem?
@nelhage
Copy link
Contributor

nelhage commented Jun 12, 2021

At a minimum it seems like the s5cmd custom retryer should probably consider read: connection reset as retryable. Per the linked issue the SDK doesn't do so by default, but for most or all S3 operations we can safely retry.

@nelhage nelhage mentioned this issue Jun 14, 2021
Merged
@igungor
Copy link
Member

igungor commented Jun 16, 2021

I'm surprised to see that this error is not retried for you. It's been fixed before #225

After the AWS SDK started to retry connection reset errors, we removed the checks but kept the relevant tests that verify "connection reset by peer" errors are retried. 6abfea8

I'll merge @nelhage's PR now.

igungor pushed a commit that referenced this issue Jun 16, 2021
@nelhage
s3: treat "connection reset" errors as retryable. (#298)
688f8e5 
Per
aws/aws-sdk-go#2926 (comment),
the upstream SDK doesn't retry connection reset errors on reads,
because at that point we can't tell if the server has already applied
the operation.

For S3 operations, I think pretty much all requests should be safely
idempotent, so I think we should retry these errors.

I saw some of my own long-running jobs fail this weekend due to
network weather and some dropped connections that s5cmd did not retry,
which I found very surprising.

Per
aws/aws-sdk-go@c3d2710,
the `strings.Contains` check does seem to be the best way to check for
this error.

Fixes #294
@igungor
Copy link
Member

igungor commented Jun 16, 2021

@kevin-wyx could you test latest commit please?

@brtasavpatel
Copy link

I am also getting this error on v1.3.0

@brtasavpatel
Copy link

I am still getting this error on s5cmd cat and it doesn't seem to retry.

@pbelmann
Copy link

pbelmann commented Jun 29, 2022 •
edited
Loading

I am also getting the same problem with s5cmd version "v2.0.0-beta.2-c978e4a"

@boraberke boraberke added non-aws Issue related to other S3-Compatible Services regression labels Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
non-aws Issue related to other S3-Compatible Services regression
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Treat "connection reset" errors as retryable. nelhage/s5cmd
6 participants
@igungor @nelhage @pbelmann @kevin-wyx @brtasavpatel @boraberke