uploading file to s3 results in error 104 (connection reset) #2207

Open
mjpan opened this Issue Apr 8, 2014 · 102 comments

Comments

Projects
None yet
@mjpan

mjpan commented Apr 8, 2014

this is on boto 2.27.0. uploading a 14 byte file works, but 512k file causes this error.
using aws cli to upload works on the 512k file

File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/s3/key.py", line 1315, in set_contents_from_filename
encrypt_key=encrypt_key)
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/s3/key.py", line 1246, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/s3/key.py", line 725, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/s3/key.py", line 914, in _send_file_internal
query_args=query_args
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/s3/connection.py", line 633, in make_request
retry_handler=retry_handler
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/connection.py", line 1046, in make_request
retry_handler=retry_handler)
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/connection.py", line 919, in _mexe
request.body, request.headers)
File "/usr/local/lib/python2.7/dist-packages/boto-2.27.0-py2.7.egg/boto/s3/key.py", line 815, in sender
http_conn.send(chunk)
File "/usr/lib/python2.7/httplib.py", line 794, in send
self.sock.sendall(data)
File "/usr/lib/python2.7/ssl.py", line 229, in sendall
v = self.send(data[count:])
File "/usr/lib/python2.7/ssl.py", line 198, in send
v = self._sslobj.write(data)
error: [Errno 104] Connection reset by peer

@fbeister

This comment has been minimized.

Show comment
Hide comment
@fbeister

fbeister May 8, 2014

Just wanted to say "me too". I use boto through duplicity to access S3. On another machine, I have boto 2.2.2 and it works there. After downgrading boto from 2.27 to 2.2.2, the error persists, so I suspect the problem to lie either in ssl.py or the OS (Ubuntu Linux 14.04)

fbeister commented May 8, 2014

Just wanted to say "me too". I use boto through duplicity to access S3. On another machine, I have boto 2.2.2 and it works there. After downgrading boto from 2.27 to 2.2.2, the error persists, so I suspect the problem to lie either in ssl.py or the OS (Ubuntu Linux 14.04)

@fbeister

This comment has been minimized.

Show comment
Hide comment
@fbeister

fbeister May 8, 2014

Found a solution (at least for duplicity):
aws/aws-cli#634

instead of "s3+http://bucketname" use "s3:///s3.amazonaws.com/bucketname"

fbeister commented May 8, 2014

Found a solution (at least for duplicity):
aws/aws-cli#634

instead of "s3+http://bucketname" use "s3:///s3.amazonaws.com/bucketname"

@danielgtaylor

This comment has been minimized.

Show comment
Hide comment
@danielgtaylor

danielgtaylor May 28, 2014

Member

From the CLI issue:

I'm reopening this issue. After debugging this I can confirm what others have said. This problem exists when trying to upload a large file to a newly created bucket that's not in the classic region.

From what I can tell the CLI is properly retrying requests and following 307 redirects. The problem, however, is that the CLI sends the entire request and then waits for a response. However, S3 will immediately send the 307 response before we've finished sending the body. Eventually it will just close the connection, and if we're still in the process of streaming the body, we will not see the response. Instead we get the ConnectionError as shown in the various debug logs above.

The normal way to address this would be to use the expect 100 continue header. However, the HTTP client we use (requests) does not support this. There might be a way to work around this, but I'll need to do some digging into the requests library to see the best way to fix this issue.

This will require adding some support for currently unsupported HTTP headers.

Member

danielgtaylor commented May 28, 2014

From the CLI issue:

I'm reopening this issue. After debugging this I can confirm what others have said. This problem exists when trying to upload a large file to a newly created bucket that's not in the classic region.

From what I can tell the CLI is properly retrying requests and following 307 redirects. The problem, however, is that the CLI sends the entire request and then waits for a response. However, S3 will immediately send the 307 response before we've finished sending the body. Eventually it will just close the connection, and if we're still in the process of streaming the body, we will not see the response. Instead we get the ConnectionError as shown in the various debug logs above.

The normal way to address this would be to use the expect 100 continue header. However, the HTTP client we use (requests) does not support this. There might be a way to work around this, but I'll need to do some digging into the requests library to see the best way to fix this issue.

This will require adding some support for currently unsupported HTTP headers.

@avandendorpe

This comment has been minimized.

Show comment
Hide comment
@avandendorpe

avandendorpe Jul 30, 2014

I think the fix for awscli addresses this as it was merged in botocore. @jamesls - is that correct?

I think the fix for awscli addresses this as it was merged in botocore. @jamesls - is that correct?

@danielgtaylor

This comment has been minimized.

Show comment
Hide comment
@danielgtaylor

danielgtaylor Aug 7, 2014

Member

The fix for Botocore was in boto/botocore@9e59c4e and seems to have solved the issue.

We need to backport it to Boto. I'll see about trying to get it into the next release.

Member

danielgtaylor commented Aug 7, 2014

The fix for Botocore was in boto/botocore@9e59c4e and seems to have solved the issue.

We need to backport it to Boto. I'll see about trying to get it into the next release.

@j0hnsmith

This comment has been minimized.

Show comment
Hide comment
@j0hnsmith

j0hnsmith Oct 2, 2014

I'm seeing a lot of these errors, please backport the fix asap.

I'm seeing a lot of these errors, please backport the fix asap.

@unwitting

This comment has been minimized.

Show comment
Hide comment
@unwitting

unwitting Oct 4, 2014

@danielgtaylor any update on getting it ported back into boto?

@danielgtaylor any update on getting it ported back into boto?

@jvantuyl

This comment has been minimized.

Show comment
Hide comment

+1

@anna-buttfield-sirca

This comment has been minimized.

Show comment
Hide comment
@anna-buttfield-sirca

anna-buttfield-sirca Oct 27, 2014

This issue hit us too. The workaround that worked for us was explicitly connecting to the bucket's region. Because we can't guarantee which region buckets are in we use this code:

conn = boto.connect_s3()
bucket = conn.get_bucket(bucket_name)
bucket_location = bucket.get_location()
if bucket_location:
    conn = boto.s3.connect_to_region(bucket_location)
    bucket = conn.get_bucket(bucket_name)

This issue hit us too. The workaround that worked for us was explicitly connecting to the bucket's region. Because we can't guarantee which region buckets are in we use this code:

conn = boto.connect_s3()
bucket = conn.get_bucket(bucket_name)
bucket_location = bucket.get_location()
if bucket_location:
    conn = boto.s3.connect_to_region(bucket_location)
    bucket = conn.get_bucket(bucket_name)
@charles-vdulac

This comment has been minimized.

Show comment
Hide comment
@pitsevich

This comment has been minimized.

Show comment
Hide comment

+1

@whitequark

This comment has been minimized.

Show comment
Hide comment

+1

@dwijnand

This comment has been minimized.

Show comment
Hide comment

👍

@eberle1080

This comment has been minimized.

Show comment
Hide comment

+1

@wengole

This comment has been minimized.

Show comment
Hide comment
@wengole

wengole Nov 21, 2014

This doesn't quite work as regions and locations have different names, however the concept does help (connecting to the correct region)

wengole commented Nov 21, 2014

This doesn't quite work as regions and locations have different names, however the concept does help (connecting to the correct region)

@fangpenlin

This comment has been minimized.

Show comment
Hide comment

+1

@volnt

This comment has been minimized.

Show comment
Hide comment

volnt commented Jan 5, 2015

+1

@hwkns

This comment has been minimized.

Show comment
Hide comment

hwkns commented Jan 9, 2015

+1

@antonagestam

This comment has been minimized.

Show comment
Hide comment
@antonagestam

antonagestam Jan 10, 2015

Come on ppl, commenting +1 isn't helping anyone.

Come on ppl, commenting +1 isn't helping anyone.

@hwkns

This comment has been minimized.

Show comment
Hide comment
@hwkns

hwkns Jan 10, 2015

@antonagestam, my intent was to help provide a sense of the number of people affected by this bug, which should hopefully lend it some priority. If there's something more I can do to help, please let me know. This is a sponsored project now, and this issue has been around for at least 9 months already, affecting quite a few users. I'm just trying not to let it languish.

hwkns commented Jan 10, 2015

@antonagestam, my intent was to help provide a sense of the number of people affected by this bug, which should hopefully lend it some priority. If there's something more I can do to help, please let me know. This is a sponsored project now, and this issue has been around for at least 9 months already, affecting quite a few users. I'm just trying not to let it languish.

@hellwolf

This comment has been minimized.

Show comment
Hide comment

+1

@ledmonster

This comment has been minimized.

Show comment
Hide comment

+1

@spikewilliams

This comment has been minimized.

Show comment
Hide comment
@spikewilliams

spikewilliams Jan 24, 2015

I am hitting this bug. Just as someone referenced above, I am also on Ubuntu Linux 14.04.

I am hitting this bug. Just as someone referenced above, I am also on Ubuntu Linux 14.04.

@matiasinsaurralde

This comment has been minimized.

Show comment
Hide comment
@sebastibe

This comment has been minimized.

Show comment
Hide comment

+1

@vperron

This comment has been minimized.

Show comment
Hide comment

vperron commented Feb 11, 2015

+1

@wvengen

This comment has been minimized.

Show comment
Hide comment

wvengen commented Feb 18, 2015

+1

@alfetopito

This comment has been minimized.

Show comment
Hide comment
@alfetopito

alfetopito Feb 18, 2015

If it helps anything, +1
(Ubuntu 14.04 x64 on EC2, Python 3.4.2, boto==2.36.0)

If it helps anything, +1
(Ubuntu 14.04 x64 on EC2, Python 3.4.2, boto==2.36.0)

@cy-yoshikazu-umino

This comment has been minimized.

Show comment
Hide comment
@nachobitpagos

This comment has been minimized.

Show comment
Hide comment
@nachobitpagos

nachobitpagos Feb 25, 2015

Hey @danielgtaylor any news on these?

Hey @danielgtaylor any news on these?

@kageurufu

This comment has been minimized.

Show comment
Hide comment

+1

@matisoffn

This comment has been minimized.

Show comment
Hide comment

+1

@broxtronix broxtronix referenced this issue in thunder-project/thunder Mar 5, 2015

Open

SSL times out pulling from S3 #125

@quentinsf

This comment has been minimized.

Show comment
Hide comment
@quentinsf

quentinsf Jul 12, 2016

DRazumovskiy's workaround of using the format:

s3://s3-eu-west-1.amazonaws.com/bucket

works for me on an EU bucket too - thanks!

DRazumovskiy's workaround of using the format:

s3://s3-eu-west-1.amazonaws.com/bucket

works for me on an EU bucket too - thanks!

@innoteam-rromano

This comment has been minimized.

Show comment
Hide comment
@innoteam-rromano

innoteam-rromano Jul 28, 2016

I had the same problem today, DRazumovskiy's workaround works for me, too.
Strange thing is I have another machine with exactly the same software versions installed and the same configuration that seems to work regularly (at least, this night's backup was correctly executed) so I really can't wonder where exactly the problem lies...

I had the same problem today, DRazumovskiy's workaround works for me, too.
Strange thing is I have another machine with exactly the same software versions installed and the same configuration that seems to work regularly (at least, this night's backup was correctly executed) so I really can't wonder where exactly the problem lies...

@ba1dr ba1dr referenced this issue in RaRe-Technologies/smart_open Aug 1, 2016

Closed

ConnectionResetError: [Errno 104] Connection reset by peer #86

@grahame grahame referenced this issue in okfn/ckanext-s3filestore Aug 8, 2016

Open

port to boto3 #11

@lgibelli

This comment has been minimized.

Show comment
Hide comment
@lgibelli

lgibelli Aug 18, 2016

+1, DRazumovskiy's workaround works for me, too.

+1, DRazumovskiy's workaround works for me, too.

@kisst

This comment has been minimized.

Show comment
Hide comment

kisst commented Sep 8, 2016

+1

@jobcpf

This comment has been minimized.

Show comment
Hide comment
@jobcpf

jobcpf Sep 9, 2016

+1 for DRazumovskiy's / quentinsf workaround thank you

jobcpf commented Sep 9, 2016

+1 for DRazumovskiy's / quentinsf workaround thank you

@jameshiew jameshiew referenced this issue in django-compressor/django-compressor Nov 28, 2016

Closed

Support for django-storages S3Boto3Storage backend? #808

bacongobbler added a commit to bacongobbler/postgres that referenced this issue Nov 30, 2016

fix(setup-envdir): convert AWS_REGION to WALE_S3_ENDPOINT
Converting $AWS_REGION into $WALE_S3_ENDPOINT avoids the
"Connection reset by peer" error when uploading files to buckets
creted in regions other than us-east-1.

See boto/boto#2207
@wimoMisterX

This comment has been minimized.

Show comment
Hide comment

+1

@ycUygB1

This comment has been minimized.

Show comment
Hide comment
@ycUygB1

ycUygB1 Feb 13, 2017

So like after three years Amazon does not care that you cannot copy data to S3 in Europe? Works for me with US Standard. Time to seriously look at Google's offering if this is how little Amazon cares.

ycUygB1 commented Feb 13, 2017

So like after three years Amazon does not care that you cannot copy data to S3 in Europe? Works for me with US Standard. Time to seriously look at Google's offering if this is how little Amazon cares.

@slavpetroff

This comment has been minimized.

Show comment
Hide comment
@slavpetroff

slavpetroff Mar 25, 2017

Hello guys! The solution for me was first, to change S3 Location from Frankfurt to London. Then additionally i've added AWS_S3_HOST = 's3-eu-west-2.amazonaws.com' to the settings files and this solved my problem. Hope this helps!

Hello guys! The solution for me was first, to change S3 Location from Frankfurt to London. Then additionally i've added AWS_S3_HOST = 's3-eu-west-2.amazonaws.com' to the settings files and this solved my problem. Hope this helps!

@dvl

This comment has been minimized.

Show comment
Hide comment
@dvl

dvl Mar 27, 2017

@ilmesi solution from 2 years ago still working on latest boto version... so next time I found this issue on Google I know what to do.

dvl commented Mar 27, 2017

@ilmesi solution from 2 years ago still working on latest boto version... so next time I found this issue on Google I know what to do.

adrpar pushed a commit to adrpar/incubator-airflow that referenced this issue Apr 25, 2017

Adrian Partl
[AIRFLOW-1023] reconnect to S3 bucket location
This fixes an issue with the S3 hook when uploading local files to a
S3 bucket that does not reside in the standard US location (e.g.
eu-west-1).

According to

boto/boto#2207

this is an issue in the boto library and does not seem to get fixed
anytime soon.

A work around is proposed in the boto issue and is implemented here.

adrpar pushed a commit to adrpar/incubator-airflow that referenced this issue May 18, 2017

Adrian Partl
[AIRFLOW-1023] reconnect to S3 bucket location
This fixes an issue with the S3 hook when uploading local files to a
S3 bucket that does not reside in the standard US location (e.g.
eu-west-1).

According to

boto/boto#2207

this is an issue in the boto library and does not seem to get fixed
anytime soon.

A work around is proposed in the boto issue and is implemented here.

test

liamim added a commit to liamim/NEXT that referenced this issue Jun 12, 2017

@jcampbell05

This comment has been minimized.

Show comment
Hide comment
@jcampbell05

jcampbell05 Jul 10, 2017

:) 2017 and this still needed to be used as a workaround.

:) 2017 and this still needed to be used as a workaround.

@karthikvadla

This comment has been minimized.

Show comment
Hide comment
@karthikvadla

karthikvadla Jul 15, 2017

@jcampbell05 I see the same error

error: [Errno 104] Connection reset by peer

i'm trying to upload files from my local machine to s3 using boto.
I'm using set_contents_from_filename(filename). It's failing for data which is morethan 5GB.

did you use upload_file from boto3. What workaround did you use.?

@jcampbell05 I see the same error

error: [Errno 104] Connection reset by peer

i'm trying to upload files from my local machine to s3 using boto.
I'm using set_contents_from_filename(filename). It's failing for data which is morethan 5GB.

did you use upload_file from boto3. What workaround did you use.?

@jcampbell05

This comment has been minimized.

Show comment
Hide comment
@jcampbell05

jcampbell05 Jul 16, 2017

Rolling back my entire server to an older OS and version of boto. Not really a long term viable solution.

Rolling back my entire server to an older OS and version of boto. Not really a long term viable solution.

@shanx

This comment has been minimized.

Show comment
Hide comment
@shanx

shanx Jul 19, 2017

If you want to transfer files > 5gb you need to use multipart uploading (unfortunately boto2 doesn't abstract this away from us). I've done an implementation for this using the example from: http://boto.cloudhackers.com/en/latest/s3_tut.html#storing-large-data in the bakthat project (a project similar to duplicity) see my pull request tsileo/bakthat#84

shanx commented Jul 19, 2017

If you want to transfer files > 5gb you need to use multipart uploading (unfortunately boto2 doesn't abstract this away from us). I've done an implementation for this using the example from: http://boto.cloudhackers.com/en/latest/s3_tut.html#storing-large-data in the bakthat project (a project similar to duplicity) see my pull request tsileo/bakthat#84

@karthikvadla

This comment has been minimized.

Show comment
Hide comment
@karthikvadla

karthikvadla Jul 19, 2017

@jcampbell05 @shanx : Thank you for your responses. I migrated from boto2.x to boto3(which has in-built support for mutlipart file upload with upload_file feature. ). I followed this.
http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.upload_file .
Which made my work simple and easier. And i tested, it works fine. :)

@jcampbell05 @shanx : Thank you for your responses. I migrated from boto2.x to boto3(which has in-built support for mutlipart file upload with upload_file feature. ). I followed this.
http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.upload_file .
Which made my work simple and easier. And i tested, it works fine. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment