New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws s3 cp hangs on download more than 500MB on content #1775

Open
neoacevedo opened this Issue Feb 5, 2016 · 31 comments

Comments

Projects
None yet
@neoacevedo

neoacevedo commented Feb 5, 2016

Hello,

I am just doing daily backups from s3 files. Bassically I copy from a s3 bucket files to local disk to zip them and copy after to another s3 bucket the zipped contain.

The issue is the aws s3 get stuck in any moment and doesn't download, I mean, the download starts but is interrupted in an specific moment in an specific file. This file can be accessed via web or any s3 explorer. the files are not big.

My aws cli is:

aws-cli/1.9.18 Python/2.7.10 Linux/4.1.13-19.31.amzn1.x86_64 botocore/1.3.18

OK I will change the issue, aws s3 cp is hanging when is downloading a lot of files. I checked with 2 s3 buckets with no long content and it downloads well but not the same when the bucket has a lot of files:

S3 bucket content between 200 and 500MB on storage are downloaded without problems but not the same when content is beyond 500MB.

Regards

@neoacevedo neoacevedo changed the title from aws s3 cp stuck on download to aws s3 cp hangs on download more than 500MB on content Feb 15, 2016

@jamesls

This comment has been minimized.

Show comment
Hide comment
@jamesls

jamesls Feb 15, 2016

Member

We'll need more information to help you troubleshoot. Can you tell us:

  • The exact AWS commands you're running? If you can give us a script/set of commands to run that demonstrate the issue, that will help in getting this resolved.
  • When the command hangs is there any network activity? i.e does the download speed drastically drop or does it completely stop transferring data entirely?
  • Can you provide --debug logs? We're specifically looking for anything with ERROR that will indicate any errors that occurred during transfer. This will give us an idea of where to start looking.
Member

jamesls commented Feb 15, 2016

We'll need more information to help you troubleshoot. Can you tell us:

  • The exact AWS commands you're running? If you can give us a script/set of commands to run that demonstrate the issue, that will help in getting this resolved.
  • When the command hangs is there any network activity? i.e does the download speed drastically drop or does it completely stop transferring data entirely?
  • Can you provide --debug logs? We're specifically looking for anything with ERROR that will indicate any errors that occurred during transfer. This will give us an idea of where to start looking.
@neoacevedo

This comment has been minimized.

Show comment
Hide comment
@neoacevedo

neoacevedo Feb 16, 2016

Hello, I run a simple command:
aws s3 cp s3://my-bucket/ my-local-directory/ --recursive

I attach the log file. the last one line shows that has downloaded 372 parts and yes, it stop totally any data transfer and no shows any network activity after that.

In the images at 15:30 UTC begun the data transfer and 1 minute after it were totally stopped:

instantanea19
instantanea20

60anios.log.zip

neoacevedo commented Feb 16, 2016

Hello, I run a simple command:
aws s3 cp s3://my-bucket/ my-local-directory/ --recursive

I attach the log file. the last one line shows that has downloaded 372 parts and yes, it stop totally any data transfer and no shows any network activity after that.

In the images at 15:30 UTC begun the data transfer and 1 minute after it were totally stopped:

instantanea19
instantanea20

60anios.log.zip

@neoacevedo

This comment has been minimized.

Show comment
Hide comment
@neoacevedo

neoacevedo Feb 19, 2016

Hello, I have found some interesting. If in my config file I put
s3 = multipart_chunksize = XXMB
it causes the s3 cp hanging. But according with aws documentation the multipart chunksize is only used if each file size is greater than default/configured multipart threshold value what is by default 8MB and no many files that I'm downloading have that size. In general are JPG files with no more than 1MB and the hanging occurs for the log file sent in a file with 520KB.

neoacevedo commented Feb 19, 2016

Hello, I have found some interesting. If in my config file I put
s3 = multipart_chunksize = XXMB
it causes the s3 cp hanging. But according with aws documentation the multipart chunksize is only used if each file size is greater than default/configured multipart threshold value what is by default 8MB and no many files that I'm downloading have that size. In general are JPG files with no more than 1MB and the hanging occurs for the log file sent in a file with 520KB.

@mohamed-ezz

This comment has been minimized.

Show comment
Hide comment
@mohamed-ezz

mohamed-ezz Feb 26, 2016

I'm having the same problem with both cp --recursive and sync, with this version:
aws-cli/1.10.8 Python/2.7.6 Linux/3.16.0-41-generic botocore/1.3.30
I also tried the ubuntu package version, and the same problem :
aws-cli/1.2.9 Python/3.4.3 Linux/3.16.0-41-generic

It always hangs towards the end, not at the beginning, which makes it more annoying. For example this is the last message I see :
Completed 6343 of 6349 part(s) with 6 file(s) remaining

I run a very basic command :
aws s3 sync s3://bucketname/dir /tmp/localdestination
or
aws s3 cp --recursive s3://bucketname/dir /tmp/localdestination

This happens too when the content has as few as 100 files, each of size 2MB.
When I try to interrupt the script with CTRL+C, the "Cleaning up. Please wait..." phase also takes a long time, so I hit CTRL+C a second time to exit.

mohamed-ezz commented Feb 26, 2016

I'm having the same problem with both cp --recursive and sync, with this version:
aws-cli/1.10.8 Python/2.7.6 Linux/3.16.0-41-generic botocore/1.3.30
I also tried the ubuntu package version, and the same problem :
aws-cli/1.2.9 Python/3.4.3 Linux/3.16.0-41-generic

It always hangs towards the end, not at the beginning, which makes it more annoying. For example this is the last message I see :
Completed 6343 of 6349 part(s) with 6 file(s) remaining

I run a very basic command :
aws s3 sync s3://bucketname/dir /tmp/localdestination
or
aws s3 cp --recursive s3://bucketname/dir /tmp/localdestination

This happens too when the content has as few as 100 files, each of size 2MB.
When I try to interrupt the script with CTRL+C, the "Cleaning up. Please wait..." phase also takes a long time, so I hit CTRL+C a second time to exit.

@w32-blaster

This comment has been minimized.

Show comment
Hide comment
@w32-blaster

w32-blaster Apr 20, 2016

+1
I have the same issue.
The files are usually very small but the number of files is very high (complete ubuntu repository).
I tried multiple versions with the same result. Process simply hangs forever.

I personally use the aws s3 sync with no special settings.

w32-blaster commented Apr 20, 2016

+1
I have the same issue.
The files are usually very small but the number of files is very high (complete ubuntu repository).
I tried multiple versions with the same result. Process simply hangs forever.

I personally use the aws s3 sync with no special settings.

@sc932

This comment has been minimized.

Show comment
Hide comment
@sc932

sc932 Apr 24, 2016

+1
Same issue on ubuntu with the same setup.

sc932 commented Apr 24, 2016

+1
Same issue on ubuntu with the same setup.

@jamesls

This comment has been minimized.

Show comment
Hide comment
@jamesls

jamesls May 24, 2016

Member

It would be helpful to get --debug logs for anyone running into these issues.

I've been running tests in a loop that downloads ~400k files (500MB+) and I'm not seeing s3 cp hang.

For anyone that's able to generate a --debug log of this issue, it'd be helpful to search for Traceback and see any errors that are occurring. The only thing I can think of is there's some error being generated that the CLI is not handling. But I'd need --debug logs to confirm this.

Member

jamesls commented May 24, 2016

It would be helpful to get --debug logs for anyone running into these issues.

I've been running tests in a loop that downloads ~400k files (500MB+) and I'm not seeing s3 cp hang.

For anyone that's able to generate a --debug log of this issue, it'd be helpful to search for Traceback and see any errors that are occurring. The only thing I can think of is there's some error being generated that the CLI is not handling. But I'd need --debug logs to confirm this.

@w32-blaster

This comment has been minimized.

Show comment
Hide comment
@w32-blaster

w32-blaster May 25, 2016

I was able to reproduce the issue 10+ times by trying to sync the ubuntu trusty apt repository.
To reproduce, first:

  • sync the mirror with apr-mirror command
  • try to upload to a bucket using aws s3 sync command

w32-blaster commented May 25, 2016

I was able to reproduce the issue 10+ times by trying to sync the ubuntu trusty apt repository.
To reproduce, first:

  • sync the mirror with apr-mirror command
  • try to upload to a bucket using aws s3 sync command
@stephengibbons-examtime

This comment has been minimized.

Show comment
Hide comment
@stephengibbons-examtime

stephengibbons-examtime May 25, 2016

We are having the same issue so sync'ing between 2 S3 buckets. The sync runs for around 3 hours 15 mins then hangs for 9 hours then resumes and finishes successfully an hour later.

The logs file don't show any errors but you can see where the sync has got stuck due to the timestamps.

We are using the Ubuntu package version:
aws --version
aws-cli/1.2.9 Python/3.4.3 Linux/3.13.0-29-generic

stephengibbons-examtime commented May 25, 2016

We are having the same issue so sync'ing between 2 S3 buckets. The sync runs for around 3 hours 15 mins then hangs for 9 hours then resumes and finishes successfully an hour later.

The logs file don't show any errors but you can see where the sync has got stuck due to the timestamps.

We are using the Ubuntu package version:
aws --version
aws-cli/1.2.9 Python/3.4.3 Linux/3.13.0-29-generic

@cgltower

This comment has been minimized.

Show comment
Hide comment
@cgltower

cgltower Jun 23, 2016

Same issue for me with aws s3 sync and a 1GB zip file. Looks like it grabs about 800MB of it on download, then crashes with no network activity. It was on an ec2 ubuntu instance so eventually I couldnt ssh to it.

aws-cli/1.10.1 Python/3.5Linux ip-10-9-10-192 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 Ubuntu 16.04 LTS 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux.1+ Linux/4.4.0-21-generic botocore/1.3.23

Command was:
$ aws s3 sync s3://testbucket--ap-southeast-2-1111111 /root/s3 --delete --debug

see attached logs

awss3sync-debug.zip

thanks

cgltower commented Jun 23, 2016

Same issue for me with aws s3 sync and a 1GB zip file. Looks like it grabs about 800MB of it on download, then crashes with no network activity. It was on an ec2 ubuntu instance so eventually I couldnt ssh to it.

aws-cli/1.10.1 Python/3.5Linux ip-10-9-10-192 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 Ubuntu 16.04 LTS 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux.1+ Linux/4.4.0-21-generic botocore/1.3.23

Command was:
$ aws s3 sync s3://testbucket--ap-southeast-2-1111111 /root/s3 --delete --debug

see attached logs

awss3sync-debug.zip

thanks

@mjanson

This comment has been minimized.

Show comment
Hide comment
@mjanson

mjanson Jul 12, 2016

Are there any updates on this issue?

mjanson commented Jul 12, 2016

Are there any updates on this issue?

@awbacker

This comment has been minimized.

Show comment
Hide comment
@awbacker

awbacker Sep 1, 2016

Hope this keeps the issue open, since it still happens for me.

$> aws --version
aws-cli/1.10.18 Python/2.7.6 Linux/3.13.0-93-generic botocore/1.4.9

$> aws s3 sync s3://eu ./local --delete --exact-timestamps --profile=eu
... [sic] 296 files downloaded

$> aws s3 sync ./local/  s3://beijing --profile=china
... [sic] lots of files uploading ..
Completed 295 of 296 part(s) with 1 file(s) remaining

I've never been patient enough to wait it out, but I have let it go on for an hour or so. I always ctrl-c out of it and relaunch. Sometimes it shows more files synced, other times not.

Its not consistently doing it, it feels a little more random

awbacker commented Sep 1, 2016

Hope this keeps the issue open, since it still happens for me.

$> aws --version
aws-cli/1.10.18 Python/2.7.6 Linux/3.13.0-93-generic botocore/1.4.9

$> aws s3 sync s3://eu ./local --delete --exact-timestamps --profile=eu
... [sic] 296 files downloaded

$> aws s3 sync ./local/  s3://beijing --profile=china
... [sic] lots of files uploading ..
Completed 295 of 296 part(s) with 1 file(s) remaining

I've never been patient enough to wait it out, but I have let it go on for an hour or so. I always ctrl-c out of it and relaunch. Sometimes it shows more files synced, other times not.

Its not consistently doing it, it feels a little more random

@kayibal

This comment has been minimized.

Show comment
Hide comment
@kayibal

kayibal Sep 1, 2016

We are experiencing exactly the same problem.

kayibal commented Sep 1, 2016

We are experiencing exactly the same problem.

@paweljedrzejczyk

This comment has been minimized.

Show comment
Hide comment
@paweljedrzejczyk

paweljedrzejczyk Sep 6, 2016

👍 I am also experiencing this.

paweljedrzejczyk commented Sep 6, 2016

👍 I am also experiencing this.

@doender

This comment has been minimized.

Show comment
Hide comment
@doender

doender Sep 8, 2016

Me too, on around 280K files

doender commented Sep 8, 2016

Me too, on around 280K files

@chadq

This comment has been minimized.

Show comment
Hide comment
@chadq

chadq Sep 16, 2016

I am like 99% sure I am bumping into this as well.

chadq commented Sep 16, 2016

I am like 99% sure I am bumping into this as well.

@kayibal

This comment has been minimized.

Show comment
Hide comment
@kayibal

kayibal Sep 16, 2016

We were able wo work around this bug by zipping all files before uploading them

kayibal commented Sep 16, 2016

We were able wo work around this bug by zipping all files before uploading them

@hdeadman

This comment has been minimized.

Show comment
Hide comment
@hdeadman

hdeadman Oct 3, 2016

I am trying to copy a large file (1.5GB) from s3 to an ec2 instance and it seems like it kills the network during the copy. I lose ssh connection and have to restart the instance in order to get back in. The copy doesn't finish either even if it is running nohup (because network is not working). The files seem to get backed up fine to s3 from an instance in us-east-1a but I am trying to copy the files back to an instance in us-east-1d and it isn't working.

I am attaching a zip with debug from two copies with different versions of the aws cli (including 1.11.0). The command I am running is at the top of each log file.

restorefaildebug.zip

UPDATE: I was able to get mine working by running fewer concurrent requests:

aws configure set default.s3.max_concurrent_requests 4

The default is 10 and apparently that was too much for my micro instance.

hdeadman commented Oct 3, 2016

I am trying to copy a large file (1.5GB) from s3 to an ec2 instance and it seems like it kills the network during the copy. I lose ssh connection and have to restart the instance in order to get back in. The copy doesn't finish either even if it is running nohup (because network is not working). The files seem to get backed up fine to s3 from an instance in us-east-1a but I am trying to copy the files back to an instance in us-east-1d and it isn't working.

I am attaching a zip with debug from two copies with different versions of the aws cli (including 1.11.0). The command I am running is at the top of each log file.

restorefaildebug.zip

UPDATE: I was able to get mine working by running fewer concurrent requests:

aws configure set default.s3.max_concurrent_requests 4

The default is 10 and apparently that was too much for my micro instance.

@w32-blaster

This comment has been minimized.

Show comment
Hide comment
@w32-blaster

w32-blaster Oct 3, 2016

@hdeadman : From what you describe, doesn't seems like the same issue. Maybe you want to open another Github issue with your problem

w32-blaster commented Oct 3, 2016

@hdeadman : From what you describe, doesn't seems like the same issue. Maybe you want to open another Github issue with your problem

@Chris-Pr

This comment has been minimized.

Show comment
Hide comment
@Chris-Pr

Chris-Pr Oct 6, 2016

I have this same issue trying to sync a range of files from small to 1.5TB in size, and from 1 to 55 million in quantity.

I removed the ~/.aws/config

And it seems to have fixed it, by resetting back to defaults.

Chris-Pr commented Oct 6, 2016

I have this same issue trying to sync a range of files from small to 1.5TB in size, and from 1 to 55 million in quantity.

I removed the ~/.aws/config

And it seems to have fixed it, by resetting back to defaults.

@JasonKaz

This comment has been minimized.

Show comment
Hide comment
@JasonKaz

JasonKaz Oct 12, 2016

I'm also getting this issue when using sync (local to S3) on a Windows 8 machine using CLI 1.11.2, it also happened on 1.10.41. This has been happening with as little as 300 files that are only ~5mb in size total. It takes about 15 minutes to do the upload, says "Completed X parts with ... file(s) remaining", and then finishes after some random amount of time. Sometimes it finishes instantly (no problem), other times I have let it run for 16+ hours and it still hasn't exited. However the file upload part does seem to finish in about 15 minutes consistently since I can see the network traffic drop, but the process doesn't end after that.

This is a major roadblock for us since this is supposed to run every few hours and we can't guarantee that it will ever complete.

JasonKaz commented Oct 12, 2016

I'm also getting this issue when using sync (local to S3) on a Windows 8 machine using CLI 1.11.2, it also happened on 1.10.41. This has been happening with as little as 300 files that are only ~5mb in size total. It takes about 15 minutes to do the upload, says "Completed X parts with ... file(s) remaining", and then finishes after some random amount of time. Sometimes it finishes instantly (no problem), other times I have let it run for 16+ hours and it still hasn't exited. However the file upload part does seem to finish in about 15 minutes consistently since I can see the network traffic drop, but the process doesn't end after that.

This is a major roadblock for us since this is supposed to run every few hours and we can't guarantee that it will ever complete.

@Chris-Pr

This comment has been minimized.

Show comment
Hide comment
@Chris-Pr

Chris-Pr Oct 12, 2016

[image: Inline images 1]
Blue = aws s3 sync
Yellow = cloudberry 4 jobs x 100 threads upload

setup
*aws s3 sync *
instance :m4.10xlarge
disk system NTFS-3G volume on a 6TB gp2 ssd

cloudberry
instance: m4.4xlarge
disk 4 x sc1 5TB

after a full yum update on a fresh system aws s3 sync is working well. Even
though NTFS-3G is extremely slow.

I would use a new fresh AWS AMI connect volume to it and try another push,
make sure your source disk is fast and your instance is large, it uses alot
of memory to sync.
if pushing to s3 from a network source put them inside the same resource
group, and make sure both instances support 10gig network.

On 13 October 2016 at 03:18, Jason Kaczmarsky notifications@github.com
wrote:

I'm also getting this issue when using sync on a Windows 8 machine using
CLI 1.11.2, it also happened on 1.10.41. This has been happening with as
little as 300 files that are only ~5mb in size total. It takes about 15
minutes to do the upload, says "Completed X parts with ... file(s)
remaining", and then finishes after some random amount of time. Sometimes
it finishes instantly (no problem), other times I have let it run for 15
hours and it still hasn't exited. This is a major roadblock for us since
this is supposed to run every few hours.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1775 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIF_lEDx7CF4seTjCUoIfD_Bjom_9kTeks5qzQhHgaJpZM4HUd2G
.

Chris Prangnell
Sydney
+61449247557

Chris-Pr commented Oct 12, 2016

[image: Inline images 1]
Blue = aws s3 sync
Yellow = cloudberry 4 jobs x 100 threads upload

setup
*aws s3 sync *
instance :m4.10xlarge
disk system NTFS-3G volume on a 6TB gp2 ssd

cloudberry
instance: m4.4xlarge
disk 4 x sc1 5TB

after a full yum update on a fresh system aws s3 sync is working well. Even
though NTFS-3G is extremely slow.

I would use a new fresh AWS AMI connect volume to it and try another push,
make sure your source disk is fast and your instance is large, it uses alot
of memory to sync.
if pushing to s3 from a network source put them inside the same resource
group, and make sure both instances support 10gig network.

On 13 October 2016 at 03:18, Jason Kaczmarsky notifications@github.com
wrote:

I'm also getting this issue when using sync on a Windows 8 machine using
CLI 1.11.2, it also happened on 1.10.41. This has been happening with as
little as 300 files that are only ~5mb in size total. It takes about 15
minutes to do the upload, says "Completed X parts with ... file(s)
remaining", and then finishes after some random amount of time. Sometimes
it finishes instantly (no problem), other times I have let it run for 15
hours and it still hasn't exited. This is a major roadblock for us since
this is supposed to run every few hours.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1775 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIF_lEDx7CF4seTjCUoIfD_Bjom_9kTeks5qzQhHgaJpZM4HUd2G
.

Chris Prangnell
Sydney
+61449247557

@brianpkennedy

This comment has been minimized.

Show comment
Hide comment
@brianpkennedy

brianpkennedy Oct 26, 2016

This may be related to the OP which was about downloading from s3 to local. My issue was the other direction (sending from local to S3).

I have been plagued with the issue trying to sync a set of files from a build directory to a remote S3 bucket. In logs, I would see a lot of output about my source files being successfully synched file by file, but would observe a very long (often 15-20 minutes) lag in output before it resumed again. I assumed the script was hung or that I was being rate limited, because it would pick back up and resume to completion after this odd pause.

When i ran the sync command with --debug I found that during the time where I believed the script was hung it was actually doing several ListObjects in the destination (my s3 bucket) that my source directory did not contain. The destination s3 remote bucket has several years worth of historical files in it that the source does not have.

Since I was not using sync to delete any files, but rather as a way to hopefully not send files that hadn't changed, going through the items in the destination is extra work that does not need to be done. The source directory's file size and object count were not of concern, so changing to use cp achieved the same desired end result in a fraction of the time (25 minutes using sync vs 40s using cp --recursive)

tl;dr

The size of your sync destination impacts the length of time a sync command takes. I changed my script to cp --recursive instead of sync and cut the time down signficantly.

brianpkennedy commented Oct 26, 2016

This may be related to the OP which was about downloading from s3 to local. My issue was the other direction (sending from local to S3).

I have been plagued with the issue trying to sync a set of files from a build directory to a remote S3 bucket. In logs, I would see a lot of output about my source files being successfully synched file by file, but would observe a very long (often 15-20 minutes) lag in output before it resumed again. I assumed the script was hung or that I was being rate limited, because it would pick back up and resume to completion after this odd pause.

When i ran the sync command with --debug I found that during the time where I believed the script was hung it was actually doing several ListObjects in the destination (my s3 bucket) that my source directory did not contain. The destination s3 remote bucket has several years worth of historical files in it that the source does not have.

Since I was not using sync to delete any files, but rather as a way to hopefully not send files that hadn't changed, going through the items in the destination is extra work that does not need to be done. The source directory's file size and object count were not of concern, so changing to use cp achieved the same desired end result in a fraction of the time (25 minutes using sync vs 40s using cp --recursive)

tl;dr

The size of your sync destination impacts the length of time a sync command takes. I changed my script to cp --recursive instead of sync and cut the time down signficantly.

@vopenka

This comment has been minimized.

Show comment
Hide comment
@vopenka

vopenka Feb 18, 2017

I had the same issue. In my case this was caused by not enough space in destination directory.

vopenka commented Feb 18, 2017

I had the same issue. In my case this was caused by not enough space in destination directory.

@Alasano

This comment has been minimized.

Show comment
Hide comment
@Alasano

Alasano Feb 28, 2017

I also faced this issue. The aws s3 cp command would hang before completion for a few minutes with files 100MB+ . I increased the multipart_threshold and multipart_chunksize as well which seems to have alleviated the problem.

aws-cli/1.11.56 Python/2.7.8 Windows/8 botocore/1.5.19

Alasano commented Feb 28, 2017

I also faced this issue. The aws s3 cp command would hang before completion for a few minutes with files 100MB+ . I increased the multipart_threshold and multipart_chunksize as well which seems to have alleviated the problem.

aws-cli/1.11.56 Python/2.7.8 Windows/8 botocore/1.5.19

@AlexShemeshWix

This comment has been minimized.

Show comment
Hide comment
@AlexShemeshWix

AlexShemeshWix Mar 12, 2017

have same problem. aws s3 cp just hangs.
Use case is very simple - im copying single file 10GB in size from s3 to ec2.
Very disappointing

AlexShemeshWix commented Mar 12, 2017

have same problem. aws s3 cp just hangs.
Use case is very simple - im copying single file 10GB in size from s3 to ec2.
Very disappointing

@AlexShemeshWix

This comment has been minimized.

Show comment
Hide comment
@AlexShemeshWix

AlexShemeshWix Mar 12, 2017

My bad. My issue was simple. Ive ran command without enough permissions. Still the problem that if file is big, and you dont have permissions it will just hang. No error message.
Have to be fixed.

AlexShemeshWix commented Mar 12, 2017

My bad. My issue was simple. Ive ran command without enough permissions. Still the problem that if file is big, and you dont have permissions it will just hang. No error message.
Have to be fixed.

@akohlsmith

This comment has been minimized.

Show comment
Hide comment
@akohlsmith

akohlsmith Jul 14, 2017

aws-cli 1.11.13 on Ubuntu (aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-72-generic botocore/1.4.70)... having this issue intermittently trying to read a 300MB file from a bucket. I do have full permissions on the bucket (and occasionally the transfer completes successfully).

Debug log shows that the transfer is spread out over numerous threads, then suddenly something catches and I'll get a network timeout, which crashes the program:

requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mybucket.s3.amazonaws.com', port=443): Read timed out.

akohlsmith commented Jul 14, 2017

aws-cli 1.11.13 on Ubuntu (aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-72-generic botocore/1.4.70)... having this issue intermittently trying to read a 300MB file from a bucket. I do have full permissions on the bucket (and occasionally the transfer completes successfully).

Debug log shows that the transfer is spread out over numerous threads, then suddenly something catches and I'll get a network timeout, which crashes the program:

requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mybucket.s3.amazonaws.com', port=443): Read timed out.

@Green-Beret

This comment has been minimized.

Show comment
Hide comment
@Green-Beret

Green-Beret Jan 16, 2018

Have same issue, it happened with both cp and sync command for a 300MB file, the output shows the transfer hangs, and after a few seconds, the read timedout error is returned.

aws version:
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-97-generic botocore/1.4.70

command used:

aws s3 cp s3://***-dumps/Tue/abc.dump.gz .
aws --cli-read-timeout=0 --cli-connect-timeout=0 s3 cp 3://***-dumps/Tue/abc.dump.gz .

PS: I can download the file from s3 console in chrome browser.

There's some debug logs for this command:
aws s3 sync s3://***-dumps/Tue/ . --exclude "*" --include "abc.dump.gz" --debug

2018-01-16 11:45:15,310 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,310 - Thread-3 - s3transfer.tasks - DEBUG - GetObjectTask(transfer_id=0, {'bucket': '***-dumps', 'extra_args': {'Range': 'bytes=301989888-'}, 'key': 'Tue/abc.dump.gz'}) about to wait for the following futures []
2018-01-16 11:45:15,311 - Thread-3 - s3transfer.tasks - DEBUG - GetObjectTask(transfer_id=0, {'bucket': '***-dumps', 'extra_args': {'Range': 'bytes=301989888-'}, 'key': 'Tue/abc.dump.gz'}) done waiting for dependent futures
2018-01-16 11:45:15,311 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,385 - Thread-4 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,580 - Thread-10 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,886 - Thread-9 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:22,682 - Thread-12 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:22,686 - Thread-12 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:24,079 - Thread-6 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:31,511 - Thread-11 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:31,515 - Thread-11 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:41,054 - Thread-8 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:41,058 - Thread-8 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:43,633 - Thread-7 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:43,637 - Thread-7 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:46:25,317 - Thread-5 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:46:25,321 - Thread-5 - s3transfer.futures - DEBUG - Submitting task IORenameFileTask(transfer_id=0, {'final_filename': '/home/test/abc.dump.gz'}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f5e3d175208> for transfer request: 0.
2018-01-16 11:46:25,321 - Thread-5 - s3transfer.utils - DEBUG - Acquiring 0
2018-01-16 11:46:25,322 - Thread-5 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:46:25,322 - Thread-13 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/home/test/abc.dump.gz'}) about to wait for the following futures []
2018-01-16 11:46:25,322 - Thread-13 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/home/test/abc.dump.gz'}) done waiting for dependent futures
2018-01-16 11:46:25,330 - Thread-13 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download failed: s3://***-dumps/Tue/abc.dump.gz to ./abc.dump.gz HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:46:25,332 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

Green-Beret commented Jan 16, 2018

Have same issue, it happened with both cp and sync command for a 300MB file, the output shows the transfer hangs, and after a few seconds, the read timedout error is returned.

aws version:
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-97-generic botocore/1.4.70

command used:

aws s3 cp s3://***-dumps/Tue/abc.dump.gz .
aws --cli-read-timeout=0 --cli-connect-timeout=0 s3 cp 3://***-dumps/Tue/abc.dump.gz .

PS: I can download the file from s3 console in chrome browser.

There's some debug logs for this command:
aws s3 sync s3://***-dumps/Tue/ . --exclude "*" --include "abc.dump.gz" --debug

2018-01-16 11:45:15,310 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,310 - Thread-3 - s3transfer.tasks - DEBUG - GetObjectTask(transfer_id=0, {'bucket': '***-dumps', 'extra_args': {'Range': 'bytes=301989888-'}, 'key': 'Tue/abc.dump.gz'}) about to wait for the following futures []
2018-01-16 11:45:15,311 - Thread-3 - s3transfer.tasks - DEBUG - GetObjectTask(transfer_id=0, {'bucket': '***-dumps', 'extra_args': {'Range': 'bytes=301989888-'}, 'key': 'Tue/abc.dump.gz'}) done waiting for dependent futures
2018-01-16 11:45:15,311 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,385 - Thread-4 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,580 - Thread-10 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:15,886 - Thread-9 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:22,682 - Thread-12 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:22,686 - Thread-12 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:24,079 - Thread-6 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:31,511 - Thread-11 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:31,515 - Thread-11 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:41,054 - Thread-8 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:41,058 - Thread-8 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:45:43,633 - Thread-7 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:45:43,637 - Thread-7 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:46:25,317 - Thread-5 - s3transfer.tasks - DEBUG - Exception raised.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 226, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 301, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 791, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 575, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 518, in _main
    for chunk in chunks:
  File "/usr/lib/python3/dist-packages/s3transfer/download.py", line 646, in __next__
    chunk = self._body.read(self._chunksize)
  File "/usr/lib/python3/dist-packages/s3transfer/utils.py", line 491, in read
    value = self._stream.read(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/botocore/response.py", line 74, in read
    chunk = self._raw_stream.read(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 311, in read
    flush_decoder = True
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 231, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:46:25,321 - Thread-5 - s3transfer.futures - DEBUG - Submitting task IORenameFileTask(transfer_id=0, {'final_filename': '/home/test/abc.dump.gz'}) to executor <s3transfer.futures.BoundedExecutor object at 0x7f5e3d175208> for transfer request: 0.
2018-01-16 11:46:25,321 - Thread-5 - s3transfer.utils - DEBUG - Acquiring 0
2018-01-16 11:46:25,322 - Thread-5 - s3transfer.utils - DEBUG - Releasing acquire 0/None
2018-01-16 11:46:25,322 - Thread-13 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/home/test/abc.dump.gz'}) about to wait for the following futures []
2018-01-16 11:46:25,322 - Thread-13 - s3transfer.tasks - DEBUG - IORenameFileTask(transfer_id=0, {'final_filename': '/home/test/abc.dump.gz'}) done waiting for dependent futures
2018-01-16 11:46:25,330 - Thread-13 - s3transfer.utils - DEBUG - Releasing acquire 0/None
download failed: s3://***-dumps/Tue/abc.dump.gz to ./abc.dump.gz HTTPSConnectionPool(host='***-dumps.s3.amazonaws.com', port=443): Read timed out.
2018-01-16 11:46:25,332 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

@akohlsmith

This comment has been minimized.

Show comment
Hide comment
@akohlsmith

akohlsmith Apr 5, 2018

I too am having this problem with 1GB files (copying from s3 to local storage).

aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-72-generic botocore/1.4.70, using the us-east (N. Virginia) datacentre.

When I use s3 cp to copy a 1GB file from a bucket to local storage, the transfer occurs and at the very end (1013MiB transferred) things just stall and time out. I do not get the file. This is entirely repeatable, and playing with the multipart thresholds/chunksize and number of concurrent transfers doesn't do much, although I can get it to hang much earlier (anywhere from 130MB to 800MB) in the process with enough playing around.

I've also played with --cli-read-timeout but it seems the transfer has completed; it doesn't seem to be waiting for any more data.

Originally I was trying to restore a file from S3 through duplicity, but it was timing out. My testing and some help from the duplicity team led me to try the aws CLI, and here I am.

akohlsmith commented Apr 5, 2018

I too am having this problem with 1GB files (copying from s3 to local storage).

aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-72-generic botocore/1.4.70, using the us-east (N. Virginia) datacentre.

When I use s3 cp to copy a 1GB file from a bucket to local storage, the transfer occurs and at the very end (1013MiB transferred) things just stall and time out. I do not get the file. This is entirely repeatable, and playing with the multipart thresholds/chunksize and number of concurrent transfers doesn't do much, although I can get it to hang much earlier (anywhere from 130MB to 800MB) in the process with enough playing around.

I've also played with --cli-read-timeout but it seems the transfer has completed; it doesn't seem to be waiting for any more data.

Originally I was trying to restore a file from S3 through duplicity, but it was timing out. My testing and some help from the duplicity team led me to try the aws CLI, and here I am.

@tchrisboles

This comment has been minimized.

Show comment
Hide comment
@tchrisboles

tchrisboles Oct 14, 2018

I am having the same trouble copying a single large binary file (10.9gb) from my s3 storage to my ec2 instance. Plenty of disk space on my ubuntu instance.

command was simplest form possible :
$aws s3 cp s3://folder-name/binary-filename ./local-dir-on-ec2/

It downloads like lightning until it reads something like "10.9gb of 10.9gb copied with 1 file remaining." and then the command is hung. No change. I waited 15 min. Restarted the same command about 8 times. Looked up this thread. Then went back after about 10 minutes, and did the same thing again. And this time, it completed the copy without hanging. !-??

Same thing happened with a second ~10gb binary file. Transfer appeared to go lightning fast and then hung up with appearance that all data had been copied. "10.4 gb of 10.4 gb copied with 1 file remaining." I tried three more times with exactly same command, and finally got the copy to work.

Amazon needs to fix this issue or s3 is of no real use to ec2 customers.

tchrisboles commented Oct 14, 2018

I am having the same trouble copying a single large binary file (10.9gb) from my s3 storage to my ec2 instance. Plenty of disk space on my ubuntu instance.

command was simplest form possible :
$aws s3 cp s3://folder-name/binary-filename ./local-dir-on-ec2/

It downloads like lightning until it reads something like "10.9gb of 10.9gb copied with 1 file remaining." and then the command is hung. No change. I waited 15 min. Restarted the same command about 8 times. Looked up this thread. Then went back after about 10 minutes, and did the same thing again. And this time, it completed the copy without hanging. !-??

Same thing happened with a second ~10gb binary file. Transfer appeared to go lightning fast and then hung up with appearance that all data had been copied. "10.4 gb of 10.4 gb copied with 1 file remaining." I tried three more times with exactly same command, and finally got the copy to work.

Amazon needs to fix this issue or s3 is of no real use to ec2 customers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment