Uploading a stream failed #1184

Closed
nbrownus opened this Issue Feb 26, 2015 · 6 comments

Comments

Projects
None yet
3 participants
@nbrownus

While uploading a stream to s3 the upload failed but the aws process did not die. The error occurred well after the upload started. I can only assume that aws tried to upload more than 9999 parts?

Output from other_process | aws s3 cp - s3://thebucket/thefile

upload failed: ./- to s3://thebucket/thefile
A client error (InvalidArgument) occurred when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

Version info

$ aws --version
aws-cli/1.7.12 Python/2.7.3 Linux/3.2.0-59-generic
@nbrownus

This comment has been minimized.

Show comment
Hide comment
@nbrownus

nbrownus Feb 26, 2015

Well I didn't read enough of the documentation.

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload. due to too many parts in upload.

Well I didn't read enough of the documentation.

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload. due to too many parts in upload.

@nbrownus nbrownus closed this Feb 26, 2015

@nbrownus

This comment has been minimized.

Show comment
Hide comment
@nbrownus

nbrownus Feb 26, 2015

Actually reopening because the upload failure did not cause aws to exit at all

Actually reopening because the upload failure did not cause aws to exit at all

@nbrownus nbrownus reopened this Feb 26, 2015

@kyleknap

This comment has been minimized.

Show comment
Hide comment
@kyleknap

kyleknap Feb 26, 2015

Member

@nbrownus
How long did you wait after the upload failed? Sometimes the threads need to finish uploading their respective part before they begin shutting down and cleaning up.

Member

kyleknap commented Feb 26, 2015

@nbrownus
How long did you wait after the upload failed? Sometimes the threads need to finish uploading their respective part before they begin shutting down and cleaning up.

@nbrownus

This comment has been minimized.

Show comment
Hide comment
@nbrownus

nbrownus Feb 26, 2015

I noticed something was wrong after bytes transmitted out of the network interface dropped to 0. I manually stopped the process about 15 minutes after traffic stopped. I saw the above error in the logs but everything before aws in the pipeline was still chugging along.

I noticed something was wrong after bytes transmitted out of the network interface dropped to 0. I manually stopped the process about 15 minutes after traffic stopped. I saw the above error in the logs but everything before aws in the pipeline was still chugging along.

@kyleknap

This comment has been minimized.

Show comment
Hide comment
@kyleknap

kyleknap Feb 26, 2015

Member

Thanks! We will look into this.

Member

kyleknap commented Feb 26, 2015

Thanks! We will look into this.

@kyleknap kyleknap added bug and removed response-needed labels Feb 26, 2015

@jamesls jamesls added the s3 label Mar 2, 2015

@kyleknap kyleknap added the s3stream label Nov 19, 2015

@kyleknap

This comment has been minimized.

Show comment
Hide comment
@kyleknap

kyleknap Aug 17, 2016

Member

With the release of 1.10.57, we replaced the underlying architecture for streaming and this should no longer be an issue as errors handling is much resilient and quicker in shutting down the entire transfer process. There should be no more hanging as well. Resolving issue.

Member

kyleknap commented Aug 17, 2016

With the release of 1.10.57, we replaced the underlying architecture for streaming and this should no longer be an issue as errors handling is much resilient and quicker in shutting down the entire transfer process. There should be no more hanging as well. Resolving issue.

@kyleknap kyleknap closed this Aug 17, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment