Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading a stream failed #1184

Closed
nbrownus opened this issue Feb 26, 2015 · 10 comments
Closed

Uploading a stream failed #1184

nbrownus opened this issue Feb 26, 2015 · 10 comments
Labels
bug This issue is a bug. s3stream s3

Comments

@nbrownus
Copy link

While uploading a stream to s3 the upload failed but the aws process did not die. The error occurred well after the upload started. I can only assume that aws tried to upload more than 9999 parts?

Output from other_process | aws s3 cp - s3://thebucket/thefile

upload failed: ./- to s3://thebucket/thefile
A client error (InvalidArgument) occurred when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

Version info

$ aws --version
aws-cli/1.7.12 Python/2.7.3 Linux/3.2.0-59-generic
@nbrownus
Copy link
Author

Well I didn't read enough of the documentation.

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload. due to too many parts in upload.

@nbrownus
Copy link
Author

Actually reopening because the upload failure did not cause aws to exit at all

@nbrownus nbrownus reopened this Feb 26, 2015
@kyleknap
Copy link
Contributor

@nbrownus
How long did you wait after the upload failed? Sometimes the threads need to finish uploading their respective part before they begin shutting down and cleaning up.

@nbrownus
Copy link
Author

I noticed something was wrong after bytes transmitted out of the network interface dropped to 0. I manually stopped the process about 15 minutes after traffic stopped. I saw the above error in the logs but everything before aws in the pipeline was still chugging along.

@kyleknap
Copy link
Contributor

Thanks! We will look into this.

@kyleknap kyleknap added bug This issue is a bug. and removed response-needed labels Feb 26, 2015
@jamesls jamesls added the s3 label Mar 2, 2015
@kyleknap
Copy link
Contributor

With the release of 1.10.57, we replaced the underlying architecture for streaming and this should no longer be an issue as errors handling is much resilient and quicker in shutting down the entire transfer process. There should be no more hanging as well. Resolving issue.

@eladnava
Copy link

@kyleknap Unfortunately I just got this error when uploading a stream on aws-cli/1.15.76:

An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

@jpellman
Copy link

I've also received this error with aws-cli/1.15.76.

@combinatorist
Copy link

And I just got this error with aws-cli/1.16.102

@combinatorist
Copy link

combinatorist commented Jul 23, 2020

... but setting --expected-size (as @nbrownus suggested) fixed it for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. s3stream s3
Projects
None yet
Development

No branches or pull requests

6 participants