Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uploading a stream failed #1184

Closed
nbrownus opened this issue Feb 26, 2015 · 8 comments
Closed

Uploading a stream failed #1184

nbrownus opened this issue Feb 26, 2015 · 8 comments
Labels

Comments

@nbrownus
Copy link

@nbrownus nbrownus commented Feb 26, 2015

While uploading a stream to s3 the upload failed but the aws process did not die. The error occurred well after the upload started. I can only assume that aws tried to upload more than 9999 parts?

Output from other_process | aws s3 cp - s3://thebucket/thefile

upload failed: ./- to s3://thebucket/thefile
A client error (InvalidArgument) occurred when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive

Version info

$ aws --version
aws-cli/1.7.12 Python/2.7.3 Linux/3.2.0-59-generic
@nbrownus
Copy link
Author

@nbrownus nbrownus commented Feb 26, 2015

Well I didn't read enough of the documentation.

--expected-size (string) This argument specifies the expected size of a stream in terms of bytes. Note that this argument is needed only when a stream is being uploaded to s3 and the size is larger than 5GB. Failure to include this argument under these conditions may result in a failed upload. due to too many parts in upload.
@nbrownus nbrownus closed this Feb 26, 2015
@nbrownus
Copy link
Author

@nbrownus nbrownus commented Feb 26, 2015

Actually reopening because the upload failure did not cause aws to exit at all

@nbrownus nbrownus reopened this Feb 26, 2015
@kyleknap
Copy link
Member

@kyleknap kyleknap commented Feb 26, 2015

@nbrownus
How long did you wait after the upload failed? Sometimes the threads need to finish uploading their respective part before they begin shutting down and cleaning up.

@nbrownus
Copy link
Author

@nbrownus nbrownus commented Feb 26, 2015

I noticed something was wrong after bytes transmitted out of the network interface dropped to 0. I manually stopped the process about 15 minutes after traffic stopped. I saw the above error in the logs but everything before aws in the pipeline was still chugging along.

@kyleknap
Copy link
Member

@kyleknap kyleknap commented Feb 26, 2015

Thanks! We will look into this.

@kyleknap kyleknap added bug and removed response-needed labels Feb 26, 2015
@jamesls jamesls added the s3 label Mar 2, 2015
@kyleknap kyleknap added the s3stream label Nov 19, 2015
@kyleknap
Copy link
Member

@kyleknap kyleknap commented Aug 17, 2016

With the release of 1.10.57, we replaced the underlying architecture for streaming and this should no longer be an issue as errors handling is much resilient and quicker in shutting down the entire transfer process. There should be no more hanging as well. Resolving issue.

@kyleknap kyleknap closed this Aug 17, 2016
@eladnava
Copy link

@eladnava eladnava commented Aug 12, 2018

@kyleknap Unfortunately I just got this error when uploading a stream on aws-cli/1.15.76:

An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive
@jpellman
Copy link

@jpellman jpellman commented Aug 23, 2018

I've also received this error with aws-cli/1.15.76.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.