Support for Multipart Uploads

[cyberduck]

6883dc5 created the issue

S3 recently added a new feature: multi-part uploads.

The benefits of using this feature with Cyberduck include:

• Resumable uploads
• Better performance (potentially) if multiple parts are uploaded in parallel

This added functionality comes almost for free in the AWS SDK for Java mentioned in ticket #5486, so it is probably useful to consider this as blocked on that ticket.

I am also willing to contribute code for this feature. In all honesty it should be nearly trivial once #5486 is resolved.

[cyberduck]

@dkocher commented

This is also discussed in this thread on the JetS3t mailing list.

[cyberduck]

@dkocher commented

In addition this would allow to support the new 5 TB Object Size Limit.

[cyberduck]

@dkocher commented

#5592 closed as duplicate.

[cyberduck]

@dkocher commented

In d6f5bd8. Resuming is not yet supported. Currently the default parallelism is 10 requests. Files larger than 16MB are uploaded using multiparts.

[cyberduck]

454a57d commented

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

[cyberduck]

@dkocher commented

Replying to [comment:6 stickenhoffen]:

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

Are you running the latest snapshot build? Please post the HTTP transcript from the log drawer in the Transfers window.

[cyberduck]

454a57d commented

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

[cyberduck]

@dkocher commented

Replying to [comment:8 stickenhoffen]:

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

Sure, [mailto:feedback@cyberduck.ch].

[cyberduck]

@dkocher commented

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

[cyberduck]

@dkocher commented

Replying to [comment:10 dkocher]:

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

There is possibly nothing we can do about these failures but to support resuming parts of the transfer that failed.

[cyberduck]

@dkocher commented

We have prepared but not yet enabled support for resuming parts of a multipart upload in 346e367. The problem is that failed uploads take up space and must be explicitly cancelled if the transfer should be aborted altogether. Currently we do this for every incomplete upload. If resuming should be possible we obviously can't do this but need an option for the user to cancel later instead of resuming.