Skip to content
This repository was archived by the owner on Jul 25, 2020. It is now read-only.

JCLOUDS-894: Add portable multipart upload#762

Merged
6 commits merged intojclouds:masterfrom
gaul:mpu
Jun 8, 2015
Merged

JCLOUDS-894: Add portable multipart upload#762
6 commits merged intojclouds:masterfrom
gaul:mpu

Conversation

@gaul
Copy link
Copy Markdown
Member

@gaul gaul commented Jun 2, 2015

This unifies the provider multipart upload code paths and removes code
duplication.

@gaul
Copy link
Copy Markdown
Member Author

gaul commented Jun 2, 2015

@danbroudy @kahing @zack-shoylev This pull request follows on to the earlier one exposing the component multipart operations.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need a better strategy here -- we should pick a combination of minimum part size, maximum part size, and number of parts. A good combination will do less work when encountering network errors and allow better use of the uplink via parallel uploads.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AWS Java S3 SDK does the following:

public static long calculateOptimalPartSize(PutObjectRequest putObjectRequest, TransferManagerConfiguration configuration) {
    double contentLength = TransferManagerUtils.getContentLength(putObjectRequest);
    double optimalPartSize = (double)contentLength / (double)MAXIMUM_UPLOAD_PARTS;
     // round up so we don't push the upload over the maximum number of parts
    optimalPartSize = Math.ceil(optimalPartSize);
    return (long)Math.max(optimalPartSize, configuration.getMinimumUploadPartSize());
}

AWS SDK defaults to a maximum of 10000 parts. The minimum default part size is 5MB. So, uploading a 51GB file, for example, would use 8500 6MB parts.

jclouds could use a similar mechanism. It would probably make sense to expose the configuration parameters to be able to change the default behavior.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reparented the S3 MultipartUploadSlicingAlgorithm to core so we have the same algorithm as before.

@ghost ghost merged commit 04c4d5c into jclouds:master Jun 8, 2015
@gaul gaul deleted the mpu branch June 8, 2015 05:48
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants