Support multipart upload of S3, OSS, COS and OBS

[zhezhidashi]

What changes are proposed in this pull request?

Support multipart upload of S3, OSS, COS and OBS.

Why are the changes needed?

1. support multipart upload mode of s3. Simple upload refers to this. Multipart upload refer to this
2. Multipart upload improve streaming upload mode. Multipart upload remove temp file, that optimizes three read and write disk operations into one. Upload each time the input data reaches the partition size, and use Netty zero-copy technology for splicing.
3. OSS, COS and OBS are similar as described to the above.

There are some experiments conducted on Mac Laptop and AWS instance. Simple upload (default) and streaming upload are two original upload methods of alluxio.

File Size: 4.8GB

MinIO (Mac Laptop):
Simple Upload: 28 seconds
Streaming Upload: 20 seconds
Multipart Upload: 12 seconds

AWS same region (r6a.xlarge):
Simple Upload: 25 seconds
Streaming Upload: 18 seconds
Multipart Upload: 12 seconds

In an environment with sufficient bandwidth (or an intranet environment), the speed increase is obvious.

Does this PR introduce any user facing changes?

alluxio.underfs.object.store.multipart.upload.timeout: Timeout for uploading part when using multipart upload.

S3:
alluxio.underfs.s3.multipart.upload.enabled: Whether to enable multipart upload for S3. If it is true, then multipart upload of S3 will be enabled. Defult value is false.
alluxio.underfs.s3.multipart.upload.partition.size: Multipart upload partition size for S3. The default partition size is 64MB.

OSS:
alluxio.underfs.oss.multipart.upload.enabled: Whether to enable multipart upload for OSS.
alluxio.underfs.oss.multipart.upload.threads: Thread pool size for OSS multipart upload.
alluxio.underfs.oss.multipart.upload.partition.size: Multipart upload partition size for OSS. The default partition size is 64MB.

COS:
alluxio.underfs.cos.multipart.upload.enabled: Whether to enable multipart upload for COS.
alluxio.underfs.cos.multipart.upload.threads: Thread pool size for COS multipart upload.
alluxio.underfs.cos.multipart.upload.partition.size: Multipart upload partition size for COS. The default partition size is 64MB.

OBS:
alluxio.underfs.obs.multipart.upload.enabled: Whether to enable multipart upload for OBS.
alluxio.underfs.obs.multipart.upload.threads: Thread pool size for OBS multipart upload.
alluxio.underfs.obs.multipart.upload.partition.size: Multipart upload partition size for OBS. The default partition size is 64MB.

[jja725]

Thanks for the improvement and it could benefit the write case a lot! I wonder what's the user interface you are using to trigger the write(alluxio, fuse, s3)?

[zhezhidashi]

Set alluxio.underfs.s3.multipart.upload.enabled=true in alluxio-site.properties to trigger the write.

[qian0817]

Can you explain the difference between the previously implemented streaming upload and the current implementation of multipart upload, as well as why there is a performance improvement? To me, they seem to be the same.

[qian0817]

The referenced document about S3 MD5 refers to the fact that the S3 server will calculate the MD5 of the uploaded fragments after uploading, rather than calculating the MD5 on the client side before uploading.

[zhezhidashi]

The difference between multipart upload and streaming upload is that, when upload a local file to s3, streaming upload will generate temporary files, while multipart upload will upload the input stream directly.

In the intranet environment, the speed of reading and writing to the disk cannot be ignored. When uploading a local file, streaming upload will read the disk twice and write to the disk once. But multipart upload will only read the disk once.

As for MD5, In aws SDK, in AmazonS3Client.class, I see this:

Actually when you don't set Md5, it will generate md5DigestStream.

In multipart upload mode, AWS SDK will calculate MD5 of every part of the file.

The document I linked before actually only said that MD5 will be calculated, and it did not say whether the S3 client or the server will calculate MD5, so it cannot be considered that the client does not calculate MD5.

In fact, think about it from another angle. If you are a developer of S3, in order to ensure data consistency, they will probably perform data verification by default.

[qian0817]

@zhezhidashi Thanks for your reply.

1. AWS S3 requires a minimum size of 5MB for each part, and other object storage may have different requirements. Your implementation does not check the part size.
   https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html

2. If you are concerned about disk write performance issues, I think you can directly modify ObjectLowLevelOutputStream to add a configuration that does not need to be written to a file. Or you can try mounting the file path as tmpfs to see if it can achieve similar improvements as S3AMultiUploadOutputStream now. There is too much similarity in logic between these two classes.

3. The behavior of MD5 errors here seems to be different from the behavior of inconsistent MD5 added in requests. If relying on SDK mechanisms, this object will still exist on S3 after the SDK throws an exception and needs to be manually deleted. If Content-MD5 is added to the request, this object will not exist on S3. And you cannot guarantee that every object storage implementation will still verify MD5 when Content-MD5 is not set.

[zhezhidashi]

Thank you! I will rethink these problem.

[zhezhidashi]

1. Support MD5 checksum.
2. Add upload partition size. The data is uploaded only when it reaches the partition size.
3. When the file is less than 5MB, or less than partition size, upload the file directly instead of uploading in parts.

[JiamingMai]

LGTM

[JiamingMai]

alluxio-bot, merge this please

[alluxio-bot]

merge failed:
Merge refused because the PR contains 16 commit with change id and pr link in the commit body

[JiamingMai]

alluxio-bot, merge this please