Inconsistent ETag on complete multipart upload POST response

[hectcastro]

When completing a multipart upload against Amazon S3, the XML payload contains a specialized ETag:

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Location>http://joker-test.s3.amazonaws.com/25.txt</Location>
   <Bucket>joker-test</Bucket>
   <Key>25.txt</Key>
   <ETag>"6ce4f6859aaecc0e7f23fc487c31f8db-2"</ETag>
</CompleteMultipartUploadResult>

According to a response from an Amazon employee on their support forums, this ETag is:

The checksum in a multipart digest is the MD5 of the concatenated MD5 bytes of each of the parts affixed with the number of parts

Basically:

ETag = MD5(Sum(p \in numberParts, MD5(PartBytes(p))) + "-" + numberParts

In this example, I uploaded a 25MB file in chunks of 20MB, so the numberParts part of that adds up. I also used an S3 client for Go that calculates the MD5 of the concatenated MD5 bytes for each part, and that is consistent as well.

When completing a similar multipart upload against Riak CS, the ETag attribute looks different (it actually looks like the UploadId):

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Location>http://isos.s3.amazonaws.com/25.txt</Location>
   <Bucket>isos</Bucket>
   <Key>25.txt</Key>
   <ETag>oD_XfZorRXOY6ouY61cFMA==</ETag>
</CompleteMultipartUploadResult>

Response from multipart upload initialization to show that the ETag is the UploadId:

<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
   <Bucket>isos</Bucket>
   <Key>25.txt</Key>
   <UploadId>oD_XfZorRXOY6ouY61cFMA==</UploadId>
</InitiateMultipartUploadResult>

Unfortunately, clients who use this ETag to confirm that all of the parts in a multipart upload request completed successfully are not able to do so against Riak CS.

[kuenishi]

fixed by #887

[hectcastro]

Nice. :)