preserve x-amz-content-sha256 if specified in presign_v4

[vsoch]

I have the following use case:

• I'm deploying Minio alongside a server (uwsgi with Django, that's not important)
• A command line client, scs-library-client through Singularity is handling parsing an object to ask for mutlipart upload presigned URLs from my server. It conforms to the S3 API for Multipart upload.
• This means that my server has endpoints to Start a multipart upload, and then a makes subsequent requests to retrieve a pre-signed URL, one for each part.
• The Singularity client then uses the signed URL to interact with Minio (for those wondering this does mean that the external URL (127.0.0.1:9000) is different from the internal one (minio:9000) and to get around this I needed to make two clients, one for internal interaction (starting the multipart upload) and the other would have been used to generated the signed urls but I wound up just using it to create a prefix, I can elaborate more on this if anyone is interested).
• After upload, Singularity again pings the server with a list of parts (numbers and ETags) that are then used for the (internal) s3 client to complete_multipart_upload.

The reason for this PR comes down to when Singularity uses the signed URL. I was using the signed url generated by the s3 client, that resulted in a signature mismatch. I next used the same flow that I saw for my working (single) PUT object request (also a signed URL from the Singularity client) that generates a URL and then uses signers.presign_v4. This continually failed. I tried about 1000 things, but the trick was that I needed to give the sha256 sum provided by the Singularity client to be included in the signature generation. Since I wasn't able to specify it for presign_v4, I essentially reproduced the function and used it as a variable. The error disappeared and my chunks sped through my localhost like a chicken with it's butt on fire! 🐔 🔥

So - this small fix will allow for an interested user that wants to use the minio-py client to generate a presigned url in the case that a server is making a request on behalf of a command line client (where the object to be uploaded lives and is not accessible to the server generating the URLs). The command line client sends the sha256sum to the server (which is not able to calculate it). While I think it would be great for minio-py to have a fully supported function to generate these presigned URLS more easily (akin to how you can generate one for a single PUT request), in the meantime this small change to allow specifying the content_hash_hex will be useful to users that have similar use case to me. For future reference, the code I am referring to is singularityhub/sregistry#298.

Signed-off-by: vsoch vsochat@stanford.edu

[vsoch]

hey maintainers! It looks like this is some kind of race condition? Or at least the failed RUN is failing because it's reporting a bucket is not empty. Let me know if there is something I can do on my end - my changes to the function should not have led to any functional changes in the code as the default for my parameter is None, and it gets set to exactly what it was before.

[harshavardhana]

hey maintainers! It looks like this is some kind of race condition? Or at least the failed RUN is failing because it's reporting a bucket is not empty. Let me know if there is something I can do on my end - my changes to the function should not have led to any functional changes in the code as the default for my parameter is None, and it gets set to exactly what it was before.

Don't worry about it we need to fix the Windows CI build

[vsoch]

So to keep it generic we already take headers so if that covers it, then we don't need additional param - just honor x-amz-content-sha256 from input headers.

@harshavardhana I noticed that and tried supplying the same content hash via the headers variable, this resulted in the invalid signature still. I haven't looked closely at generate_canonical_request but it takes in content_sha256 separately so it must be handled differently, or at least having it provided alongside a header with the correct sha256 somehow adds an incorrect value.

[harshavardhana]

@harshavardhana I noticed that and tried supplying the same content hash via the headers variable, this resulted in the invalid signature still. I haven't looked closely at generate_canonical_request but it takes in content_sha256 separately so it must be handled differently, or at least having it provided alongside a header with the correct sha256 somehow adds an incorrect value.

The reason is that the code should look for x-amz-content-sha256 present on headers and use that instead

[vsoch]

The one last bit I'd want to ask about is where we can document this? If it's included in this repository I can add it here, otherwise you can point me to where a user would want to find this detail.

[harshavardhana]

The one last bit I'd want to ask about is where we can document this? If it's included in this repository I can add it here, otherwise you can point me to where a user would want to find this detail.

Just add it as a comment, we don't really want to encourage this as most of our users are not aware of AWS S3 behavior and we would like to keep that abstracted.

[vsoch]

okay, I'll add a comment to before the loop. One second!

[vsoch]

okay I added a comment, and I also added a very detailed commit message so it's present in the git history too (and unexpected source of goodness sometimes!)

[harshavardhana]

ping @vadmeste

[vadmeste]

LGTM