Support generation of presigned urls for multipart uploads

[damianh]

Currently, to generate a presigned url, the following code is used.

GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
{
	BucketName = bucketName,
	Key = objectKey,
	Expires = expiryTime,
	Verb = HttpVerb.PUT,
};

When uploading a part, one must include an uploadId and partNumber as part of the query string. Since it is not possible to pass
that into the GetPreSignedUrlRequest, the url will result in a signature mismatch.

Possible Solution

Extend GetPreSignedUrlRequest with properties UploadId and PartNumber, and include these values in the signature calculation
as part of the canonicalized query parameters.

GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
{
	BucketName = bucketName,
	Key = objectKey,
	Expires = expiryTime,
	Verb = HttpVerb.PUT,
	UploadId = uploadId,
	PartNumber = partNumber
};

Context

Am creating a tool to allows users to upload large files to S3 and absolutely need multi-part support. I am leveraging lambda to generate each part URL on demand. To get it working, I had to import https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-examples-using-sdks.html#sig-v4-examples-using-sdk-dotnet to generate the signatures myself (all the signing stuff in the SDK is internal). While this works, I think it would be better if the SDK supported it out of the box.

Related items on the web of people with same or similar issue below. Seems this has been a need for a few years.

• https://stackoverflow.com/questions/20847196/amazon-s3-multipart-upload-using-query-string-authentication-and-net-sdk
• https://stackoverflow.com/questions/37217532/is-there-a-net-library-that-can-sign-a-request-with-aws-v4-signature
• https://stackoverflow.com/questions/42676885/aws-v4-signing-of-net-httpclient
• http://gauravmantri.com/2014/01/06/create-pre-signed-url-using-c-for-uploading-large-files-in-amazon-s3/

[damianh]

If I can get some indication that this is desirable (and workable), I can attempt a PR :)

[lemoinem]

This is really something we need.

I have a very similar use case, and had something kind of working but based on CloudFront's Signed URLs instead of S3's.

But in the end, for very large files (30+ parts), CloudFront (or S3) just gets a case of the crazies and rejects all parts beyond an arbitrary (and transient) threshold.

Amazon Support just told me it would be better to go through S3's signed URLs, however, the C# SDK cannot generate appropriate Signed URLs. This is a major blocker for us and rather critical...

I'd be REALLY happy to see a new version of the SDK with this feature. Otherwise we'll have to hack it like you did, which is far from ideal.

[johnpreed]

We need this as well for the same reason - multipart upload. Is there any workaround? I need this now.

[damianh]

If anyone needs to move forward to get working this is what I did...

1. Import the necessary files from Examples of Signature Calculations Using C# (AWS Signature Version 4) into your project.

2. Use this function to generate the presigned url for each part. You may need to customize this to your needs (note: bucketname is in path and not part of host name here.)

    private static string GeneratePresignedUrlForMultipartUpload(
        string serviceUrl,
        string regionEndpointServiceName,
        string accessKey,
        string secretKey,
        string regionEndpoint,
        string bucketName,
        string key,
        int partNumber,
        string uploadId,
        TimeSpan expiresIn)
    {
        var expires = DateTime.UtcNow.Add(expiresIn);
        var endpointUrl = $"{serviceUrl}/{bucketName}/{key}"; // AmazonS3Config.ForcePathStyle is set to true
        var period = Convert.ToInt64((expires.ToUniversalTime() - DateTime.UtcNow).TotalSeconds);
        var query = $"partNumber={partNumber}" +
                    $"&uploadId={uploadId}" +
                    $"&{AWS4SignerBase.X_Amz_Expires}={HttpHelpers.UrlEncode(period.ToString())}";
        var headers = new Dictionary<string, string>();
        var signer = new AWS4SignerForQueryParameterAuth
        {
            EndpointUri = new Uri(endpointUrl),
            HttpMethod = "PUT",
            Service = regionEndpointServiceName,
            Region = regionEndpoint
        };
        var authorization = signer.ComputeSignature(
            headers,
            query,
            "UNSIGNED-PAYLOAD",
            accessKey,
            secretKey);
   
        return $"{endpointUrl}?{query}&{authorization}";
   

In anycase, this needs to be part of the SDK.

[lemoinem]

Just a quick note:

After talking with AWS Support, they developed and deployed a patch to CloudFront (worldwide, already rolled out).
My solution is now working and seems to be working fairly well, if I may say so myself.

I basically setup a CloudFront Distribution to upload to S3 (there are many tutorials and doc regarding this). I'm then using signed CloudFront Signed URLs (which are very well supported by the SDK) to do a generate the multipart upload URLs.

I realize it doesn't address exactly your use case (S3 Signed URLs for multipart uploads), but it provide a usable workaround in the meantime.

For us, the additional complexity of the setup is a non-issue since CloudFront also offer additional features such as supporting custom HTTPS certificates, allowing us to present only first party HTTPS URLs to our customers (for us, a browser is doing the upload). These are features that would be show stopper for us if they were missing.

[damianh]

@vellozzi @normj Would to you accept a PR for this?

[normj]

Yes, I would be open for a PR to support presigned url for multi part uploads.

[sandyghai]

I was managed to achieve this in serverless architecture by creating a Canonical Request for each part upload using Signature Version 4. You will find the document here https://sandyghai.github.io/AWS-S3-Multipart-Upload-Using-Presigned-Url/

[brett-matthews]

@sandyghai do you have an example code you could share? I have attempted to use @damianh example but I just receive a 403 when using the Signature on the S3 PUT. Don't know where to start in debugging...

[AhmadiRamin]

Is this part of the SDK now?

[AhmadiRamin]

@damianh your code isn't working, I'm getting 403 error uploading parts! it's been a week I'm looking for a way to upload multipart but still struggling, there are so many limitations, shame AWS!!