Uploading Files to s3 sometimes fails due to Amazon.S3.AmazonS3Exception: 'The request signature we calculated does not match the signature you provided'

[MhamzaM]

Describe the bug

The below code was working fine until I was using .net AWSSDK.S3 Version="3.7.8.2", after I updated the version to "3.7.411.7" the fileTransferUtility.Upload() function started resulting in unexpected behavior. Sometime the call get successful and sometimes it throws an exception with message "The request signature we calculated does not match the signature you provided. Check your key and signing method".

public static async Task UploadFileToAWS(AWSAppConfig aWSAppConfig, int locationId, string sourceFileKey, string bucketName, MemoryStream file, string path = null)
{
    try
    {
        
        IAmazonS3 s3Client = GetAmazonS3Client(aWSAppConfig);

        var fileTransferUtility = new TransferUtility(s3Client);
        
        if (!await AmazonS3Util.DoesS3BucketExistV2Async(s3Client, bucketName))
        {
            var request = new PutBucketRequest
            {
                BucketName = bucketName,
                UseClientRegion = true,
                ObjectOwnership = ObjectOwnership.ObjectWriter,

            };
            await s3Client.PutBucketAsync(request);
            
            DeletePublicAccessBlockRequest deletePublicAccessBlockRequest = new DeletePublicAccessBlockRequest
            {
                BucketName = bucketName
            };
            await s3Client.DeletePublicAccessBlockAsync(deletePublicAccessBlockRequest);
       
        }

        if(path != null)
        {
            bucketName += "/" + path;
        }

        var fileTransferUtilityRequest = new TransferUtilityUploadRequest
        {
            BucketName = bucketName,
            InputStream = file,
            StorageClass = S3StorageClass.StandardInfrequentAccess,
            PartSize = 6291456, // 6 MB.
            Key = sourceFileKey,
            CannedACL = S3CannedACL.PublicRead
            
        };

        fileTransferUtility.Upload(fileTransferUtilityRequest);                
    }
    catch (Exception)
    {
        throw;
    }
}

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The TransferUtility.Upload should always upload the file instead of throwing exception. I have downgraded the version to the previous one and it's now working flawlessly.

Current Behavior

Amazon.S3.AmazonS3Exception: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'

Reproduction Steps

public static async Task UploadFileToAWS(AWSAppConfig aWSAppConfig, int locationId, string sourceFileKey, string bucketName, MemoryStream file, string path = null)
{
    try
    {
        
        IAmazonS3 s3Client = GetAmazonS3Client(aWSAppConfig);

        var fileTransferUtility = new TransferUtility(s3Client);
        
        if (!await AmazonS3Util.DoesS3BucketExistV2Async(s3Client, bucketName))
        {
            var request = new PutBucketRequest
            {
                BucketName = bucketName,
                UseClientRegion = true,
                ObjectOwnership = ObjectOwnership.ObjectWriter,

            };
            await s3Client.PutBucketAsync(request);
            
            DeletePublicAccessBlockRequest deletePublicAccessBlockRequest = new DeletePublicAccessBlockRequest
            {
                BucketName = bucketName
            };
            await s3Client.DeletePublicAccessBlockAsync(deletePublicAccessBlockRequest);
       
        }

        if(path != null)
        {
            bucketName += "/" + path;
        }

        var fileTransferUtilityRequest = new TransferUtilityUploadRequest
        {
            BucketName = bucketName,
            InputStream = file,
            StorageClass = S3StorageClass.StandardInfrequentAccess,
            PartSize = 6291456, // 6 MB.
            Key = sourceFileKey,
            CannedACL = S3CannedACL.PublicRead
            
        };

        fileTransferUtility.Upload(fileTransferUtilityRequest);                
    }
    catch (Exception)
    {
        throw;
    }
}

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 Version="3.7.411.7"

Targeted .NET Platform

.net 8

Operating System and version

windows 11

[dscpinheiro]

Are you using S3 or a 3rd-party implementation? We made a change last week so that the SDK automatically calculates checksums for all upload calls: #3610 (here's a similar report - where they were using MinIO: #3615)

Ignore the above message, the error you're seeing is not related to the change we made last week. Starting in version 3.7.100 of AWSSDK.S3, you cannot use part of the key prefix in the bucket name (see #2622 (comment) for more details).

To fix your problem, you'll need to change your code to:

if (path != null)
{
    sourceFileKey = path + "/" + sourceFileKey;
}

var fileTransferUtilityRequest = new TransferUtilityUploadRequest
{
    BucketName = bucketName,
    InputStream = file,
    StorageClass = S3StorageClass.StandardInfrequentAccess,
    PartSize = 6291456, // 6 MB.
    Key = sourceFileKey,
    CannedACL = S3CannedACL.PublicRead
};

[MhamzaM]

@dscpinheiro Where is this change documented? I couldn't find it.

[dscpinheiro]

No, there are more details in the comment I linked but the previous behavior worked by accident.

In retrospect we should've communicated it better, but the current version of the S3 package is working as expected.