S3 request signing broken for S3-compatible services

[Sumo-MBryant]

If one is to use an S3-compatible service (e.g. for integration testing by mocking S3), signed requests are no longer valid as of this commit fbea753, which appears to use a heuristic to detect whether a request is being made to S3.

When using an S3-compatible service the ClientConfig.ServiceUrl might no longer satisfy the heuristic which appears to check the endpoint host name, and as such the resourcePath is double encoded and no longer signs correctly.

This is the heuristic:

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Util/S3Uri.cs

Line 35 in fbea753

private const string S3EndpointPattern = @"^(.+\.)?s3[.-]([a-z0-9-]+)\.";

and its use:

aws-sdk-net/sdk/src/Core/Amazon.Util/AWSSDKUtils.cs

Lines 318 to 324 in fbea753

	// S3 is a special case. For S3 skip the pre encode.
	// For everything else URL pre encode the resource path segments.
	if (!S3Uri.IsS3Uri(endpoint))
	{
	encodedSegments = encodedSegments.Select(segment => UrlEncode(segment, true));
	pathWasPreEncoded = true;
	}

Expected Behavior

When connecting to an S3-compatible service using ClientConfig.ServiceUrl, all S3 requests including those using characters that are transformed during URL encoding should be correctly signed according the version 4 signature scheme.

The CanonicalizeResourcePath() function should only single encode requests.

Current Behavior

The S3-compatible service rejects the request that include characters that are transformed during URL encoding.

Consider an attempt to store an object to a content bucket and object key: Alp?ha/Be%ta /Test&ing.

CanonicalizeResourcePath() was called with the following arguments:

endpoint = "http://192.168.99.100:33886/"
resourcePath = "/content/Alp?ha/Be%ta /Test&ing"
detectPreEncode = true

The function logged the following:

Double encoded /content/Alp?ha/Be%ta /Test&ing with endpoint http://192.168.99.100:33886/ for canonicalization: /content/Alp%253Fha/Be%2525ta%2520/Test%2526ing

Possible Solution

1. It appears that the that signing code is invoked from the S3 client code which intrinsically knows only single URL encoding is required. Expose the functionality through the AWS4Signer such that all S3 requests are only single encoded.

2. If that isn't possible, provide some means for configuration to disable the heuristic and double encoding.

Context

We have some integration tests that ensure our code correctly handles paths sensitive to URL encoding when storing objects in S3. Upon upgrade to AWSSDK.Core 3.3.21.15 and later, this test started to fail.

Your Environment

• AWSSDK.Core version used: 3.3.21.20
• Service assembly and version used: AWSSDK.S3 3.3.18
• Operating System and version: Microsoft Windows 10
• Targeted .NET platform: .NET Core 2.0

[sstevenkang]

Thanks for reporting this with great detail. We are not currently prioritizing work to accommodate for non-S3 services. We'll add this into our backlog as a feature request.

[Genbox]

@sstevenkang This has severe security complications as people implement workarounds by disabling signing. I would urge you to reconsider prioritizing this issue as it seems like a simple fix.

[mechanic22]

Does anyone know of a workaround for this issue?

[Sumo-MBryant]

Our original workaround was to pin AWSSDK.Core to an old version without the issue. This only works if you don't want to use new features such as AWSSDK.SecretsManager for which there is no version that is compatible with an "old" version of AWSSDK.Core.

We've disabled the affected tests for the time being.

[maxima120]

@Sumo-MBryant please share the Core version without the issue. I rolled back as far as 3.3.22 and it still fails.

[maxima120]

I would urge you to reconsider prioritizing this issue as it seems like a simple fix.

they don't like people using other platforms. Since Amazon pays devs' salaries, the "open sourceness" is a farce.

[Sumo-MBryant]

We used 3.3.19 to avoid the problem.

[langebo]

Is there any chance we can at least get an ETA or is this welcome issue for amazon? We use the workaround described by @Sumo-MBryant to support foreign s3 compatible storage providers.

We would also like to add healthchecks in regard of network connectivity to s3, but the library we use requires a higher version of AWSSDK.Core.

[Adriien-M]

I downgraded Core library to 3.3.19 as suggested by @Sumo-Bryant but honestly, it is not acceptable from Amazon to not fix this issue...

[kevinolinger]

Downgrading is not a option if you need to write to a Snowball Edge endpoint, as the lower version doesn't allow disabling chunk encoding which is required for the endpoint. This bug has been open for a year and half now, I'm starting think AWS doesn't care too much about the dotnet SDK.