Signer does not match official Signature Version 4 Test Suite

[mhart]

There is a test suite provided here:

http://docs.aws.amazon.com/general/latest/gr/signature-v4-test-suite.html

And there is also documentation provided here:

http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

That says "Normalize URI paths according to RFC 3986 by removing redundant and relative path components".

As the documentation implies above, and as the test suite shows, there are some examples of requests (in get-relative.sreq and get-slashes.sreq in the test suite, among others) where the request provided is, say, GET /foo/.. http/1.1 and the expected canonical request (in get-relative.creq) is resolved to /.

However, this is not what the aws-sdk does:

var req = new AWS.S3().getObject({Bucket: 'bucket', Key: '//key//..//whatever/.'}).build()
console.log(new AWS.Signers.V4(req.httpRequest, 's3').canonicalString())

Results in:

GET
///key//..//whatever/.

host:bucket.s3.amazonaws.com
x-amz-date:Sun, 27 Dec 2015 16:02:10 GMT

host;x-amz-date
e3b0c44298fc1c149afbf4c5113fb92427ae41e4649b934ca495991b7852b855

If the SDK was resolving according to the documentation (and test suite), then the canonical request would be /whatever/ instead of ///key//..//whatever/. – so the SDK doesn't actually pass the test suite (I can provide you with the exact tests it fails if you like).

An important thing to note here is that it appears that the actual AWS services themselves also don't appear to adhere to the documentation or test suite.

So, I'm left wondering if it's actually the test suite and documentation themselves that are incorrect – because both the SDK and actual AWS services appear not to do any normalisation whatsoever. However, it would be good to clarify this and determine which source is actually correct!

[mhart]

To add to this, the managing of duplicate query keys is a case where the SDK does match the behaviour of the Test Suite, but neither of these match the behaviour of the live S3 service.

To show this, try the following (from get-vanilla-query-order-value.req in the test suite):

console.log(new AWS.Signers.V4(new AWS.HttpRequest({path: '/?foo=b&foo=a'})).canonicalString())

Which gives:

POST
/
foo=a&foo=b



e3b0c44298fc1c149afbf4c8996fb95827ae41e4649b934ca495991b7852b855

As you can see, the SDK sorts the query key foo based on its value – this matches the expected outcome in the test suite too.

However, if you make a similar query directly to S3 (replace AKIDEXAMPLE with a valid Access Key):

require('https').request({
  hostname: 's3.amazonaws.com',
  path: '/?foo=b&foo=a&' +
    'X-Amz-Expires=86400&' +
    'X-Amz-Date=20151226T200800Z&' +
    'X-Amz-Algorithm=AWS4-HMAC-SHA256&' +
    'X-Amz-Credential=AKIDEXAMPLE%2F20151226%2Fus-east-1%2Fs3%2Faws4_request&' +
    'X-Amz-SignedHeaders=host&' +
    'X-Amz-Signature=5'
}, res => res.pipe(process.stdout)).end()

It responds with:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
...
<CanonicalRequest>
GET
/
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=AKIDEXAMPLE%2F20151226%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Date=20151226T200800Z&amp;X-Amz-Expires=86400&amp;X-Amz-SignedHeaders=host&amp;foo=b
host:s3.amazonaws.com

host
UNSIGNED-PAYLOAD
</CanonicalRequest>
...
</Error>

Which, if you format it a little nicer, you can see it expects the canonical query string to be:

X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=AKIDEXAMPLE%2F20151226%2Fus-east-1%2Fs3%2Faws4_request&
X-Amz-Date=20151226T200800Z&
X-Amz-Expires=86400&
X-Amz-SignedHeaders=host&
foo=b

So it a) doesn't sort the query key foo based on value and b) doesn't even expect there to be multiple instances of it – it only uses the first instance with value b.

So this is an example of where the SDK doesn't match the live S3 service in signing behaviour – but it does seem to match the Test Suite.

[mhart]

Ugh, the plot thickens even further...

So if you try another service, for example, es.us-east-1.amazonaws.com – it appears to follow the AWS Test Suite behaviour exactly – that is, the service returns a Canonical String including a fully resolved path (unlike this SDK), as well as the sorted query strings (like this SDK).

So something's definitely fishy. The S3 service differs from (at least) the ES service, as well as the docs/tests – and this SDK seems to be somewhere in the middle.

[mhart]

And even further...

It seems that many services don't decode the path correctly and so they have an incorrect expectation for the Canonical String. This also makes it impossible to include certain characters in the path of a request to these services because there's no way to sign them correctly.

The TL;DR version is that S3 gets this particular thing right, and if you request /%3Fa=b%20c it will expect a canonical string for /%3Fa%3Db%20c – but for ES (and many other services) if you request the same thing it will expect a canonical string for /%253Fa%3Db%2520c – double encoding the percent signs.

Here's a simple example using curl (you can add -v to ensure that curl isn't doing any weird escaping itself – you also get the same result making the request in Node.js):

$ curl 'https://es.us-east-1.amazonaws.com/_search/%3Fa=b%20c' \
-H 'Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20151227/us-east-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=7' \
-H 'X-Amz-Date: 20151227T231859Z'

<InvalidSignatureException>
...
The Canonical String for this request should have been
'GET
/_search/%253Fa%3Db%2520c

...

See how the expected Canonical String has not parsed the URL correctly, and actually encodes the % character to be %25 instead of using the original url-encoded URL?

This makes it impossible, for example, to include a query that has a '?' character in it – because if you URL encode it to %3F, as above, then the Canonical String will be wrong, and if you don't URL encode it, then AWS will assume it's the query string divider:

curl 'https://es.us-east-1.amazonaws.com/_search/?a=b%20c' \
-H 'Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20151227/us-east-1/es/aws4_request, SignedHeaders=host;x-amz-date, Signature=7' \
-H 'X-Amz-Date: 20151227T231859Z'

<InvalidSignatureException>
...
The Canonical String for this request should have been
'GET
/_search/
a=b%20c
...

Also note how the %20 is left alone in this case – and the % is not double encoded – so the service is clearly treating the query string differently from the base path.

Funnily enough, S3 seems to get this one right. To wit:

$ curl 'https://s3.amazonaws.com/_search/%3Fa=b%20c' \
-H 'Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20151227/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=7' \
-H 'X-Amz-Date: 20151227T234059Z' \
-H 'X-Amz-Content-Sha256: 1234123412341234'

<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
...
<CanonicalRequest>
GET
/_search/%3Fa%3Db%20c

...

[mhart]

Alright, have dug further... It seems that everything's broken in this regard. None of the live services, or tests, or SDKs seem to show any consistency with how they manage the Canonical String – especially when it comes to URL encoding and query string parameters.

Here's a table showing how each service expects you to create the canonical string (for the path and query portions)

Original Query	S3	ES (and all? others)	Tests	Notes
/%20	/%20	/%2520	/%20	The tests and S3 agree, but non-S3 services double-escape
/%2a	/%2A	/%252a	?	S3 capitalizes the encoding
/%41	/A	/%2541	?	S3 decodes non-reserved characters, except...
/%2f	//	error	?	S3 decodes slashes too, wtf!? It seems that no service could support a query that requires a %2F in the path.
/ü	/%C3%BC	/%C3%BC	?	URL encoding is correct
/€	/%C2%AC	/%C2%AC	?	URL encoding is incorrect for all services, should be %E2%82%AC. %C2%AC decodes to ¬
/ü%41	/%EF%BF%BD	/%C3%BC%2541	?	S3 totally screws up here, using the encoding for U+FFFD
/ü%41%41%41%41	/%EF%BF%BD	(as above with more trailing %2541)	?	S3 still only with a single U+FFFD character
/ü%41%41%41%41%41	/%EF%BF%BDAAAAA	(as above with more trailing %2541)	?	S3 suddenly relents and decodes the %41 correctly after 5 in a row
/?a=b&a=B	/ a=b	/ a=B&a=b	/ a=B&a=b	The tests and non-S3 services agree here and sort – S3 just chooses the first query param it sees
//a/b/..//c/.?a=b	//a/b/..//c/. a=b	/a/c a=b	/a/c a=b	The tests and non-S3 services agree. S3 does not.
/%41?%41=%41	/A A=A	/%2541 A=A	?	non-S3 services decode the query string, but not the path... wtf.
/%41?%2f=%2f	/A %2F=%2F	/%2541 %2F=%2F	?	S3 does not decode %2F to / if it's in the query, only the path (see earlier). All services also capitalize the encoding.
/ü?a=ü	/%C3%BC a=%EF%BF%BD	/%C3%BC a=%C3%BC	?	S3 manages to encode the path correctly, but screws up the query string.

[mhart]

As per usual, the forums have been an absolute ghost town on this whole thing... https://forums.aws.amazon.com/thread.jspa?messageID=693401

[chrisradek]

@mhart
Thanks for all the work and evidence you've provided. I will look into this and bring it up internally to figure out why there are such discrepancies. I apologize for the delayed response.

[kyeotic]

@chrisradek has there been any movement on this?

[mmelvin0]

@mhart I came here trying to debug this exact same discrepancy between Postman's AWS implementation and my own in-house library. Thank you for the detailed breakdown. I'm abandoning all hope.