aws-sigv4 request signature does not match expected value

[jaw111]

I did this

I'm trying to sign GET requests to the SPARQL endpoint of an Amazon Neptune cluster where query string parameters are used to pass information. If the parameter values contain spaces, forward slashes /, or colons :, then the signature calculated by curl does not match what the server expects.

An example request:

curl "https://${NEPTUNE_HOSTNAME}:8182/sparql/gsp" \
  --url-query "graph=urn:uuid:6e8bc430-9c3a-11d9-9669-0800200c9a66" \
  --aws-sigv4 "aws:amz:${AWS_REGION}:neptune-db" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}"

Gets a response like this:

{
  "code": "AccessDeniedException",
  "requestId": "96c53027-1a0c-96a1-5a8b-18a20bee9465",
  "detailedMessage": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."
}

I expected the following

The calculated signature should be correct.

curl/libcurl version

curl 8.2.1 (x86_64-w64-mingw32) libcurl/8.2.1 OpenSSL/3.1.2 (Schannel) zlib/1.3 brotli/1.1.0 zstd/1.5.5 WinIDN libssh2/1.11.0 nghttp2/1.55.1 ngtcp2/0.19.0 nghttp3/0.15.0
Release-Date: 2023-07-26
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe UnixSockets zstd

operating system

Windows 64-bit

[bagder]

Combined with #7559, it appears as if the list of "bad" characters is now *, : and /. I figure it might need some closer reading of the "specs" to figure out the complete list and how they should be treated correctly.

The 7559 problem is a documented known bug

[bagder]

I suspect this is because the %HH-encoding curl does is using lowercase letters, and not uppercase as the "spec" seems to suggest it should use.

/cc @outscale-mgo

[bagder]

I'm pretty sure we need to "canonicalize" the URL query to make aws-sigv4 work proper. Because right now a URL like: https://example.com/?foo=-*/ is not even going to get those three last characters URL encoded.

[outscale-mgo]

It's exactly that, the URL need to be encoded manually.

[jaw111]

OK, I tried manually encoding the parameter value and that works:

curl "https://${NEPTUNE_HOSTNAME}:8182/sparql/gsp?graph=urn%3Auuid%3A6e8bc430-9c3a-11d9-9669-0800200c9a66" \
  --aws-sigv4 "aws:amz:${AWS_REGION}:neptune-db" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}"

Though ideally using --url-query option should take care of the canonicalization/normalization.

RFC 3986 section 2.1, where this encoding is defined, states:

For consistency, URI producers and normalizers should use uppercase hexadecimal digits for all percent-encodings.

[bagder]

#11806 is my attempt at a fix