--aws-sigv4 doesn't sign requests with query string parameters with no value correctly

[dzaslavskiy]

This request gives an error:

curl --user "<key>:<secret>" --aws-sigv4 aws:amz:<region>:es -k -X GET 'https://<es-host>/_cat/indices?v'

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

Removing the query string or or adding a value to the query parameter fixes the issue

curl --user "<key>:<secret>" --aws-sigv4 aws:amz:<region>:es -k -X GET 'https://<es-host>/_cat/indices'

green open index-name index-uuid 5 1       16    0 285.4kb 142.7kb

curl --user "<key>:<secret>" --aws-sigv4 aws:amz:<region>:es -k -X GET 'https://<es-host>/_cat/indices?v=true'

green open index-name index-uuid 5 1       16    0 285.4kb 142.7kb

curl/libcurl version

curl 7.79.1 (x86_64-w64-mingw32) libcurl/7.79.1 OpenSSL/1.1.1l (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.0 libidn2/2.3.1 libssh2/1.10.0 nghttp2/1.45.1
Release-Date: 2021-09-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP zstd

operating system

windows 10; git-bash

[bagder]

/cc @outscale-mgo

[outscale-mgo]

I'm presently trying to find the bug, out of curiosity, which call exactly did you made (having the full URL could held debug I guess) ?

[dzaslavskiy]

$ ./curl --user "..." --aws-sigv4 aws:amz:us-east-1:es -X GET "https://search-curl-test-yk7vggewbojld4psjkwqnjifry.us-east-1.es.amazonaws.com/_cat/indices?v=true"

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1 Q9Me-vJiQXGZwplpTsTxpg   1   0          1            0        5kb            5kb

$ ./curl --user "..." --aws-sigv4 aws:amz:us-east-1:es -X GET "https://search-curl-test-yk7vggewbojld4psjkwqnjifry.us-east-1.es.amazonaws.com/_cat/indices?v"

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

./curl --user "..." --aws-sigv4 aws:amz:us-east-1:es -X GET "https://search-curl-test-yk7vggewbojld4psjkwqnjifry.us-east-1.es.amazonaws.com/_cat/indices?v="

health status index     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana_1 Q9Me-vJiQXGZwplpTsTxpg   1   0          1            0        5kb            5kb

3. d. in the aws docs mentions this case:

For each parameter, append the URI-encoded parameter name, followed by the equals sign character (=), followed by the URI-encoded parameter value. Use an empty string for parameters that have no value.

Seems like an equals sign is not being appended here.

EDIT: I think there may some other issues here. The query string is not being fully canonicalized. For example, the docs require sorting query parameters by field name in ascii order.

[outscale-mgo]

thanks for the reply, I've made a Pull Request that should improve url encoding here: #7600
Note that presently the PR doesn't fix your issue, but should do so, so I'm going to rework the PR to handle this issue, and sort the parameters.

[outscale-mgo]

just to let you know I've update the PR, would be nice if you test it, and tell us if it work with your use case.

[dzaslavskiy]

I have done a few tests on that branch and it works great.