aws-sigv4 signature covers non-existent Expect: header

[apparentorder]

I did this

curl -v
 --header 'expect:'
 --header 'Content-Type: application/json'
 --data '[ ... foo ... ]'
 --aws-sigv4 aws:amz:eu-central-1:appsync
 --user 'akid:secret'
 'https://xxx.appsync-api.eu-central-1.amazonaws.com/graphql'

I expected the following

Correct AWS SigV4, i.e. expect is not part of curl's generated SignedHeaders, as it is not part of the emitted request headers, like this:

authorization: AWS4-HMAC-SHA256
 Credential=xxx/20230812/eu-central-1/appsync/aws4_request,
 SignedHeaders=content-type;host;x-amz-date,
 Signature=xxx

But what actually happens is this:

authorization: AWS4-HMAC-SHA256
 Credential=xxx/20230812/eu-central-1/appsync/aws4_request,
 SignedHeaders=content-type;expect;host;x-amz-date,
 Signature=xxx

Per man page, using --header 'expect:' should remove that header. The header is indeed not present, but it seems that by forcing the removal of this non-existent header, it somehow gets added to the list of headers to be considered for aws-sigv4.

Additional context:

Some AWS APIs just ignore this (e.g. sts), but others report failure, e.g. Appsync GraphQL with IAM Auth:

  "errors" : [ {
    "errorType" : "IncompleteSignatureException",
    "message" : "'expect' is named as a SignedHeader, but it does not exist in the HTTP request."
  } ]

This seems to happen in libcurl – I've discovered it while playing around with Hurl, which explicitly sets Expect:.

curl/libcurl version

curl 8.0.1 (aarch64-amazon-linux-gnu) libcurl/8.0.1 OpenSSL/3.0.8 zlib/1.2.11 libidn2/2.3.2 nghttp2/1.51.0

operating system

Amazon Linux 2023

[jay]

Try #11668

[apparentorder]

Just tested with 370aabb3ef0dc4fe1bcedd52ce706f4fceed3cea and it works now. Thanks for the quick response!

[jay]

With that commit if you send a no-value header to AWS like -H "foo;" (note the semi-colon which causes libcurl to translate it to 'foo:') and it sends "foo:" does AWS show you any signature error for that?

[apparentorder]

Note that different AWS endpoints seem to use different implementations (see initial report: sts for example ignores missing Expect header).

Testing against an Appsync GraphQL endpoint:

• Using expect; causes this request header to be present and empty, and part of SignedHeaders, but the API complains just the same: 'expect' is named as a SignedHeader, but it does not exist in the HTTP request.
• Using foo; causes the request header to be present and empty, and part of SignedHeaders, but the API responds successfully

Testing against STS:

• Using expect; causes this request header to be present and empty, and part of SignedHeaders, and STS fails with 417 Expectation Failed
• Using foo; causes the request header to be present and empty, and part of SignedHeaders, and the API responds successfully

By the way, using "Expect; " (with a trailing space) makes curl ignore that argument completely, i.e. no header is emitted. Should behave the same as "without space", or cause an error, I'd say.

[jay]

Thanks for testing. I tried to get the behavior as close as possible to the way http.c handles these headers. We want SignedHeaders to only include what is actually being sent.

'expect' is named as a SignedHeader, but it does not exist in the HTTP request.

Since the no-value Expect: header actually is sent in the HTTP request, from a libcurl perspective this is ok. The server error message is misleading.

417 Expectation Failed

Yes that is expected since any 'expect' header value other than 100-continue is undefined.

By the way, using "Expect; " (with a trailing space) makes curl ignore that argument completely, i.e. no header is emitted. Should behave the same as "without space", or cause an error, I'd say.

I forget whether that's a bug or intended. I'll take a look.