aws-sigv4 generates extra X-Amz-Date header (possibly different)

[apparentorder]

I did this

curl -v --aws-sigv4 aws:amz:foo --header 'X-Amz-Date: 20230813T061344Z' --user foo:bar https://httpbin.org/post

I expected the following

The provided X-Amz-Date is used for signature calculation and is emitted once.

But actually it's emitted twice:

> GET /post HTTP/2
> Host: httpbin.org
> authorization: AWS4-HMAC-SHA256 Credential=foo/20230825/foo/httpbin/aws4_request, SignedHeaders=host;x-amz-date, Signature=bcecfc18afb2e19fe031bfd2bbaa68faffb3ff3cc3e990b014217422f422860a
> x-amz-date: 20230825T202910Z
> user-agent: curl/7.81.0
> accept: */*
> x-amz-date: 20230813T061344Z

On some curl versions (tested: 7.81.0), the first X-Amz-Date is actually generated with a different date (the current date), and that current date is used for signature calculation instead of the X-Amz-Date given on the command line.

Other versions (tested: 8.1.2) emit the given X-Amz-Date twice as well, but both with the given value; in that case, the signature is calculated as expected.

As this is a rare / weird use-case, maybe it should just be added to the list of known bugs (?)

curl/libcurl version

curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.14

curl 8.1.2 (x86_64-apple-darwin22.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.51.0

operating system

curl 8.1.2 on macOS 13.5, or
Darwin xxx 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64 x86_64

curl 7.81.0 on Ubuntu 22.04.2, or
Linux ip-10-0-0-148 5.19.0-1025-aws #26~22.04.1-Ubuntu SMP Mon Apr 24 01:58:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[jay]

Is there a reason to support user supplied X-Amz-Date?

/cc @outscale-mgo

[apparentorder]

Reasons I can think of are

• testing, i.e. achieve a stable signature value to check against – this is what I was trying to do (in a libcurl client), and this worked on 8.x and I was kinda surprised when 7.x produced different results
• security experiments to see how a server behaves with offset and malformatted dates

[bagder]

Is there a reason to support user supplied X-Amz-Date?

curl supports custom headers of any kind, outputting two like this is at least a bug.

Not using the user provided date header seems like an oversight, as exactly like @apparentorder did, I would have expected curl to work with it.

[outscale-mgo]

It's clearly a bug, I was pretty sure I was handling the case in the code, but I guess I didn't test this enough, sorry for the bug, I'll look at it asap.

This is what AWS "standard" say about date:

X-Amz-Date

    The date that is used to create the signature. The format must be ISO 8601 basic format (YYYYMMDD'T'HHMMSS'Z'). For example, the following date time is a valid X-Amz-Date value: 20120325T120000Z.

    Condition: X-Amz-Date is optional for all requests; it can be used to override the date used for signing requests. If the Date header is specified in the ISO 8601 basic format, X-Amz-Date is not required. When X-Amz-Date is used, it always overrides the value of the Date header. For more information, see [Elements of an AWS API request signature](https://docs.aws.amazon.com/IAM/latest/UserGuide/signing-elements.html) in the IAM User Guide.

and then:

Specifies the date and time of the request. Including the date and time in a request helps prevent third parties from intercepting your request and resubmitting it later. The date that you specify in the credential scope must match the date of your request.

The time stamp must be in UTC and use the following ISO 8601 format: YYYYMMDDTHHMMSSZ. For example, 20220830T123600Z. Do not include milliseconds in the time stamp.

You can use a date or an x-amz-date header, or include x-amz-date as a query parameter. If we can't find an x-amz-date header, then we look for a date header.

[jay]

Thanks