Repository navigation
Description
I did this
Disclaimer: I ran into this while using AI coding agents during the construction of a conformance test suite for object storage clients. I was using curl as a reference there. I have independently confirmed the findings by running all the reproducers and examining the sources, but all the original research was done by the coding agent. The below report is all human-written, but based on research with coding agents.
I found an issue while trying to use curl on S3 Express One Zone buckets. These can have lower latency and are have behavior closer to standard filesystem directories (and are hence also called directory buckets). To summarize:
- Starting from 8.14, curl encodes an already-encoded path like
/a%23b.txtfor S3 Express another time. This is necessary for some services (like S3 Tables), hence why this was implemented in http_aws_sigv4: Fixes for sigv4 url encoding and canonicalization #17129, but not for S3 and S3 Express. For S3 it works because there is a special opt-out (see cause section below), but that opt-out namess3-express, while the S3 Express SigV4 name is actuallys3express(without the space).
Full reproduction
- Set up a directory bucket in S3 Express One Zone and set up your environment variables so that you are logged in with the right identity (not trivial, unfortunately, we're assuming you're doing it in eu-central-1 here, in AZ euc1-az1) .
- Create a session:
curl -sS \
"https://<your directory bucket name>.s3express-euc1-az1.eu-central-1.amazonaws.com/?session" \
--aws-sigv4 "aws:amz:eu-central-1:s3express" \
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
-H "x-amz-security-token: $AWS_SESSION_TOKEN" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" > /tmp/session.xml
This works! Now we can put something in there (assuming we read the session.xml and put the right stuff in SAK, SSK and STOK env vars (omitted here):
curl -sS -X PUT \
--data-binary hello \
"https://<your directory bucket name>.s3express-euc1-az1.eu-central-1.amazonaws.com/demo/plain.txt" \
--aws-sigv4 "aws:amz:eu-central-1:s3express" \
--user "$SAK:$SSK" \
-H "x-amz-s3session-token: $STOK" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
-w 'HTTP %{http_code}\n'
This gives HTTP 200. But if we now try to do a key like the following:
curl -sS -X PUT \
--data-binary hello \
"https://<your directory bucket name>.s3express-euc1-az1.eu-central-1.amazonaws.com/demo/a%23b.txt" \
--aws-sigv4 "aws:amz:eu-central-1:s3express" \
--user "$SAK:$SSK" \
-H "x-amz-s3session-token: $STOK" \
-H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
-w 'HTTP %{http_code}\n'
You get 403 with code SignatureDoesNotMatch.
I expected the following
For the final command to succeed with status code 200.
Cause
static bool should_urlencode(struct Curl_str *service_name)
{
/*
* These services require unmodified (not additionally URL-encoded) URL
* paths.
* should_urlencode == true is equivalent to should_urlencode_uri_path
* from the AWS SDK. Urls are already normalized by the curl URL parser
*/
if(curlx_str_cmp(service_name, "s3") ||
curlx_str_cmp(service_name, "s3-express") ||
curlx_str_cmp(service_name, "s3-outposts")) {
return FALSE;
}
return TRUE;
}I'm guessing the s3-express was a typo that was simply never noticed. It should be s3express. Furthermore, it doesn't cover s3-object-lambda. You can see the Smithy definitions and search for disableDoubleEncoding to find which services need the opt-out.
Updating that function to include s3express instead should fix the issue.
curl/libcurl version
curl 8.22.0 (x86_64-pc-linux-gnu) libcurl/8.22.0 OpenSSL/3.6.4 zlib/1.3.2 brotli/1.2.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.70.0 ngtcp2/1.23.0 nghttp3/1.16.0 mit-krb5/1.22.2
Release-Date: 2026-09-02
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt mqtts pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz PSL SPNEGO SSL threadsafe UnixSockets zstd
- All versions starting from 8.14 (which includes the work of http_aws_sigv4: Fixes for sigv4 url encoding and canonicalization #17129), fail. (I explicitly tested only master 47c08c8, 8.22 and 8.14).
- 8.13.0 and earlier do not fail (also tested 8.5.0, which also succeeds)
operating system
Linux ys7imh9-pop 7.1.5-76070105-generic #202607241434~1787955023~24.04~38a419f SMP PREEMPT_DYNAMIC Fri A x86_64 x86_64 x86_64 GNU/Linux
Activity
tiptenbrink commentedon Sep 26, 2026
Pinging also @nbaws, who was the original author of the code.
jay commentedon Sep 26, 2026
...
Ok sounds good. Would you be willing to submit a PR?
http_aws_sigv4: fix the s3express service name
http_aws_sigv4: fix the s3express service name
http_aws_sigv4: fix the s3express service name
tiptenbrink commentedon Sep 26, 2026
Oops, PR in my fork that I used to verify it closed the issue automatically. Reopened it. PR as requested, hope it's okay!
http_aws_sigv4: fix the s3express service name