Skip to content

aws-sigv4: S3 Express (s3express) paths get encoded twice #23120

Closed
tiptenbrink/curl
#4

Description

@tiptenbrink

I did this

Disclaimer: I ran into this while using AI coding agents during the construction of a conformance test suite for object storage clients. I was using curl as a reference there. I have independently confirmed the findings by running all the reproducers and examining the sources, but all the original research was done by the coding agent. The below report is all human-written, but based on research with coding agents.

I found an issue while trying to use curl on S3 Express One Zone buckets. These can have lower latency and are have behavior closer to standard filesystem directories (and are hence also called directory buckets). To summarize:

  • Starting from 8.14, curl encodes an already-encoded path like /a%23b.txt for S3 Express another time. This is necessary for some services (like S3 Tables), hence why this was implemented in http_aws_sigv4: Fixes for sigv4 url encoding and canonicalization #17129, but not for S3 and S3 Express. For S3 it works because there is a special opt-out (see cause section below), but that opt-out names s3-express, while the S3 Express SigV4 name is actually s3express (without the space).

Full reproduction

  • Set up a directory bucket in S3 Express One Zone and set up your environment variables so that you are logged in with the right identity (not trivial, unfortunately, we're assuming you're doing it in eu-central-1 here, in AZ euc1-az1) .
  • Create a session:
curl -sS \
  "https://<your directory bucket name>.s3express-euc1-az1.eu-central-1.amazonaws.com/?session" \
  --aws-sigv4 "aws:amz:eu-central-1:s3express" \
  --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
  -H "x-amz-security-token: $AWS_SESSION_TOKEN" \
  -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" > /tmp/session.xml

This works! Now we can put something in there (assuming we read the session.xml and put the right stuff in SAK, SSK and STOK env vars (omitted here):

curl -sS -X PUT \
  --data-binary hello \
  "https://<your directory bucket name>.s3express-euc1-az1.eu-central-1.amazonaws.com/demo/plain.txt" \
  --aws-sigv4 "aws:amz:eu-central-1:s3express" \
  --user "$SAK:$SSK" \
  -H "x-amz-s3session-token: $STOK" \
  -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
  -w 'HTTP %{http_code}\n'

This gives HTTP 200. But if we now try to do a key like the following:

curl -sS -X PUT \
  --data-binary hello \
  "https://<your directory bucket name>.s3express-euc1-az1.eu-central-1.amazonaws.com/demo/a%23b.txt" \
  --aws-sigv4 "aws:amz:eu-central-1:s3express" \
  --user "$SAK:$SSK" \
  -H "x-amz-s3session-token: $STOK" \
  -H "x-amz-content-sha256: UNSIGNED-PAYLOAD" \
  -w 'HTTP %{http_code}\n'

You get 403 with code SignatureDoesNotMatch.

I expected the following

For the final command to succeed with status code 200.

Cause

lib/http_aws_sigv4.c:263:

static bool should_urlencode(struct Curl_str *service_name)
{
  /*
   * These services require unmodified (not additionally URL-encoded) URL
   * paths.
   * should_urlencode == true is equivalent to should_urlencode_uri_path
   * from the AWS SDK. Urls are already normalized by the curl URL parser
   */
  if(curlx_str_cmp(service_name, "s3") ||
     curlx_str_cmp(service_name, "s3-express") ||
     curlx_str_cmp(service_name, "s3-outposts")) {
    return FALSE;
  }
  return TRUE;
}

I'm guessing the s3-express was a typo that was simply never noticed. It should be s3express. Furthermore, it doesn't cover s3-object-lambda. You can see the Smithy definitions and search for disableDoubleEncoding to find which services need the opt-out.

Updating that function to include s3express instead should fix the issue.

curl/libcurl version

curl 8.22.0 (x86_64-pc-linux-gnu) libcurl/8.22.0 OpenSSL/3.6.4 zlib/1.3.2 brotli/1.2.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.70.0 ngtcp2/1.23.0 nghttp3/1.16.0 mit-krb5/1.22.2
Release-Date: 2026-09-02
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt mqtts pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz PSL SPNEGO SSL threadsafe UnixSockets zstd

operating system

Linux ys7imh9-pop 7.1.5-76070105-generic #202607241434~1787955023~24.04~38a419f SMP PREEMPT_DYNAMIC Fri A x86_64 x86_64 x86_64 GNU/Linux

Activity

  1. tiptenbrink commented on Sep 26, 2026

    @tiptenbrink
    ContributorAuthor

    Pinging also @nbaws, who was the original author of the code.

  2. jay commented on Sep 26, 2026

    @jay
    Member
    • For S3 it works because there is a special opt-out (see cause section below), but that opt-out names s3-express, while the S3 Express SigV4 name is actually s3express (without the space).

    ...

    I'm guessing the s3-express was a typo that was simply never noticed. It should be s3express. Furthermore, it doesn't cover s3-object-lambda. You can see the Smithy definitions and search for disableDoubleEncoding to find which services need the opt-out.

    $ curl -k -fsS -L https://raw.githubusercontent.com/smithy-lang/smithy-rs/2e6eae6a56ea269a5ad7efaa5b3855c4648ebb83/aws/sdk/aws-models/s3.json | jq -r '.. | objects | select(.disableDoubleEncoding == true and has("signingName")) | .signingName' | sort | uniq
    s3
    s3-object-lambda
    s3-outposts
    s3express
    

    Ok sounds good. Would you be willing to submit a PR?

  3. added 3 commits that reference this issue on Sep 26, 2026
    fd648ff
    923ecd9
    7a90685
  4. tiptenbrink commented on Sep 26, 2026

    @tiptenbrink
    ContributorAuthor

    Oops, PR in my fork that I used to verify it closed the issue automatically. Reopened it. PR as requested, hope it's okay!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions