net.s3, net.http: add an S3-compatible client and a scheme-handler dispatch in http.fetch

[davlgd]

S3 is one of the most widely deployed cloud APIs across the major providers and self-hosted object stores. Having a first-class client in V, with no third-party dependencies and no need to shell out to aws-cli, makes the language a credible choice for cloud workloads.

I am proposing this module for native integration under vlib/net/s3 because of how common the stack is, but please tell me if you would rather see it land under vlib/x/ first. The module is self-contained either way.

vlib/net/s3 is an S3-compatible client with no third-party dependencies. Signing is built on crypto.hmac and crypto.sha256 only, and is verified against the AWS-published Signature V4 reference vectors.

It exposes single-shot ops (put, get, get_string, stat, exists, size, delete, presign, list), bucket ops (create_bucket, delete_bucket, bucket_exists with client-side validation of S3 bucket-naming rules), and multipart upload. The multipart layer covers upload_file (auto single/multipart), upload_file_multipart, upload_bytes_multipart, plus a streaming MultipartUploader for chunks generated on the fly, with up to Client.queue_size parallel part uploads with per-part retry and exponential back-off.

A small File handle (Client.file(key) returning a reference type that forwards to the client via f.read(), f.write(), f.presign(), and friends) is provided for ergonomic call sites. s3.fetch('s3://bucket/key', ...) acts as a direct entry point, plus an init() registers the s3:// scheme with net.http so http.fetch(url: 's3://...') works once the module is imported.

Credentials.from_env() resolves credentials with first-non-empty resolution across S3_, AWS_, CELLAR_ADDON_, SCW_, B2_, R2_, SPACES_* so the same client targets the major hosted and self-hosted S3 providers with no reconfiguration.

A short usage example:

import net.s3
c := s3.new_client(s3.Credentials.from_env())
c.put('hello.txt', 'Hi from V!'.bytes())!
text := c.get_string('hello.txt')!
url := c.presign('hello.txt', expires_in: 3600)!

vlib/net/http/http.v adds a tiny scheme-handler registry to let modules like net.s3 plug into http.fetch for non-HTTP schemes. register_scheme(scheme, handler) and unregister_scheme(scheme) let modules claim a scheme from their own init(). fetch() looks up a handler for any non-empty, non-HTTP/HTTPS scheme and delegates if one is registered. Otherwise the call path is byte-for-byte identical to master, with no behaviour change for plain HTTP calls. The new __global scheme_handlers map is opt-in via @[has_globals] on the module, per V's standard convention.

On the architecture side, client.v holds the Client struct, single-shot CRUD and the do_http core. multipart.v holds the multipart upload, the parallel worker pool and the MultipartUploader. bucket.v handles bucket lifecycle and bucket-name validation. signer.v is pure SigV4 (no HTTP coupling) and exposes sign_request and presign_url. credentials.v holds the Credentials struct, env merging and region resolution. encoding.v has URI percent-encoding, hex helpers and CRLF guards. errors.v defines S3Error and parses the XML envelope. fetch.v implements the s3.fetch('s3://...') URL helper. http_bridge.v registers the s3:// scheme with net.http. file.v exposes the File handle bound to a Client and a key. list.v parses ListObjectsV2 XML. types.v gathers the shared option and response structs (Acl, StorageClass and friends).

The signer is intentionally decoupled from net.http: it is a pure function that takes credentials and a SignRequest and returns the headers. This keeps it testable against the published SigV4 vectors and reusable outside the HTTP client.

On the security side, every user-supplied value going into a signed header or signed query param is screened for CR/LF before signing (InvalidHeader, InvalidQueryParam, InvalidCredentials). The signer rejects anything outside GET, PUT, POST, DELETE and HEAD. s3.fetch only accepts s3:// URLs, https://, http://, file:// and friends are rejected up-front to prevent accidental cross-protocol calls with S3 credentials. The default scheme is HTTPS whenever endpoint does not carry an explicit http:// prefix and insecure_http is not set. When compiled with -d s3_debug, request headers are dumped but Authorization and X-Amz-Security-Token values are masked. redact_url strips query strings before logging so presigned URLs never leak signature material into error text.

The unit suite covers 8 files and runs offline by default. v test vlib/net/s3/ reports 8 passed, 8 total. The tests cover signer vectors (AWS reference, canonical request, HMAC chain cross-checked with openssl), encoding, error parsing, credentials env merging, bucket-name validation, list-response parsing, fetch URL parsing, redaction and CRLF rejection.

Integration tests live in integration_test.v, gated on S3_INTEGRATION=1. They exercise put/get round-trip, range read, presign GET, list with prefix, exists/delete, s3.fetch and the http.fetch('s3://...') bridge, multipart (parallel, with out-of-order completion), bucket lifecycle and an invalid-credentials cleanup-error path. Example invocation:

S3_INTEGRATION=1 \
S3_HOST=https://s3.example.com \
S3_KEY_ID=... S3_KEY_SECRET=... \
S3_BUCKET=v-s3-tests \
v test vlib/net/s3/integration_test.v

The module has been verified against AWS S3 SigV4 reference vectors (byte-exact Authorization header matches in signer_test.v), against a local self-hosted S3-compatible server in debug and -prod builds (full integration suite green), and against a hosted S3-compatible production endpoint (full integration suite green).

This work builds on top of #27000 (HEAD, 1xx, 204 and 304 responses on Content-Length-bearing connections), which had to land first for stat, exists and range responses to behave correctly under V's HTTP client.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 509dabe79b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve leading/trailing slashes in object keys

build_object_path currently normalizes keys with strip_slashes(key), which changes valid S3 object names (for example, folder/ becomes folder, and /x becomes x). S3 keys are byte-exact identifiers, so this causes put/get/stat/delete to address a different object than requested whenever callers intentionally use leading or trailing / characters.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Default AWS global endpoint to us-east-1 region

When endpoint is s3.amazonaws.com and no explicit region is provided, this parser does not extract a region and falls back to 'auto', which is then used in the SigV4 credential scope. AWS S3 expects us-east-1 for the global endpoint, so requests signed with region auto will fail authentication unless users manually set region.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Allow multipart uploads with exactly 10,000 parts

The TooManyParts guard runs after incrementing part_number, so an upload that legitimately uses exactly max_parts parts is still marked as an error right after enqueueing the final allowed part. This rejects protocol-valid uploads at the S3 limit (10,000 parts), e.g. very large uploads with small configured part sizes.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Honor POST method in s3.fetch uploads

The .post branch is grouped with .put and calls c.put(...), which always signs and sends a PUT request. As a result, callers requesting method: .post silently send the wrong method on the wire, which breaks endpoints/workflows that require POST semantics.

Useful? React with 👍 / 👎.

[davlgd]

Pushed 23501bfe8 addressing the four points.

P1 — build_object_path no longer calls strip_slashes on the key. S3 keys are opaque identifiers, so folder/, /x and a//b now reach the wire byte-exact (only percent-encoded). The bucket continues to be slash-stripped since it isn't a key. Covered by a new test_build_object_path_preserves_leading_and_trailing_slashes in fetch_test.v.

P1 — guess_region now returns us-east-1 instead of auto for any *.amazonaws.com endpoint that doesn't carry an explicit region in the host. Covers s3.amazonaws.com and the legacy s3-external-1.amazonaws.com forms. Three assertions added to test_guess_region_paths in credentials_test.v. Non-AWS endpoints still resolve to auto.

P2 — In upload_bytes_multipart, the TooManyParts check moved before the enqueue (was: enqueue → increment → check). Previously an upload that used exactly max_parts parts was aborted right after queuing the legitimate final part; now max_parts is allowed inclusive, matching the loose MultipartUploader.upload path (line 85) which already had the check in the right place.

P2 — s3.fetch no longer aliases .post to PUT. The .post arm is removed from the match and falls through to the else branch returning InvalidMethod. Rationale: s3.fetch is a "URL → bytes" helper, and S3 POST endpoints (CreateMultipartUpload, CompleteMultipartUpload, browser form upload) all take a structured body that fetch doesn't generate — silently rewriting to PUT was misleading. Callers needing POST should use the multipart API directly. Covered by test_fetch_rejects_post_method.

v fmt, v vet and the 8 unit tests are green in both default and -prod builds.

[medvednikov]

Thanks!

Having a first-class client in V, with no third-party dependencies and no need to shell out to aws-cli, makes the language a credible choice for cloud workloads.

I agree.