Adopt @forkzero/s3-website-test-kit and s3proxy 4.2 fetchWeb()

[gmoon]

Two related cleanups now that the shared test kit is published and s3proxy 4.2 adds fetchWeb(). Both remove code this repo currently hand-rolls.

1. Consume @forkzero/s3-website-test-kit (drop the in-repo shared-testing/)

This repo ships its own copy of shared-testing/. The load-test configs, scenarios, and utils now live in the published package @forkzero/s3-website-test-kit (already consumed by gmoon/s3proxy), so the two repos stay in sync instead of drifting.

• npm install --save-dev @forkzero/s3-website-test-kit artillery
• Add an .npmrc pinning the scope to public npm (in case a dev's global ~/.npmrc maps @forkzero to GitHub Packages):

  @forkzero:registry=https://registry.npmjs.org
  

• Repoint the Makefile/test scripts at node_modules/@forkzero/s3-website-test-kit/configs/… and …/scenarios/core/… (note scenarios split into core/ (portable) and s3proxy/ (e.g. health.yml)).
• Delete shared-testing/.

This can be done now, independently of #2.

2. Adopt fetchWeb() in server.js (needs s3proxy 4.2)

server.js currently hand-rolls the exact Request → Response adaptation that s3proxy 4.2's proxy.fetchWeb() replaces:

// before
app.on(['GET', 'HEAD'], '/*', async (c) => {
  const url = new URL(c.req.url)
  const { stream, status, headers } = await proxy.fetch({
    url: url.pathname + url.search,
    method: c.req.method,
    headers: Object.fromEntries(c.req.raw.headers),
  })
  return new Response(Readable.toWeb(stream), { status, headers })
})

// after
app.on(['GET', 'HEAD'], '/*', (c) => proxy.fetchWeb(c.req.raw))

fetchWeb throws the typed S3ProxyError on 404/403/416, so the existing XML app.onError keeps rendering error bodies exactly as today — no behavior change. Also drops the now-unused import { Readable } from 'node:stream'.

• Bump s3proxy to ^4.2.0 in package.json.
• Replace the GET/HEAD handler as above.
• Keep the XML onError and /health, /version routes unchanged.

Order

Do #1 anytime; #2 once s3proxy@4.2.0 is on npm.

[gmoon]

Status update (for whoever picks this up)

Part 2 is now unblocked. s3proxy@4.2.1 is on npm (4.2.0 and 4.2.1 both published), so fetchWeb() is available — the "do #2 once 4.2.0 is on npm" condition in the Order section is satisfied. Both parts can proceed now. Bump to s3proxy@^4.2.1.

Mind the kit version. @forkzero/s3-website-test-kit is currently published at 0.1.0 only, and 0.1.0 predates three fixes that landed on the kit's main in forkzero/s3-website-test-kit#5 (not yet merged/released):

• special-character key was sent unencoded → false 404 (fixed across all scenarios)
• no shipped conformance-gate config (added configs/conformance.yml + a deterministic single-flow scenarios/core/conformance.yml; exits non-zero on any failed assertion)
• test data: BUCKET now required + a new bootstrap-s3-target.sh that stands up a conformance target (correct content-types, 403 via bucket policy)

So: wait for the kit's 0.2.0 release and pin @forkzero/s3-website-test-kit@^0.2.0. Installing 0.1.0 now would import the special-char bug and give you no one-line conformance gate. (0.2.0 = merge #5 → version bump → npm publish; not done yet.)

When repointing the Makefile/test scripts, wire in the new capabilities, not just the load configs:

• Conformance gate (headline feature):

  artillery run \
    --config node_modules/@forkzero/s3-website-test-kit/configs/conformance.yml \
    --target http://localhost:8080 \
    node_modules/@forkzero/s3-website-test-kit/scenarios/core/conformance.yml
  

  Note: the conformance config omits a processor on purpose — pairing the expect plugin with the kit's ESM processor crashes Artillery (object is not extensible). Use a load config (e.g. configs/docker-container.yml) if you want the JSON summary.
• scenarios/s3proxy/health.yml for the /health check (target-specific — don't run the s3proxy-only scenarios against native S3).
• test-data/bootstrap-s3-target.sh if you need to create the test bucket.

One open caveat (forkzero/s3-website-test-kit#4): core/conformance.yml's 404 case asserts contentType: application/xml. s3proxy returns exactly that, so your conformance run against s3proxy-docker will pass — but that assertion is being made target-agnostic (native S3 website hosting returns text/html), so it may move to scenarios/s3proxy/ in a later kit release. No action needed for this repo.

Everything else in the original issue (drop shared-testing/, add the .npmrc scope pin, split core/ vs s3proxy/) is still accurate — shared-testing/ still exists here and nothing references the kit yet.

[gmoon]

✅ @forkzero/s3-website-test-kit@0.2.0 is now published (latest on npm) — includes the conformance gate (configs/conformance.yml), bootstrap-s3-target.sh, and the special-char encoding fix. Pin ^0.2.0 and this part is fully unblocked.

[gmoon]

Closing — this is being handled by the s3proxy-docker agent, which owns this repo's changes.

Handoff notes for whoever picks it up (state has advanced since filing):

• Test kit is now at @forkzero/s3-website-test-kit@0.2.1 (not 0.2.0). Use ^0.2.1. Since 0.2.0 the scenarios split into core/ (portable) and s3proxy/ (target-specific), and 0.2.1 made the core conformance gate target-agnostic (core 404 asserts statusCode only; the application/xml error-body assertion moved to the opt-in scenarios/s3proxy/error-contract.yml).
• s3proxy fetchWeb() shipped in 4.2.0 and is current at 4.2.1 — part 2 is unblocked.
• New: a conformance CI gate. gmoon/s3proxy now runs the kit's expect-enabled conformance config in CI as a hard gate (core + s3proxy/error-contract.yml) — see ci: gate on the s3-website-test-kit conformance suite gmoon/s3proxy#152. This repo's server.js renders the same XML error contract, so it's a strong candidate to adopt the same gate here, not just the load config.

Original scope (consume the kit / drop shared-testing/; adopt fetchWeb() in server.js) still stands.