ci: gate on the s3-website-test-kit conformance suite#152
Merged
Conversation
CI's Performance Tests job only ran the kit's load config, which enables no assertions — it measures throughput and that requests don't error, but never checks the HTTP contract. A regression that still returns 200 (e.g. a dropped content-type header) slips through green. Add `make conformance-docker`: stands up the same test image on port 8081 and runs the kit's `expect`-enabled conformance config against two scenarios — the portable `scenarios/core/conformance.yml` (status / content-type / content-length) and the s3proxy-specific `scenarios/s3proxy/error-contract.yml` (404/403 -> application/xml, which fastify-docker.ts renders). Any mismatch exits non-zero. Wire it into the Performance Tests job as a fail-fast step before the load run. No load-timing gate: absolute p95/latency thresholds are flaky on shared CI runners (cf. the streaming-memory RSS-threshold flake); timing stays a tracked artifact, correctness is what we gate on. Verified locally against the built container: core 16/16 and error-contract 4/4 expectations pass, `make conformance-docker` exits 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 14, 2026
gmoon
added a commit
that referenced
this pull request
Jul 14, 2026
The lattice was ~878h behind code (health WARN). Reconciled via the lattice CLI (add/resolve/verify/edit/edge, then drift --acknowledge; lint + drift now clean). New nodes: - REQ-CORE-007 / IMP-CORE-005 — fetchWeb() Web-standard adapter (PR #144/#145); IMP supersedes IMP-EXAMPLE-001's hand-rolled Hono conversion. - REQ-BUILD-003 / IMP-BUILD-002 — @forkzero/s3-website-test-kit extraction + adoption and the conformance CI gate (PRs #150-#152). - REQ-EXAMPLE-003 / IMP-EXAMPLE-002 — container-server consolidation into forkzero/s3proxy-docker + local tsx CI targets (PR #156; #153/#155 context folded in); supersedes REQ-EXAMPLE-002. - REQ-TEST-002 / IMP-TEST-002 — de-flake of the streaming-memory RSS bound under coverage instrumentation (PR #154). Modified: REQ-EXAMPLE-002 -> wontfix + status superseded (its premise, examples/fastify-docker.ts, was deleted); IMP-EXAMPLE-001 -> status superseded (historical body preserved). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI's Performance Tests job runs only the kit's load config (
configs/load-test.yml), which enables no assertions — it measures throughput and that requests don't error out, but never checks the HTTP contract. A correctness regression that still returns200sails through green.Concretely: earlier this session I dropped the
ContentTypemapping ins3-gateway.ts(a realistic "refactor dropped a line" bug). The load run stayed green (vusers.failed: 0, every object still 200); only the conformance gate caught it (4contentTypeassertions failed, exit 21). That gate wasn't in CI — this PR adds it.Change
make conformance-docker— stands up the sames3proxy:testimage on port 8081 and runs the kit'sexpect-enabledconfigs/conformance.ymlagainst two scenarios:scenarios/core/conformance.yml— portable: status codes, content-types, content-lengths.scenarios/s3proxy/error-contract.yml— s3proxy-specific:404/403→application/xml, whichfastify-docker.ts(the image's server) renders.Any mismatch exits non-zero. Modeled on the existing
test-validation-dockercontainer lifecycle.CI: a
Run conformance gatestep in Performance Tests, placed before the load run so a contract regression fails fast.Docs updated (
docs/maintenance.md).Deliberately not gating on load timing
Absolute p95/latency thresholds are flaky on shared CI runners — noisy neighbours, variable network to real S3, a shifting runner baseline (cf. the
streaming-memoryRSS-threshold flake that already bit this repo). Timing stays a tracked artifact; correctness is what we gate on.Verification
Ran
make conformance-dockerlocally against the built container:makeexit 0🤖 Generated with Claude Code