uat: their release gate runs on every commit, held to the recorded matrix - #82
Merged
Conversation
…trix (#61) Upstream's gate is the 63-case Robot UAT, run by hand against a billed cluster before each release. It now runs here per commit, free: the stack up on k3d with the image built from the commit, their suite unmodified from a pinned checkout, and assert-matrix.py holding the result to uat/expected-failures.txt in both directions — a new failure is a regression, an expected failure that starts passing fails too until the record moves with it. The full local run that seeded the expectations doubles as the re-measure the pass matrix lacked: 50 of 63 on v0.4.0, identical to the v0.1.0 recording, each failure still accounted for. A smoke job runs first: the landing page's own commands against the candidate — docker run, the list-microvm-images call, /_m80/health at 29/29 — because a docs prompt nobody executes is a docs prompt that rots. Both jobs proven locally before this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WVmBmk6XERjKxHa4HyezvT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #61, with the landing-page smoke folded in per the issue thread.
The uat job stands the stack up on k3d with
m80:candidatebuilt from the commit (up.shalready imports local builds — that was the point of the knob), runs KubeMicroVM's suite unmodified from a checkout pinned by SHA (bumping the pin is a deliberate commit — their suite moving is exactly what the gate exists to notice), and holds the result touat/expected-failures.txtin both directions viauat/assert-matrix.py: a failure not listed is a regression and fails; a listed failure that passes also fails, with instructions, so "50 of 63" can never rot in the flattering direction. Results upload as artifacts either way; the operator's log prints on failure.The smoke job runs the landing page's literal commands against the same candidate —
docker run, thelist-microvm-imagescall from "Give this to your agent",/_m80/healthat 29/29 with nothing pending. The CLI runs from the UAT runner image so its version is pinned by a Dockerfile rather than the runner's whim.The seed run is the re-measure. The full suite ran locally against v0.4.0 to capture the canonical expectation set: 50 of 63, identical to the v0.1.0 recording, every failure in the same documented class. docs/kubemicrovm.md's "not re-measured" caveat is replaced by the standing-gate statement. The gate was then proven against that same run:
matrix holds: every failure expected, every expectation still failing.~35 minutes end to end locally; the workflow caps at 60.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WVmBmk6XERjKxHa4HyezvT