feat(triage): adjudicate a failing verifying test against its requirement [roadmap:regression-triage]#34
Merged
Merged
Conversation
Adjudicate a failing verifying test against its requirement: confirm the failure by re-running the spec, re-drive the capability with the requirement-derived goal, and issue a verdict — not_reproducible, stale_test (with the compiled repair candidate and its fidelity as evidence), product_regression (with the honest reason), or inconclusive (infrastructure noise never masquerades as either actionable verdict). Regression and inconclusive reasons feed the failure-learning store. Composes existing seams only: capability selection, the drive seam, the compiler, and the fidelity gate. Signed-off-by: Tom Ballard <tom@armytage.co>
proofkeeper triage (--graph-file|--corpus) --capability <id> --url <u> with --spec disambiguation, --json for the stable machine contract, and the exit code as the verdict: 0 stale test / not reproducible, 1 product regression, 2 usage, 3 inconclusive. Signed-off-by: Tom Ballard <tom@armytage.co>
…regression-triage] Unit (default CI): verifier resolution (single/ambiguous/wrong/none), all four verdicts with scripted runner/drive doubles, learning recorded only for regression/inconclusive, parser guards, exit-code mapping, and the human/JSON renderings. Real browser (PROOFKEEPER_E2E, wired into the e2e job): a renamed button adjudicates as STALE TEST with a stable repair candidate; a broken flow adjudicates as PRODUCT REGRESSION. Signed-off-by: Tom Ballard <tom@armytage.co>
…admap:regression-triage] Implements lore-proofkeeper/requirements/req-regression-triage.md. Adds the requirement and design artifacts (slice 1: verdict core and command; the repair PR and regression report are a later slice), the README 'Triage a failing test' section, and the changelog entry. Signed-off-by: Tom Ballard <tom@armytage.co>
tcballard
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slice 1 of the
regression-triageroadmap — the flagship differentiator. Stacked on #33 (the roadmap artifact must exist for the corpus gates); merge #33 first, then retarget/merge this.When a committed
## Verified Bytest goes red, the question that matters is which thing broke. Because the requirement is ground truth, Proofkeeper can adjudicate — agents without a requirements corpus cannot.What
proofkeeper triage (--graph-file | --corpus) --capability <id> --url <url>verified_byedges;--specdisambiguates multiples). Passing →NOT REPRODUCIBLE, done.0STALE TEST — the re-drive finished, asserted, and its compiled spec passed the fidelity gate: the capability still verifies; the repair candidate's path + fidelity are the evidence. (The automated repair PR is slice 2, through the human-reviewed write-back path.)1PRODUCT REGRESSION — the re-drive gave up / exhausted its budget / asserted nothing: the requirement's outcomes could not be driven; the model's honest reason is the evidence, and it feeds failure-learning.3INCONCLUSIVE — infrastructure noise (spec can't run, model outage, repair candidate errored or didn't stabilize). Conservatism is the invariant: neither actionable verdict is ever guessed.2stays usage.--jsonemits the stable machine verdict.New module
src/triage/triage.ts(runTriage,resolveVerifier, verdict union — exported from the library); CLI command + USAGE + exit-code docs. Reuse only: capability selection, the drive seam,CodegenCompiler,assessFidelity,FileLearningStore— no new drive/gate/runner behavior.Proven end to end
tests/triage.integration.test.ts(real browser, wired into the CI e2e job): a spec recorded against yesterday's UI fails against today's —Unit suite: 16 new tests — every verdict path with scripted doubles, verifier resolution, learning boundaries, parser, exit-code map, human/JSON rendering.
Corpus (dogfood)
req-regression-triage+design-regression-triage(Related Roadmaps →regression-triage), README "Triage a failing test" section, changelog entry. Gates:rac validate43/43, relationships 0 issues, review clean, self-coverage 18/18.Verification
PROOFKEEPER_E2E=1triage integration suite: both verdicts green in real Chromium.