feat(triage): adjudicate a failing verifying test against its requirement [roadmap:regression-triage]

.github/workflows/ci.yml

@@ -54,6 +54,7 @@ jobs:
           tests/terminal.integration.test.ts
           tests/http.integration.test.ts
           tests/qa.integration.test.ts
+          tests/triage.integration.test.ts
           tests/plan.integration.test.ts
           tests/observe.integration.test.ts
           tests/extension.integration.test.ts

CHANGELOG.md

@@ -14,6 +14,15 @@ match exactly, or the workflow fails the publish.
 
 ## Unreleased
 
+- **Triage a failing test: stale test or broken product.** New `proofkeeper
+  triage` command — when a `## Verified By` test goes red, it confirms the
+  failure, re-drives the capability against the requirement, and adjudicates:
+  **stale test** (exit 0 — the capability still verifies; a stable repair
+  candidate is compiled), **product regression** (exit 1 — the requirement's
+  outcomes could not be driven, with the reason as evidence), or
+  **inconclusive** (exit 3 — infrastructure noise, never a guessed verdict).
+  `--json` emits the machine contract; regressions feed failure-learning.
+
 - **The drive has a trust boundary.** Everything the model observes is content
   from the product under test — untrusted input. The loop now enforces a policy
   on the model's tool calls: the **shell is off by default** (`run_command`

README.md

@@ -107,6 +107,22 @@ OPENAI_API_KEY=… GITHUB_TOKEN=… proofkeeper qa \
 
 `proofkeeper qa` (alias `verify`) runs the whole loop for one capability: pick → drive → compile → fidelity → run → optionally propose the write-back. With `--config` it scopes to a pull request, driving every capability the changed files touch — concurrently, context-isolated — and posting one comment that updates in place. `--plan` has the model write a test plan first.
 
+## Triage a failing test
+
+When a committed `## Verified By` test goes red, the question that matters is *which thing broke*. Because the requirement is ground truth, Proofkeeper can adjudicate:
+
+```bash
+proofkeeper triage --corpus path/to/rac/ --capability REQ-CHECKOUT --url http://localhost:3000/
+```
+
+It re-runs the failing spec to confirm, re-drives the capability against the requirement, and issues a verdict — the exit code is the contract:
+
+- **`0` — stale test.** The capability still verifies; the UI moved under the spec. A stable repair candidate is compiled and its path printed (an automated repair PR is planned).
+- **`1` — product regression.** The requirement's outcomes could no longer be driven; the reason (the model's honest give-up) is the evidence.
+- **`3` — inconclusive.** Infrastructure got in the way (model outage, runner errors, an unstable repair) — never a guessed verdict.
+
+`--json` emits the machine-readable verdict for CI. Agents without a requirements corpus can update a broken test or escalate it — they can't tell you whether the product still does what it's supposed to.
+
 ## Bring your own model
 
 No model is bundled. Two adapters cover most providers; the `ModelClient` interface covers the rest.

lore-proofkeeper/designs/design-regression-triage.md

@@ -0,0 +1,104 @@
+---
+schema_version: 1
+id: PK-KWGPQ30SFT0X
+type: design
+---
+# Triage Verdict Core — Confirm, Re-Drive, Adjudicate
+
+## Context
+
+Slice 1 of the regression-triage roadmap: the adjudication core behind
+`proofkeeper triage`. The whole pipeline already exists — graph loading,
+capability selection with `verifiedBy` paths, the drive seam, the compiler,
+the fidelity gate, failure learning — so this design is one orchestrator and
+one command over reused parts, plus the verdict model that makes the outcome
+trustworthy.
+
+## User Need
+
+A maintainer looking at a red verifying test needs to know, without hand
+investigation, whether to fix the test or fix the product — with evidence
+either way, and an honest "cannot tell" when infrastructure got in the way.
+
+## Design
+
+- **Verdict model** (`src/triage/triage.ts`): a discriminated union —
+  `not_reproducible`, `stale_test` (carries the compiled repair candidate and
+  its fidelity verdict), `product_regression` (carries the reason), and
+  `inconclusive` (carries the reason). `runTriage` orchestrates:
+  1. resolve the verifying spec (`resolveVerifier`: one entry ⇒ implicit,
+     several ⇒ `--spec` required, wrong spec ⇒ named error);
+  2. confirm — re-run the old spec; passing ⇒ `not_reproducible`;
+  3. re-drive with the requirement-derived goal, threading prior failures and
+     the trust boundary exactly as `runQa` does;
+  4. adjudicate — finished + asserted + stable candidate ⇒ `stale_test`;
+     gave up / budget / assertion-free ⇒ `product_regression`; everything
+     else (runner error on the old spec, drive threw, candidate errored or
+     failed to stabilize) ⇒ `inconclusive`.
+- **Conservatism is the invariant.** The two actionable verdicts are only
+  issued on clear evidence; instability and infrastructure noise never
+  masquerade as either. A fresh spec that held live but flakes under the gate
+  is `inconclusive`, not a regression.
+- **CLI** (`src/cli.ts`): `parseTriageArgs` (pure, exported, `requireValue`
+  guards like every sibling), human rendering (verdict line + evidence +
+  next-step hint), `--json` stable contract, and `triageExitCode` — 0 stale /
+  not-reproducible, 1 regression, 2 usage (via `TriageInputError` →
+  `UsageError`), 3 inconclusive. Documented in USAGE.
+- **Learning**: regression and inconclusive reasons are recorded with a
+  `[triage:<kind>]` prefix so future drives and reports can distinguish
+  triage-sourced knowledge.
+
+## Constraints
+
+- Reuse only: no new drive, compiler, gate, or runner behavior; the
+  orchestrator composes existing seams and stays unit-testable with the same
+  doubles as `runQa`.
+- `runTriage` never throws for drive/runner failures — those are verdicts;
+  it throws only `TriageInputError` (caller-fixable) and programming errors.
+- Exit code 3 extends the CLI contract additively; 0/1/2 keep their meanings.
+
+## Rationale
+
+Confirming the failure before re-driving keeps the expensive, model-driven
+step off the path for specs that pass locally (mis-configured CI, stale
+checkout). Adjudicating on "finished + asserted + stable" reuses the exact
+bar `runQa` sets for "verified", so a stale-test verdict means precisely
+"this capability would verify afresh today".
+
+## Alternatives
+
+- **Diff the old and new specs to classify the failure.** Rejected for the
+  verdict: spec text similarity is not evidence about the product; the live
+  re-drive is. (A diff is useful *presentation* for the slice-2 repair PR.)
+- **Auto-replace the stale spec immediately.** Rejected: write-backs are
+  human-reviewed PRs (the trust boundary); slice 2 adds that path.
+- **Treat an unstable repair as regression.** Rejected: the drive's
+  assertions held live at record time, so instability is flake evidence, not
+  product evidence.
+
+## Accessibility
+
+Not applicable — CLI text; the verdict line leads and the evidence follows.
+
+## Style Guidance
+
+Verdict words are loud and unambiguous (STALE TEST / PRODUCT REGRESSION /
+INCONCLUSIVE); reasons are complete sentences carrying the model's own words
+where they are the evidence.
+
+## Open Questions
+
+- Whether triage should accept a CI report (file of failing specs) and batch
+  over capabilities. Deferred to a later slice alongside the repair PR.
+
+## Related Requirements
+
+- req-regression-triage
+
+## Related Roadmaps
+
+- regression-triage
+
+## Status
+
+Accepted

lore-proofkeeper/requirements/req-regression-triage.md

@@ -0,0 +1,62 @@
+---
+schema_version: 1
+id: PK-KWGPQ2742QKS
+type: requirement
+---
+# Triage Verdict — Stale Test or Broken Product
+
+## Problem
+
+A red verifying test only says something broke — not *which thing*. Without
+adjudication, a maintainer investigates every failure by hand, and the two
+failure classes demand opposite responses: a stale test needs a repaired spec,
+a product regression needs a bug fix. QA agents without a requirements corpus
+cannot make this distinction; Proofkeeper can, because the requirement text is
+ground truth a fresh drive can be judged against. This requirement covers
+slice 1 of the regression-triage roadmap: the verdict core and the CLI
+command. The automated repair PR and the regression issue report follow in a
+later slice.
+
+## Requirements
+
+- [REQ-001] `proofkeeper triage` takes a coverage source, a capability id, and a target URL; the capability's verifying spec is resolved from its `verified_by` edges, with `--spec` required (and validated) only when there are several.
+- [REQ-002] The failure is confirmed first: the verifying spec is re-run against the target, and a passing spec yields a `not_reproducible` verdict with no re-drive.
+- [REQ-003] A failing spec triggers a re-drive of the capability with the requirement-derived goal; a drive that finishes, asserts, and compiles to a spec that passes the fidelity gate yields `stale_test` with the repair candidate's path and fidelity as evidence.
+- [REQ-004] A re-drive that gives up, exhausts its step budget, or asserts nothing yields `product_regression` with the honest reason as evidence.
+- [REQ-005] Anything muddied by infrastructure — the verifying spec cannot run, the re-drive itself fails, the repair candidate never runs cleanly or does not stabilize — yields `inconclusive`, never a guessed verdict.
+- [REQ-006] The exit code is the machine contract: 0 for stale test / not reproducible, 1 for product regression, 2 for usage errors, 3 for inconclusive; `--json` emits the stable structured verdict.
+- [REQ-007] `product_regression` and `inconclusive` reasons are recorded to the failure-learning store; verifying outcomes record nothing.
+
+## Success Metrics
+
+- A renamed control (requirement unchanged) adjudicates as `stale_test` with a
+  stable repair candidate, end to end in a real browser.
+- A broken flow (outcome never reachable) adjudicates as `product_regression`
+  naming the model's give-up reason.
+- Exit codes 0/1/3 are distinguishable in CI without parsing output.
+
+## Risks
+
+- A model could mis-drive a healthy product into a false regression verdict.
+  Mitigation: prior failure learning steers the re-drive, the verdict carries
+  the full reason for human review, and the fidelity gate guards the opposite
+  direction (no false stale-test without a stable repair).
+- Confirming the failure re-runs a known-failing spec, which can be slow
+  (action timeouts). Mitigation: bounded by the runner's existing wall-clock
+  caps.
+
+## Assumptions
+
+- The requirement-derived goal (`defaultGoal`) carries enough observable
+  intent to adjudicate against; ambiguity degrades to `inconclusive`.
+- The trust boundary and resilience behavior of the ordinary drive apply
+  unchanged to re-drives.
+
+## Related Roadmaps
+
+- regression-triage
+
+## Verified By
+
+- `tests/triage.test.ts`
+- `tests/triage.integration.test.ts`