docs(roadmap): capture six next-generation QA roadmaps#33
Merged
Conversation
…ion-triage] Implements lore-proofkeeper/roadmaps/regression-triage.md. Adjudicate a failing verifying test against the requirement text: stale test (repair via human-reviewed PR) vs product regression (evidence report). The requirements corpus as ground truth is the capability prompt-driven QA agents cannot copy. Signed-off-by: Tom Ballard <tom@armytage.co>
…ia-scenarios] Implements lore-proofkeeper/roadmaps/criteria-scenarios.md. Decompose a requirement's acceptance criteria into one drive per criterion, negative paths included, with criteria-level coverage — verified means every stated criterion, not one test exists. Signed-off-by: Tom Ballard <tom@armytage.co>
…eeper-action] Implements lore-proofkeeper/roadmaps/proofkeeper-action.md. One-YAML-block adoption: a published composite action wrapping scoped QA on pull requests, with security documentation that makes the trust boundary enterprise-approvable. Signed-off-by: Tom Ballard <tom@armytage.co>
Implements lore-proofkeeper/roadmaps/qa-benchmark.md. Open sample apps with seeded corpora; verified-capability rate, token cost, and wall-clock published with deterministic scoring — the re-runnable argument behind any competitive claim. Signed-off-by: Tom Ballard <tom@armytage.co>
Implements lore-proofkeeper/roadmaps/visible-cost.md. Budget enforcement over the measured per-drive token usage, spend in the evidence surfaces, and Wayfinder composition over OPENAI_BASE_URL — compose, don't embed. Signed-off-by: Tom Ballard <tom@armytage.co>
…matrix] Implements lore-proofkeeper/roadmaps/fidelity-matrix.md. Distribute fidelity attempts across browsers/targets with pass-rate policy and cold-condition runs, so stable carries statistical weight. Signed-off-by: Tom Ballard <tom@armytage.co>
…map:qa-benchmark] Implements lore-proofkeeper/roadmaps/qa-benchmark.md. The harness, sample apps, corpora, and results page live in itsthelore/rac-benchmarks as the autonomousqa/ subdir member (one repo per concern, subdir per member — benchmarks are ADR-092's named example); Proofkeeper's repo keeps only the pinned release-gate hook. The benchmark is framed agent-agnostic: a property of the corpora, apps, and deterministic scoring, with Proofkeeper as the reference agent. Signed-off-by: Tom Ballard <tom@armytage.co>
tcballard
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Captures the post-review strategy as six discrete roadmap artifacts in the corpus — durable intent per ADR-093 (roadmap intent lives in the corpus; execution tracked in issues), identified by stable codename per ADR-094. One commit per artifact.
The strategic frame: Proofkeeper's structural advantage in the automated-QA space is that it verifies against a requirements corpus (typed
verified_byedges), not a prompt. Each roadmap cashes that in.regression-triagecriteria-scenariosproofkeeper-actionqa-benchmarkrac-benchmarksas theautonomousqa/subdir member per ADR-092 (one repo per concern) — agent-agnostic by construction, Proofkeeper as the reference agent; this repo keeps only the pinned release-gate hook.visible-cost--max-tokensbudget over the now-measured usage, spend in the evidence surfaces, Wayfinder composition overOPENAI_BASE_URL(compose, don't embed).fidelity-matrixHousekeeping items (release cut, #28, branch protection, rac pin cadence) are process actions and deliberately not captured as roadmap artifacts.
Gates
rac validate lore-proofkeeper/— 41/41 valid.rac relationships lore-proofkeeper/ --validate— 62 checked, 0 issues (allRelated Roadmapscross-references resolve).rac review lore-proofkeeper/— nothing needs attention, health 100/100.