feat(prompts): the merits stage contract, and the merits fan-out - #1027
Merged
Conversation
… proof The artifact is the first runnable merits cell: the prompt contract that tells a predictor and an evaluator what a merits event is, the fan-out that mints one, the provisioning guard that lets it keep its snapshot, and an offline cascade that drives it end to end. Prompts. `predict.md` gains `### Stage: merits`: the judgment vocabulary, `probability` read as P(disturbed) with DIG and an equally divided affirmance declared undisturbed, the mandatory vote block and why naming a Justice costs nothing, what the granted docket does and does not disclose (the merits briefs and the argument are recorded as events, never as text, and the provisioned documents are cert-stage), the cert signals and the frozen salience band being spent rather than wrong, the statpack merits section on its stated terms — the ten-Term window counted rather than read off an untruncated table, the 30-parsed floor, the grant-Term axis, the censoring direction — and the one declared `merits-v1` claim. `evaluate.md` gains the matching rules: `correct` routed on the outcome's judgment, `judgment_correct` beside it, the merits baseline in `segment_base_rate`, and `base_rate_basis` left null — which is what makes the harness stamp `base_rate_salience_version` null, correct because the merits pool is no salience-band product. No merits cell publishes a skill score. `docs/decision-model.md` pre-registers that no merits skill number may be published against a pool whose grant Terms carry unlabelled GVRs, and that the fan-out owes a label-independent guard first. That guard is not built. Detecting the contaminated Terms per cell was considered and rejected: the escapees show up as a partly-labelled Term rather than an empty one, and the table an evaluator can read is keyed on the docket-number Term, so the test would pass exactly the Terms the pack's own caveat names. So `brier_skill_score` is omitted on every merits cell, stated in the prompt and enforced by a new `validate` check rather than left to prose. `segment_base_rate` is still recorded — the pool the cell faced is a fact about the run, and recording it is not a skill claim over it. Fan-out. `store.forecastable_events` gains the merits admission, keyed on the merits stage, an unlatched judgment, and `opens_merits_proceeding` on the row — the same predicate that mints the event and that the statpack merits section measures its rate over, re-checked so a docket re-resolved to `gvr` sheds its cell. The guard test it replaces is now five, one per arm. The residue the predicate cannot close — predict scope excludes IFP grants, the merits population does not — is stated at the seam with its size rather than bounded away. Provisioning. The forward-cell leakage guard is keyed on the event, because one docket carries several events' outcomes at once: a granted docket's grant order is a disclosed cert outcome and is also what opens the merits proceeding, so without the key every merits cell would be refused by the entry that minted it and would run snapshot-less and context-less. On the merits event the test is a parsed judgment plus the cert scan's own recall minus its one cert-before- judgment grant branch — the conservative parser alone would let "Opinion Issued" through, and the two failure costs point in opposite directions. Proof. The offline cascade smoke drives a merits cell provision -> stub predict -> judgment outcome -> evaluate -> validate over the fixture's granted docket, and asserts it lands in the leaderboard's unranked `merits` stages block with nothing entering the cert board. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The merits stage becomes runnable: prompt sections in both prompts, the fan-out widened to the merits event, and an offline merits cell proving the path.
predict.md→### Stage: merits—judgment's six values,probability= P(disturbed) (withgrantedas the same binary andpredicted_disposition = other), the mandatory intersection-only-scored vote block, what the record does and does not show, why the cert signals and salience band are spent rather than wrong, the statpack merits section with its cautions, and the singlemerits-v1claim.evaluate.md— a merits block mirroring the interim one, ajudgment_correctfield bullet, and a stage-awarecorrect._merits_forecastable(merits stage + order kind + no latched judgment +opens_merits_proceeding+ row scope). The deliberate hold-back test is replaced by five, one per arm.test_stub_cascade_merits_smokedrives provision → stub predict → judgment outcome → evaluate → validate, asserting the cell lands in the leaderboard's merits stage block with the cert board still empty.Two findings that changed the design
1. Provisioning would have refused every merits cell.
provision-snapshot --refuse-terminalmatched"Petition GRANTED."throughmatch_disposition_signal— so the very entry that mints a merits event would have refused its own snapshot. Every production merits cell would have run snapshot-less and context-less: no frozen conditioning, therefore no claim block. The guard is now keyed on the event (--event), because one docket carries several events' outcomes at once: on the merits event the outcome is the judgment and the cert grant is legitimate record; on every other event the existing rule is unchanged. This is the workflow line — passingmatrix.event_idthrough to the provisioner.zizmorandactionlintboth clean; workflow-reviewer found nothing.2. Merits skill scores are suppressed, not computed — a deliberate deviation.
docs/decision-model.mdpre-registers that no merits skill number may be published against a pool with unlabelled GVRs, and that the fan-out owes a label-independent guard first. That guard is not built, and a per-cell heuristic cannot substitute: the escapees surface as partly-labelled Terms (OT2023/OT2024 rendergvrat 0.4%/0.8%, not zero), and the agent-readable table is keyed on the docket-number Term. Sobrier_skill_scoreis omitted on every merits cell, enforced by a newvalidatecheck.segment_base_rateis still recorded, so the number is there to audit when the guard lands. This honours the pre-registration ordering rather than publishing a number the record says we may not.Review
outcome-decomposition.md,README.md,budget.md(the merits stream is now priced, and its lack of a quota justified) anddata-pipeline.md.Stated at the seam rather than bounded away: predict scope excludes IFP grants while the statpack merits population does not — ~13% of the pack's plain grants (86/679, verified).
Digest: YES (both prompts). Expected pre-freeze; the paste happens once, post-promotion.
Gate: lint / types / test (1912 passed) / schemas / data (11,212 artifacts, 5,509 references) — green, plus
zizmor@1.26.1andactionlint 1.7.12clean.Closes #1009.