v2.0.4 — Verifier-collapse detector
Patch release. Adds a stdlib-only verifier-collapse detector to the
consistency dimension and wires the resulting flag into both the
/judge --explain output (Markdown + explain.v1 JSON) and the
Stop-hook ship-gate. Closes a latent bug in
_analyze_consistency: the prior low-variance branch would
reward a flatlined verifier with a +1 "highly consistent"
bonus — the new detector composes with that path so collapsed
verifiers net to a dock instead.
Added
-
skills/judge/scripts/score.py::_detect_verifier_collapse—
offline statistics over the rolling window of recent scorecards
for the same skill. Flagsverifier_collapse=truewhen, over at
leastmin_samplesof the lastwindowcards (defaults 5/10):- fraction of composites
>= top_threshold(default 8.5)
crossestop_bucket_fraction(default 0.95), and std_devof composites< max_std_dev(default 0.3 —
tighter than the existing 0.8 "highly consistent" cutoff).
On a hit,
_analyze_consistencydocks the dimension by
consistency_dock(default 3) and appends a one-line reason.
The dock is wide enough to net out the existing low-variance
+1bonus that the same data would otherwise trigger. - fraction of composites
-
dimensions.consistency.verifier_collapse/
verifier_collapse_reason/verifier_collapse_statson the
scorecard, mirrored at the scorecard top level as
verifier_collapse: boolfor one-jq-query CI consumption. -
judge-config.json.verifier_collapseblock (enabled by default;
setenabled: falseto disable). Knobs:window,min_samples,
top_threshold,top_bucket_fraction,max_std_dev,
consistency_dock,gate_mode. -
explain.v1JSON surfacesverifier_collapse(top-level) plus
per-dimensionverifier_collapse/verifier_collapse_reason/
verifier_collapse_statson the consistency entry. The Markdown
renderer adds a "⚠️ Verifier collapse detected" callout above
the dimension table, anchored on Verdict's own consistency
dimension plus the Soft-SVeRL project anchor (no sibling
benchmark analogies, per the G13 cross-pollination rule). -
hooks/judge-on-stop.shhonours
judge-config.json.verifier_collapse.gate_mode:warn(default) — stderrVerdict WARNING: verifier collapse detected for $SKILL_NAME — $REASON, exit code unchanged.fail— stderrVerdict BLOCKED: ...+exit 2(same shape
as the existing threshold-breach gate).off— silent.
-
schemas/scorecard.v1.schema.jsonnow declares the new optional
fields (top-levelverifier_collapseplus the three new
per-dimension keys). Backward-compatible additive extension; no
required-field change.
Tests
tests/test_verifier_collapse.py(new): clean varied history
produces no flag; collapsed history (10× 9.5) flags + docks the
consistency dim; below-min_samplesproduces no flag;
enabled: falseproduces no flag; explain.v1 JSON carries
top-level + per-dim fields; Markdown renderer emits the callout
with the Soft-SVeRL anchor; hook gate-modewarnexits 0,
failexits 2,offis silent.
Notes
-
The detector is offline-only (pure stdlib statistics over
scores/) and is the default-on companion to the off-by-default
LLM second-opinion analyzer. LLM judging is not made the default
by this change — the offline heuristic is the moat. -
The signal is derived from Verdict's own consistency dimension
plus the Soft-SVeRL project anchor. Sibling-benchmark analogies
were considered and dropped per the G13 anti-cross-pollination
rule; no external evaluation suites are named in code, docs, or
CHANGELOG for this change.