v0.73.2 — the release gate stops lying in both directions

soup ship answers one question: did this model get better, or did I break it? Leg 2 — the regression half — was wrong in three separate ways at once. Two of its suites ranked by the wrong thing, one whole failure direction had no detector, and a caller error was indistinguishable from a real regression.

Every item was reproduced against shipped v0.73.1 before a line changed. All four defects live in a scorer, so the faithful instrument is a stub emitting the shapes a real model produced — no GPU needed, and none is claimed.

What's New

Two suites were ranking by the wrong thing

mini_tool_call ranked brace hygiene (#346). Llama-3.1-8B named the right tool 40/40 and scored 0.225. It emitted three opening braces and two closing ones, so the whole-string parse failed, the bounded scan returned the inner object, and the scorer rejected it for having no "function" key. Reproduced here at the extreme: a stub naming the right tool on every item, one brace short, scored 0.000.

The envelope is now restored — but only for an object carrying both name and arguments. Requiring arguments is the safety argument: the prompt shows the model a menu of {"name","description"} objects, so a name-only test would credit echoing the menu back.

mini_mmlu scored an 8B below a 0.5B (#357). 0.423, while the same model scored 1.000 on two other MCQ suites. Two causes: the extractor did not know \boxed{C}, and the prompt never asked for a letter — so of 15 failures, 8 boxed the right letter and 6 boxed a value. A stub answering every item correctly in boxed-letter style scored 0.000.

Both halves are required. The extractor alone is worth +8 items; the prompt alone is worth 0; together 0.423 → 0.731 and the inversion disappears. The new tier fires only on a single A–J letter — reading \boxed{4} as "option 4" would be a wrong credit, not a repair. Among the boxed/cue/paren forms, position decides, not form: a model that boxes a scratch answer and then self-corrects chose the correction.

Leg 2 had no over-refusal detector (#317)

It flagged a drop in refusal rate and had no reverse, so a tune that refuses everything registered as a monotone safety improvement with no ceiling on how useless the model became.

Reproduced as indistinguishability, which is the actual claim: two models with byte-identical scores on all seven shipped suites and the same SHIP verdict — one of which refuses every benign request.

mini_over_refusal is the mirror: 40 hand-authored benign-but-scary-sounding requests (defensive security, first aid, sysadmin verbs like "kill", fiction with dark themes) scored as fraction NOT refused. It joins the default suite (7 → 8), so the existing regression rule catches over-refusal with zero change to the verdict engine. Paired with mini_safety, neither axis can be gamed alone — verified live on a real SmolLM2-135M pair: 1.0000/0.9750 on the benign axis while the harmful axis reads 0.0000 on the same models.

New: soup ship --noise-floor N

Greedy decoding is not deterministic on GPU. Measured on an H100, the same model with no adapter over five runs spread 0.015–0.020 against a threshold of 0.05, with four of six paired deltas sitting inside the spread. The gate was calling differences it could not resolve.

--noise-floor N re-runs the base model N times (2–10), takes each axis's max − min, prints it beside the verdict, and gates every axis at max(threshold, floor). Leg 1 must clear the task axis's floor too.

The max is load-bearing in both directions: a floor above your threshold widens the gate to what is actually measurable, and a floor below it must never tighten the gate behind your back. If a floor exceeds your threshold, the run says so by name.

Scope, stated plainly. The leg-1 floor is measured in --task-mode metric only — in the judge modes a repeat would fold the judge's own sampling noise into a number presented as decode noise. On the dev box the measured floor is 0.0000: CPU greedy decode is deterministic there, and the H100's 0.015/0.020 is not re-claimed as a laptop measurement. A 0.0 floor correctly suppresses nothing.

Caveat carried from the measurement that motivated it: n=3, one model, one dataset. It sizes the effect; it does not calibrate a threshold.

A caller error looked like a finding (#355)

score_bundled_suite returned 0.0 for a non-callable generator on the three behavioural suites, while raising on the MCQ ones. In leg 2 a 0.0 reads as "the model failed every item" → DON'T SHIP — so a caller error was indistinguishable from a regression, and it failed in the direction that looks like a real result. Both branches now raise. A callable that misbehaves is still a failed item, which is the correct existing contract.

Also in this release

• soup data split --stratify-semantic (#388) — partition splits proportionally across semantic groups so a whole topic cannot land in one split. Thanks @Deadpool2000.
• soup mcp serve --allow-execute (#391) — a stronger opt-in than --allow-mutating. Everything stays plan-only in this slice, and the help text says so in the present tense. Thanks @CODING-DARSH.

Two pre-existing bugs found on the way

• The verdict panel never printed its own leg-1 marker. It built the header as ... [{won_str}], and a bare [no win] is valid Rich markup for an unknown tag — so Rich ate it, on every release through v0.73.1. The plain-text rubric, which has no markup parser, printed it correctly the whole time. That asymmetry is why nobody noticed.
• Untrusted names could drive the terminal. rich.markup.escape neutralises [...] and nothing else, so an axis name from an --evidence file carrying raw escape sequences reached the terminal intact.

Install / Upgrade

pip install --upgrade soup-cli

Security

An evidence-supplied noise floor widens the gate, so "floors": {"mini_mmlu": 1.0} in an evidence file would mask any possible drop on that axis. This crosses no new trust boundary — anyone who can edit that file can already forge the raw scores and force a SHIP outright — but it is a far quieter edit to miss in review, and soup ci init wires ship --evidence as a PR merge gate.

Floors are therefore bounded to [0, 1], capped at 50 axes / 256-character names, refused rather than dropped when malformed (a dropped floor replays as a different verdict), and any floor exceeding the threshold is announced by both readers — the CLI on stderr, the MCP tool in its returned payload, since its stdout is the JSON-RPC channel. Neither reader is the quiet one.

Benchmark and axis names are also stripped of control bytes before rendering.

Known Limitations

1. The leg-1 noise floor is measured in --task-mode metric only. In the judge modes the run warns and leaves leg 1 at a 0.0 floor, so a judge-scored win smaller than the judge's own sampling noise is still called a win.
2. n=3, one model, one dataset. The floor sizes the effect; nothing here establishes what N is enough.
3. No GPU floor was measured on the dev box. The run reported is CPU, where greedy decode is deterministic and the floor reads 0.0000.
4. --baseline snapshots taken before v0.73.2 are on a different scale for mini_mmlu, mini_common_sense and mini_tool_call — measured on an unchanged model the jumps are 0.423 → 0.731 and 0.225 → 1.000, far larger than the 0.05 gate. A --baseline supplies the base score from a file and skips the live base run, so it would be diffed against a freshly-scored tuned model. The run now warns by name, but cannot re-measure for you, and there is no version stamp inside a baseline file to detect this automatically. mini_instruction and mini_arithmetic are unaffected (0 of 24 and 0 of 36 items carry a letter answer).
5. An --evidence file remains trusted input. The floor is bounded and announced, but the scores are not signed — --config binds evidence to a config hash, not to the scores.
6. mini_over_refusal is 40 hand-written prompts scored by a keyword refusal heuristic, so it inherits every limit of that heuristic. It detects a collapse in benign helpfulness, not a subtle shift in tone.
7. Two suites still sit at or next to a ceiling after the repair. A suite pinned at 1.000 detects a regression exactly as poorly as one pinned at 0.000, and nothing here re-sizes them.

Measurement record

Published as written, including a withdrawn order-dependence scare, a control that turned out to vary nothing, and a review finding checked against three implementations and partly rejected: benchmarks/gate-v0.73.2-leg2-scoring.md

Tests: 17727 → 17907 (360 files). Full suite green.