⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.
Context
calibrateActThreshold (src/review/risk-control.ts:138) has two refusal statuses that #9048 deliberately
split apart because they need opposite remediations:
insufficient_labels — "go collect more labels";
no_certifiable_threshold — "you have plenty of labels, the error rate is too high; investigate precision
or accept a weaker alpha" (field doc at src/review/risk-control.ts:108-121, distinct audit event at
src/review/risk-control-wire.ts:159-172).
#9066 then added the Bonferroni multiplicity correction: every candidate's Clopper–Pearson bound is now tested
at testDelta = delta / candidates.length (src/review/risk-control.ts:148). The label-supply gate was not
updated with it. Line 139 still computes:
const needed = minimumCalibrationLabels(alpha, delta);
and line 161 gates each candidate on if (n >= needed) using that same full-δ value.
The zero-error floor is ceil(ln(δ)/ln(1−α)), which grows as δ shrinks. At α=0.05, δ=0.05 the full-δ floor is
59 labels; at δ/K with K=59 distinct observed confidences it is 138. On the merge arm (α=0.005) the full-δ
floor is 598 and at K=300 it is 1736. So for every arm whose usable label count lands in the gap
[minimumCalibrationLabels(α, δ), minimumCalibrationLabels(α, δ/K)), a calibration set with zero errors
clears line 140, computes bounds at every candidate, certifies nothing, and falls through to line 189 —
returning no_certifiable_threshold.
recalibrateArm then writes risk_control_no_certifiable_threshold with the detail string
"59 labels available but no threshold achieves α=0.05" (src/review/risk-control-wire.ts:170), telling an
operator to investigate close/merge precision on an arm that made zero mistakes and whose only real
shortfall is label count. That is precisely the misdirection #9048 was filed to end, reintroduced by #9066.
A second, smaller instance of the same conflation sits in the same file:
src/review/risk-control-wire.ts:196-199 catches any read/write failure and sets
summary[arm] = "insufficient_labels" — reporting an infrastructure error as a statistical shortfall, in the
one log line (src/queue/job-dispatch.ts:281-282) an operator reads to see what the daily tick did.
Requirements
calibrateActThreshold must compare the total label count against the floor at the δ it actually tests
with. Concretely: after computing candidates, compute effectiveNeeded = minimumCalibrationLabels(alpha, delta / candidates.length) and use it for both the insufficient_labels decision and the per-candidate
n >= needed gate. The pre-scan check at line 140 must remain (it is what guarantees candidates.length >= 1
before the division) — keep it, then re-check against effectiveNeeded once candidates is known.
insufficient_labels's needed field must carry effectiveNeeded, not the full-δ value, so the operator
message (src/review/risk-control-wire.ts:156) states the number of labels actually required.
no_certifiable_threshold must remain reachable and must mean only what its doc says: ample labels for the
tested δ, but no λ's bound clears α. A zero-error calibration set below effectiveNeeded must return
insufficient_labels.
validateCalibrationPayload (src/review/risk-control.ts:200) enforces nAtLambda >= minimumCalibrationLabels(v.alpha, v.delta).
It receives a stored payload that carries the ORIGINAL delta, not the split one, so this check must stay
exactly as written — do not change it. Add a comment at that check naming why it uses the unsplit δ while the
scan uses the split one.
runRiskControlRecalibration's catch (src/review/risk-control-wire.ts:196-199) must stop reporting an
error as insufficient_labels. Widen the returned map's value type to
CalibrationResult["status"] | "error" and emit "error" there.
⚠️ Required pattern: mirror the existing structure of calibrateActThreshold — one pure function, the same
four result shapes, the same minimumCalibrationLabels helper. What does NOT satisfy this issue: adding a
new exported "effective floor" function that callers must remember to use instead of fixing the scan;
changing delta semantics so the stored payload no longer records the originally-requested δ; introducing a
fifth CalibrationResult status; or leaving runRiskControlRecalibration's catch returning a statistical
status.
Deliverables
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
changing the needed value reported in the message without changing which status is returned — does not
resolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. Both touched files are inside src/**, which
is inside coverage.include, so every changed line and branch is measured and gated. Both arms of every
changed conditional need a test: the below-floor and at-or-above-floor arms of the new label check, the
candidates.length === 1 edge (where delta / 1 === delta and behaviour must be identical to today), and both
the throwing and non-throwing paths of runRiskControlRecalibration's try/catch. The regression test named in
Deliverable 2 is mandatory.
Expected Outcome
An arm with zero adjudication errors and too few labels is told to collect more labels, with the correct
target number, instead of being told its error rate is too high. The #9048 status split holds under #9066's
Bonferroni correction, and an infrastructure failure is no longer logged as a statistical shortfall.
Links & Resources
src/review/risk-control.ts (lines 138-190, 200-211), src/review/risk-control-wire.ts (lines 130-202),
src/queue/job-dispatch.ts:278-284. Closed precedents: #9048 (the status split), #9066 (the Bonferroni
correction that this issue reconciles with it).
Context
calibrateActThreshold(src/review/risk-control.ts:138) has two refusal statuses that #9048 deliberatelysplit apart because they need opposite remediations:
insufficient_labels— "go collect more labels";no_certifiable_threshold— "you have plenty of labels, the error rate is too high; investigate precisionor accept a weaker alpha" (field doc at
src/review/risk-control.ts:108-121, distinct audit event atsrc/review/risk-control-wire.ts:159-172).#9066 then added the Bonferroni multiplicity correction: every candidate's Clopper–Pearson bound is now tested
at
testDelta = delta / candidates.length(src/review/risk-control.ts:148). The label-supply gate was notupdated with it. Line 139 still computes:
and line 161 gates each candidate on
if (n >= needed)using that same full-δ value.The zero-error floor is
ceil(ln(δ)/ln(1−α)), which grows as δ shrinks. At α=0.05, δ=0.05 the full-δ floor is59 labels; at δ/K with K=59 distinct observed confidences it is 138. On the merge arm (α=0.005) the full-δ
floor is 598 and at K=300 it is 1736. So for every arm whose usable label count lands in the gap
[minimumCalibrationLabels(α, δ), minimumCalibrationLabels(α, δ/K)), a calibration set with zero errorsclears line 140, computes bounds at every candidate, certifies nothing, and falls through to line 189 —
returning
no_certifiable_threshold.recalibrateArmthen writesrisk_control_no_certifiable_thresholdwith the detail string"59 labels available but no threshold achieves α=0.05"(src/review/risk-control-wire.ts:170), telling anoperator to investigate close/merge precision on an arm that made zero mistakes and whose only real
shortfall is label count. That is precisely the misdirection #9048 was filed to end, reintroduced by #9066.
A second, smaller instance of the same conflation sits in the same file:
src/review/risk-control-wire.ts:196-199catches any read/write failure and setssummary[arm] = "insufficient_labels"— reporting an infrastructure error as a statistical shortfall, in theone log line (
src/queue/job-dispatch.ts:281-282) an operator reads to see what the daily tick did.Requirements
calibrateActThresholdmust compare the total label count against the floor at the δ it actually testswith. Concretely: after computing
candidates, computeeffectiveNeeded = minimumCalibrationLabels(alpha, delta / candidates.length)and use it for both theinsufficient_labelsdecision and the per-candidaten >= neededgate. The pre-scan check at line 140 must remain (it is what guaranteescandidates.length >= 1before the division) — keep it, then re-check against
effectiveNeededoncecandidatesis known.insufficient_labels'sneededfield must carryeffectiveNeeded, not the full-δ value, so the operatormessage (
src/review/risk-control-wire.ts:156) states the number of labels actually required.no_certifiable_thresholdmust remain reachable and must mean only what its doc says: ample labels for thetested δ, but no λ's bound clears α. A zero-error calibration set below
effectiveNeededmust returninsufficient_labels.validateCalibrationPayload(src/review/risk-control.ts:200) enforcesnAtLambda >= minimumCalibrationLabels(v.alpha, v.delta).It receives a stored payload that carries the ORIGINAL
delta, not the split one, so this check must stayexactly as written — do not change it. Add a comment at that check naming why it uses the unsplit δ while the
scan uses the split one.
runRiskControlRecalibration's catch (src/review/risk-control-wire.ts:196-199) must stop reporting anerror as
insufficient_labels. Widen the returned map's value type toCalibrationResult["status"] | "error"and emit"error"there.Deliverables
calibrateActThresholdinsrc/review/risk-control.tscomputes its label floor atdelta / candidates.lengthand returns
insufficient_labels(notno_certifiable_threshold) for a zero-error set below that floor.insufficient_labels.neededreports the split-δ floor; a new named regression test in the risk-controlunit test file asserts a 59-pair, zero-error, 59-distinct-confidence input returns
status: "insufficient_labels"withneeded === 138, where today it returnsno_certifiable_threshold.no_certifiable_thresholdis still returned for a set that clears the split-δ floor buthas an error rate no λ can certify — the status must not become unreachable.
runRiskControlRecalibrationinsrc/review/risk-control-wire.tsreturns"error"(not"insufficient_labels") from its catch, with the return type widened accordingly and a test covering thethrowing path.
validateCalibrationPayloadis unchanged in behaviour, with an added comment explaining the unsplit-δcheck, and an existing-behaviour test pinned so a later edit cannot silently loosen it.
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
changing the
neededvalue reported in the message without changing which status is returned — does notresolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. Both touched files are inside
src/**, whichis inside
coverage.include, so every changed line and branch is measured and gated. Both arms of everychanged conditional need a test: the below-floor and at-or-above-floor arms of the new label check, the
candidates.length === 1edge (wheredelta / 1 === deltaand behaviour must be identical to today), and boththe throwing and non-throwing paths of
runRiskControlRecalibration's try/catch. The regression test named inDeliverable 2 is mandatory.
Expected Outcome
An arm with zero adjudication errors and too few labels is told to collect more labels, with the correct
target number, instead of being told its error rate is too high. The #9048 status split holds under #9066's
Bonferroni correction, and an infrastructure failure is no longer logged as a statistical shortfall.
Links & Resources
src/review/risk-control.ts(lines 138-190, 200-211),src/review/risk-control-wire.ts(lines 130-202),src/queue/job-dispatch.ts:278-284. Closed precedents: #9048 (the status split), #9066 (the Bonferronicorrection that this issue reconciles with it).