Skip to content

orb(risk-control): the zero-error label floor ignores the Bonferroni δ/K split #9637

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

calibrateActThreshold (src/review/risk-control.ts:138) has two refusal statuses that #9048 deliberately
split apart because they need opposite remediations:

  • insufficient_labels — "go collect more labels";
  • no_certifiable_threshold — "you have plenty of labels, the error rate is too high; investigate precision
    or accept a weaker alpha" (field doc at src/review/risk-control.ts:108-121, distinct audit event at
    src/review/risk-control-wire.ts:159-172).

#9066 then added the Bonferroni multiplicity correction: every candidate's Clopper–Pearson bound is now tested
at testDelta = delta / candidates.length (src/review/risk-control.ts:148). The label-supply gate was not
updated with it. Line 139 still computes:

const needed = minimumCalibrationLabels(alpha, delta);

and line 161 gates each candidate on if (n >= needed) using that same full-δ value.

The zero-error floor is ceil(ln(δ)/ln(1−α)), which grows as δ shrinks. At α=0.05, δ=0.05 the full-δ floor is
59 labels; at δ/K with K=59 distinct observed confidences it is 138. On the merge arm (α=0.005) the full-δ
floor is 598 and at K=300 it is 1736. So for every arm whose usable label count lands in the gap
[minimumCalibrationLabels(α, δ), minimumCalibrationLabels(α, δ/K)), a calibration set with zero errors
clears line 140, computes bounds at every candidate, certifies nothing, and falls through to line 189 —
returning no_certifiable_threshold.

recalibrateArm then writes risk_control_no_certifiable_threshold with the detail string
"59 labels available but no threshold achieves α=0.05" (src/review/risk-control-wire.ts:170), telling an
operator to investigate close/merge precision on an arm that made zero mistakes and whose only real
shortfall is label count. That is precisely the misdirection #9048 was filed to end, reintroduced by #9066.

A second, smaller instance of the same conflation sits in the same file:
src/review/risk-control-wire.ts:196-199 catches any read/write failure and sets
summary[arm] = "insufficient_labels" — reporting an infrastructure error as a statistical shortfall, in the
one log line (src/queue/job-dispatch.ts:281-282) an operator reads to see what the daily tick did.

Requirements

  • calibrateActThreshold must compare the total label count against the floor at the δ it actually tests
    with. Concretely: after computing candidates, compute effectiveNeeded = minimumCalibrationLabels(alpha, delta / candidates.length) and use it for both the insufficient_labels decision and the per-candidate
    n >= needed gate. The pre-scan check at line 140 must remain (it is what guarantees candidates.length >= 1
    before the division) — keep it, then re-check against effectiveNeeded once candidates is known.
  • insufficient_labels's needed field must carry effectiveNeeded, not the full-δ value, so the operator
    message (src/review/risk-control-wire.ts:156) states the number of labels actually required.
  • no_certifiable_threshold must remain reachable and must mean only what its doc says: ample labels for the
    tested δ, but no λ's bound clears α. A zero-error calibration set below effectiveNeeded must return
    insufficient_labels.
  • validateCalibrationPayload (src/review/risk-control.ts:200) enforces nAtLambda >= minimumCalibrationLabels(v.alpha, v.delta).
    It receives a stored payload that carries the ORIGINAL delta, not the split one, so this check must stay
    exactly as written — do not change it. Add a comment at that check naming why it uses the unsplit δ while the
    scan uses the split one.
  • runRiskControlRecalibration's catch (src/review/risk-control-wire.ts:196-199) must stop reporting an
    error as insufficient_labels. Widen the returned map's value type to
    CalibrationResult["status"] | "error" and emit "error" there.

⚠️ Required pattern: mirror the existing structure of calibrateActThreshold — one pure function, the same
four result shapes, the same minimumCalibrationLabels helper. What does NOT satisfy this issue: adding a
new exported "effective floor" function that callers must remember to use instead of fixing the scan;
changing delta semantics so the stored payload no longer records the originally-requested δ; introducing a
fifth CalibrationResult status; or leaving runRiskControlRecalibration's catch returning a statistical
status.

Deliverables

  • calibrateActThreshold in src/review/risk-control.ts computes its label floor at delta / candidates.length
    and returns insufficient_labels (not no_certifiable_threshold) for a zero-error set below that floor.
  • insufficient_labels.needed reports the split-δ floor; a new named regression test in the risk-control
    unit test file asserts a 59-pair, zero-error, 59-distinct-confidence input returns
    status: "insufficient_labels" with needed === 138, where today it returns no_certifiable_threshold.
  • A test asserting no_certifiable_threshold is still returned for a set that clears the split-δ floor but
    has an error rate no λ can certify — the status must not become unreachable.
  • runRiskControlRecalibration in src/review/risk-control-wire.ts returns "error" (not
    "insufficient_labels") from its catch, with the return type widened accordingly and a test covering the
    throwing path.
  • validateCalibrationPayload is unchanged in behaviour, with an added comment explaining the unsplit-δ
    check, and an existing-behaviour test pinned so a later edit cannot silently loosen it.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
changing the needed value reported in the message without changing which status is returned — does not
resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. Both touched files are inside src/**, which
is inside coverage.include, so every changed line and branch is measured and gated. Both arms of every
changed conditional need a test: the below-floor and at-or-above-floor arms of the new label check, the
candidates.length === 1 edge (where delta / 1 === delta and behaviour must be identical to today), and both
the throwing and non-throwing paths of runRiskControlRecalibration's try/catch. The regression test named in
Deliverable 2 is mandatory.

Expected Outcome

An arm with zero adjudication errors and too few labels is told to collect more labels, with the correct
target number, instead of being told its error rate is too high. The #9048 status split holds under #9066's
Bonferroni correction, and an infrastructure failure is no longer logged as a statistical shortfall.

Links & Resources

src/review/risk-control.ts (lines 138-190, 200-211), src/review/risk-control-wire.ts (lines 130-202),
src/queue/job-dispatch.ts:278-284. Closed precedents: #9048 (the status split), #9066 (the Bonferroni
correction that this issue reconciles with it).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions