Skip to content

orb(calibration): an empty slop band reports `mergeRate: 0 #9641

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

buildSlopOutcomeCalibration (src/services/outcome-calibration.ts:78) emits one row per slop band. It always
emits all four bands (SLOP_BAND_ORDER.map, line 99), including bands with zero resolved PRs. For those it
reports a fabricated rate:

return { band, sampleSize, merged, closed, mergeRate: sampleSize > 0 ? round(merged / sampleSize) : 0 };

Five lines later the same function does the opposite for the pooled figure:

overallMergeRate: totalResolved > 0 ? round(totalMerged / totalResolved) : null,

The null-not-0 rule is stated repeatedly across this subsystem as load-bearing:
RulePrecisionReport.precision ("never coerced to 0 or 1",
packages/loopover-engine/src/calibration/signal-tracking.ts:65-67), ReliabilityBucket.precision ("null,
never 0, when cases sits below the curve's sample floor",
packages/loopover-engine/src/calibration/reliability-curve.ts:12-16), BenchmarkGroundTruthSet.coverage.unresolvedRate
("null for an empty work-unit set — never 0"), and PublicRulePrecisionRow.precision. The slop-band row is
the one place that breaks it, and the type says so: SlopBandCalibration.mergeRate is number, not
number | null (line 21).

The consequence is visible in the MCP surface, which renders the value with no sample guard:
packages/loopover-mcp/bin/loopover-mcp.ts:2587 emits
- ${band.band}: ${rate(band.mergeRate)} merge rate over ${band.sampleSize ?? 0} PR(s) — so a band nobody
has any data for reads as "0% merge rate", which for a slop-band calibration table means "every PR in this band
was closed", the strongest possible discrimination claim. The UI card does guard
(apps/loopover-ui/src/components/site/app-panels/slop-band-calibration-card.tsx:41 gates on hasSamples),
which is why the type-level defect has not been noticed: two consumers of the same field disagree about whether
0 means zero or unknown.

Requirements

  • SlopBandCalibration.mergeRate becomes number | null, and buildSlopOutcomeCalibration returns null for
    a band with sampleSize === 0, matching overallMergeRate's existing treatment in the same function.
  • computeDiscriminates (src/services/outcome-calibration.ts:112-120) must keep working unchanged in
    behaviour. It already filters on sampleSize >= MIN_BAND_SAMPLE before comparing, so no null can reach the
    comparison — but the comparison must be made null-safe at the type level without widening which bands it
    considers.
  • The MCP renderer at packages/loopover-mcp/bin/loopover-mcp.ts:2587 must render a null mergeRate as n/a
    (not 0%), matching how the UI card already presents an unsampled band.
  • src/openapi/schemas.ts must be updated for the nullable field and npm run ui:openapi re-run and committed.
    The UI's slop-band card must accept the nullable type without a behaviour change.

⚠️ Required pattern: mirror overallMergeRate in the SAME function (src/services/outcome-calibration.ts:107)
— the identical denominator > 0 ? round(...) : null shape. What does NOT satisfy this issue: omitting
zero-sample bands from the bands array (the four-band roster is what makes a missing band legible);
introducing a separate hasSamples boolean instead of making the rate nullable; or fixing the type without
fixing the MCP renderer, which would then print n/a as NaN% or 0%.

Deliverables

  • SlopBandCalibration.mergeRate is number | null and buildSlopOutcomeCalibration returns null for a
    zero-sample band.
  • computeDiscriminates is null-safe and a test asserts its verdict is unchanged for a fixture with one
    zero-sample band and three sampled bands.
  • packages/loopover-mcp/bin/loopover-mcp.ts renders a null mergeRate as n/a, asserted by a test.
  • src/openapi/schemas.ts reflects the nullable field and the regenerated ui:openapi artifact is
    committed.
  • A named regression test asserting a corpus with zero high-band resolved PRs produces
    bands.find(b => b.band === "high").mergeRate === null, where today it is 0.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
widening the type without updating the MCP renderer — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. coverage.include covers src/**/*.ts AND
packages/loopover-mcp/bin/**/*.ts, so src/services/outcome-calibration.ts, src/openapi/schemas.ts and the
MCP renderer change are ALL measured and gated — the MCP side is not exempt, and both arms of its new
null-vs-number branch need a test. Both arms of the new sampleSize > 0 ternary need a test too. Only
apps/loopover-ui/** sits outside coverage.include, so Codecov does not gate the UI card's type widening —
that change must still compile and keep its existing behaviour.

Expected Outcome

A slop band with no resolved PRs reads as "no data" on every consumer, instead of reading as a 0% merge rate —
the strongest discrimination claim the table can make — on the MCP surface.

Links & Resources

src/services/outcome-calibration.ts:21, 99-120, packages/loopover-mcp/bin/loopover-mcp.ts:2587,
apps/loopover-ui/src/components/site/app-panels/slop-band-calibration-card.tsx:41,
src/openapi/schemas.ts. Precedents for the null-not-zero rule:
packages/loopover-engine/src/calibration/reliability-curve.ts:12-16,
packages/loopover-engine/src/calibration/signal-tracking.ts:65-67.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions