Skip to content

orb(eval): EvalScoreRecord.coverage is published as null #9643

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

EvalScoreRecordScore (src/review/eval-score-records.ts:28-35) carries decided, abstained and
coverage. #9215 defines the relationship: coverage = decided / (decided + abstained).

buildEvalScoreRecordsFromRulePrecision (line 85) emits:

score: {
  decided: row.decided,
  confirmed: row.confirmed,
  precision: row.precision,
  recall: null,
  coverage: null,
  abstained: 0,
},

Its doc comment (lines 79-83) explains two of these three deliberately:

recall and abstained do not apply to this work-unit kind: ORB's gate rules fire deterministically (no
agent choosing to abstain) and this data measures precision, not a false-negative rate. recall is null
(genuinely inapplicable, never a misleading 0); abstained is 0 (there is no abstention concept here, so
0 is the correct value, not a masked null).

coverage is not mentioned. And by the module's own reasoning it is not unknown: with abstained asserted to
be exactly 0 and decided a real count, decided / (decided + 0) is 1 for any decided > 0. Publishing
null tells a consumer "this record does not state its coverage" when the record has in fact just asserted
full coverage two lines below. A validator re-deriving the score per #9215's contract computes 1 and
disagrees with the published field — the one thing an independently-re-derivable record format exists to
prevent.

The decided === 0 case is genuinely undefined (0/0) and must stay null, exactly as
PublicRulePrecisionRow.precision stays null below its own floor.

Requirements

  • buildEvalScoreRecordsFromRulePrecision must compute coverage from the record's own decided and
    abstained rather than hardcoding null, expressed as the Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215 formula
    (decided / (decided + abstained)) so a future kind with real abstentions needs no second implementation.
  • Export the formula as a named pure helper in the same module (evalScoreCoverage(decided, abstained))
    returning number | null, so benchmark: leaderboard artifacts as spec-conformant EvalScoreRecords #9265's benchmark_run emitter uses the identical definition rather than
    re-deriving it. null when decided + abstained === 0.
  • The doc comment at lines 79-83 must be extended to state what coverage means for this work-unit kind and
    why, matching the existing per-field justifications for recall and abstained.
  • recordDigest is computed over the whole record (finalizeRecord, line 68), so changing coverage changes
    the digest. verifyEvalScoreRecordDigest must still round-trip; no stored-record migration is needed because
    records are built on read, not persisted.

⚠️ Required pattern: mirror the null-vs-zero justification style already used for recall and abstained in
this same doc comment, and the denominator > 0 ? ... : null shape used by
PublicRulePrecisionRow.precision (src/review/public-rule-precision.ts:74). What does NOT satisfy this
issue: hardcoding coverage: 1; leaving coverage: null and only documenting it; adding the helper without
using it in buildEvalScoreRecordsFromRulePrecision; or changing abstained away from 0.

Deliverables

  • evalScoreCoverage(decided, abstained) is exported from src/review/eval-score-records.ts and returns
    null when decided + abstained === 0, else the quotient.
  • buildEvalScoreRecordsFromRulePrecision uses it; a rule row with decided: 12 emits coverage: 1.
  • A test asserting a rule row with decided: 0 still emits coverage: null.
  • A test asserting verifyEvalScoreRecordDigest returns true for records built after the change.
  • The doc comment states the coverage rule for the outcome_confirmed_precision work-unit kind.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
computing coverage inline without exporting the shared helper — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. src/review/eval-score-records.ts is inside
src/**, which is inside coverage.include, so it is measured and gated. Both arms of the new
decided + abstained === 0 conditional need a test, as named in Deliverables 2 and 3.

Expected Outcome

Every emitted EvalScoreRecord states a coverage a third party re-deriving it from decided and abstained
computes to the same value, instead of declaring unknown a quantity the record itself pins down.

Links & Resources

src/review/eval-score-records.ts:28-35, 73-118, 137-143, src/review/public-rule-precision.ts:26-35.
Related: #9215 (record shape and coverage semantics), #9265 (the benchmark_run emitter that will reuse the
helper), #9266.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions