You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.
Context
EvalScoreRecordScore (src/review/eval-score-records.ts:28-35) carries decided, abstained and coverage. #9215 defines the relationship: coverage = decided / (decided + abstained).
Its doc comment (lines 79-83) explains two of these three deliberately:
recall and abstained do not apply to this work-unit kind: ORB's gate rules fire deterministically (no
agent choosing to abstain) and this data measures precision, not a false-negative rate. recall is null
(genuinely inapplicable, never a misleading 0); abstained is 0 (there is no abstention concept here, so 0 is the correct value, not a masked null).
coverage is not mentioned. And by the module's own reasoning it is not unknown: with abstained asserted to
be exactly 0 and decided a real count, decided / (decided + 0) is 1 for any decided > 0. Publishing null tells a consumer "this record does not state its coverage" when the record has in fact just asserted
full coverage two lines below. A validator re-deriving the score per #9215's contract computes 1 and
disagrees with the published field — the one thing an independently-re-derivable record format exists to
prevent.
The decided === 0 case is genuinely undefined (0/0) and must stay null, exactly as PublicRulePrecisionRow.precision stays null below its own floor.
Export the formula as a named pure helper in the same module (evalScoreCoverage(decided, abstained))
returning number | null, so benchmark: leaderboard artifacts as spec-conformant EvalScoreRecords #9265's benchmark_run emitter uses the identical definition rather than
re-deriving it. null when decided + abstained === 0.
The doc comment at lines 79-83 must be extended to state what coverage means for this work-unit kind and
why, matching the existing per-field justifications for recall and abstained.
recordDigest is computed over the whole record (finalizeRecord, line 68), so changing coverage changes
the digest. verifyEvalScoreRecordDigest must still round-trip; no stored-record migration is needed because
records are built on read, not persisted.
⚠️ Required pattern: mirror the null-vs-zero justification style already used for recall and abstained in
this same doc comment, and the denominator > 0 ? ... : null shape used by PublicRulePrecisionRow.precision (src/review/public-rule-precision.ts:74). What does NOT satisfy this
issue: hardcoding coverage: 1; leaving coverage: null and only documenting it; adding the helper without
using it in buildEvalScoreRecordsFromRulePrecision; or changing abstained away from 0.
Deliverables
evalScoreCoverage(decided, abstained) is exported from src/review/eval-score-records.ts and returns null when decided + abstained === 0, else the quotient.
buildEvalScoreRecordsFromRulePrecision uses it; a rule row with decided: 12 emits coverage: 1.
A test asserting a rule row with decided: 0 still emits coverage: null.
A test asserting verifyEvalScoreRecordDigest returns true for records built after the change.
The doc comment states the coverage rule for the outcome_confirmed_precision work-unit kind.
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
computing coverage inline without exporting the shared helper — does not resolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. src/review/eval-score-records.ts is inside src/**, which is inside coverage.include, so it is measured and gated. Both arms of the new decided + abstained === 0 conditional need a test, as named in Deliverables 2 and 3.
Expected Outcome
Every emitted EvalScoreRecord states a coverage a third party re-deriving it from decided and abstained
computes to the same value, instead of declaring unknown a quantity the record itself pins down.
Links & Resources
src/review/eval-score-records.ts:28-35, 73-118, 137-143, src/review/public-rule-precision.ts:26-35.
Related: #9215 (record shape and coverage semantics), #9265 (the benchmark_run emitter that will reuse the
helper), #9266.
Context
EvalScoreRecordScore(src/review/eval-score-records.ts:28-35) carriesdecided,abstainedandcoverage. #9215 defines the relationship:coverage = decided / (decided + abstained).buildEvalScoreRecordsFromRulePrecision(line 85) emits:Its doc comment (lines 79-83) explains two of these three deliberately:
coverageis not mentioned. And by the module's own reasoning it is not unknown: withabstainedasserted tobe exactly
0anddecideda real count,decided / (decided + 0)is1for anydecided > 0. Publishingnulltells a consumer "this record does not state its coverage" when the record has in fact just assertedfull coverage two lines below. A validator re-deriving the score per #9215's contract computes
1anddisagrees with the published field — the one thing an independently-re-derivable record format exists to
prevent.
The
decided === 0case is genuinely undefined (0/0) and must staynull, exactly asPublicRulePrecisionRow.precisionstays null below its own floor.Requirements
buildEvalScoreRecordsFromRulePrecisionmust computecoveragefrom the record's owndecidedandabstainedrather than hardcodingnull, expressed as the Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215 formula(
decided / (decided + abstained)) so a future kind with real abstentions needs no second implementation.evalScoreCoverage(decided, abstained))returning
number | null, so benchmark: leaderboard artifacts as spec-conformant EvalScoreRecords #9265'sbenchmark_runemitter uses the identical definition rather thanre-deriving it.
nullwhendecided + abstained === 0.coveragemeans for this work-unit kind andwhy, matching the existing per-field justifications for
recallandabstained.recordDigestis computed over the whole record (finalizeRecord, line 68), so changingcoveragechangesthe digest.
verifyEvalScoreRecordDigestmust still round-trip; no stored-record migration is needed becauserecords are built on read, not persisted.
Deliverables
evalScoreCoverage(decided, abstained)is exported fromsrc/review/eval-score-records.tsand returnsnullwhendecided + abstained === 0, else the quotient.buildEvalScoreRecordsFromRulePrecisionuses it; a rule row withdecided: 12emitscoverage: 1.decided: 0still emitscoverage: null.verifyEvalScoreRecordDigestreturns true for records built after the change.outcome_confirmed_precisionwork-unit kind.All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
computing coverage inline without exporting the shared helper — does not resolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted.
src/review/eval-score-records.tsis insidesrc/**, which is insidecoverage.include, so it is measured and gated. Both arms of the newdecided + abstained === 0conditional need a test, as named in Deliverables 2 and 3.Expected Outcome
Every emitted
EvalScoreRecordstates a coverage a third party re-deriving it fromdecidedandabstainedcomputes to the same value, instead of declaring unknown a quantity the record itself pins down.
Links & Resources
src/review/eval-score-records.ts:28-35, 73-118, 137-143,src/review/public-rule-precision.ts:26-35.Related: #9215 (record shape and coverage semantics), #9265 (the
benchmark_runemitter that will reuse thehelper), #9266.