Skip to content

benchmark: leaderboard artifacts as spec-conformant EvalScoreRecords #9265

Description

@JSONbored

Context

Sub-issue of #9216 (frozen-repo benchmark harness), epic #8534. Requirement 5 of that issue is explicit: leaderboard artifacts are exactly the eval-interface's consumable score records — no parallel format. This issue emits them.

Requirements

  1. Emit EvalScoreRecord as specified on Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider #9215workUnit.kind === "benchmark_run", carrying benchmarkId and snapshotRef. No benchmark-specific record shape, no extra top-level fields.
  2. Commitments block must be fully populated: snapshotChecksum (benchmark: frozen-repo snapshot builder — leak-proof repo state at commit T #9259), scoringRuleVersion, window bounds, splitSeed and heldOutFraction (benchmark: anti-overfit controls — seeded held-out, per-repo splits, submission caps, rotation #9263). A record whose commitments do not let a third party re-derive the number is not publishable — that is the entire point of the block.
  3. Trust tier from the actual execution (benchmark: untrusted candidate-agent execution — sandbox, limits, attested when hardware allows #9264), never asserted optimistically: attested only with a real envelope, otherwise reproducible.
  4. Held-out scores obey benchmark: anti-overfit controls — seeded held-out, per-repo splits, submission caps, rotation #9263's publication cadence — the emitter is where that policy is enforced, so a leaderboard cannot accidentally leak held-out membership by publishing per-submission.
  5. Digest-committed and immutable: recordDigest over canonical JSON; a corrected score is a new record with a later issuedAt, never a mutation.

Deliverables

Expected outcome

The leaderboard and the validator-consumable score feed are the same data, so a number on the board is a number a third party can verify.

References

Sub-issue of #9216. Record shape: #9215. Depends on #9259, #9262, #9263, and the serving endpoint issue. Epic #8534.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions