benchmark: frozen-repo eval harness — score any agent's proposed maintainer actions against realized history

[JSONbored]

Context

Today's backtest machinery scores our own pipeline against our own history. An objective-optimization subnet regime needs a benchmark over arbitrary agents: freeze a repo at commit T, let any candidate agent propose the next maintainer actions, score the proposals against what actually happened, under anti-overfit controls, inside the attested runtime. This issue is the scoping seed for that harness.

⚠️ Blocked by the validator-facing eval-interface spec — do not decompose or start implementation before it closes. The spec fixes the score-record shape and trust tiers this harness must emit.

Scope to resolve at decomposition (after the spec)

1. Task format: frozen snapshot reference, the action-space schema (merge/close/request-changes/label/hold + parameters), prediction horizon.
2. Scoring: reuse the existing confusion-matrix + Pareto-floor primitives; realized history as ground truth with the established reversal/outcome provenance rules.
3. Anti-overfit controls: held-out windows, per-repo splits, submission caps, seeded splits — the same discipline the internal backtest gate already enforces.
4. Execution: candidate agents are untrusted by construction and run inside the attested runtime (reusing the attester harness + dev-mode stack from Epic: attested evaluation on the hosted bare-metal execution plane #8534's sub-issues).
5. Output: leaderboard artifacts are exactly the spec's consumable score records — no parallel format.

Expected outcome

The "climb a benchmark" half of the objective regime exists as a concrete, decomposable plan the moment the interface spec lands — built on primitives that already ship, not a parallel system.

References

Relates to #8534, #8082 (the primitives), and the validator-facing eval-interface spec (blocking).

[linear-code]

JSO-6390