feat(benchmark): multi-class proposal scorer on the shared primitives (#9262) - #9599
Merged
Conversation
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-28 23:36:43 UTC
Review summary Nits — 1 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 6e9a31d | Commit Preview URL Branch Preview URL |
Jul 28 2026, 11:20 PM |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9599 +/- ##
==========================================
+ Coverage 89.89% 89.91% +0.01%
==========================================
Files 883 884 +1
Lines 111692 111911 +219
Branches 26579 26605 +26
==========================================
+ Hits 100408 100623 +215
Misses 9992 9992
- Partials 1292 1296 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
JSONbored
force-pushed
the
feat/benchmark-scorer-9262
branch
from
July 28, 2026 21:52
d1e8271 to
f41fd97
Compare
JSONbored
force-pushed
the
feat/benchmark-scorer-9262
branch
from
July 28, 2026 21:54
f41fd97 to
6426fc9
Compare
JSONbored
force-pushed
the
feat/benchmark-scorer-9262
branch
from
July 28, 2026 22:55
6426fc9 to
4b0caee
Compare
…#9262) Scoring built ON scoreBacktest and compareBacktestScores, deliberately not beside them: this module is an adapter that reshapes (proposal, ground truth) pairs into the exact inputs those functions already take. A second scoring implementation is the drift the issue exists to prevent, so the guarantee is mechanical -- an equivalent binary case scored through this adapter and through scoreBacktest directly produces the identical report, asserted in the tests. The reuse borrows the primitive's positive-class slot per action: for action A the positive class is "the realized action was A" and the classifier answers "the agent proposed A". The `ruleId` slot carries the action name, which makes each per-action report self-labeling and makes compareBacktestScores' rule-mismatch throw double as a guard against comparing two different actions. AGGREGATION -- the recorded decision, in the module header. Macro is the headline, micro published alongside. Under single-label scoring micro precision, micro recall and accuracy are all the same number and it is reported once, honestly labeled, rather than three times as if independent; being pooled, it is dominated by the most frequent action, so an agent answering "merge" to everything scores 0.8 on this corpus while being worthless. Macro weights each action equally and exposes exactly that agent through its floor-level close/request_changes numbers, which is why it answers the benchmark's actual question. An action with no realized instances has null metrics and is EXCLUDED from the macro mean rather than counted as 0, and actionsScored publishes how many entered it. Coverage follows #9215: coverage = decided / (decided + abstained); abstentions lower coverage and never enter a confusion-matrix count. Silence and a declared abstention are the same act and are scored the same way. #9261's unresolved units leave the denominator before scoring starts, and proposals for units outside the snapshot are counted as unscorable -- ignored for scoring, so padding inflates nothing, but visible, because a nonzero count means the agent answered a different question than the one asked. The Pareto floor extends to the multi-class case: ANY regressed action decides the verdict, so gaining on merge while losing on close is a trade, and a trade is not a win. Closes #9262
JSONbored
force-pushed
the
feat/benchmark-scorer-9262
branch
from
July 28, 2026 23:17
4b0caee to
6e9a31d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Multi-class scoring built on
scoreBacktestandcompareBacktestScores, deliberately not beside them. This module is an adapter that reshapes (proposal, ground truth) pairs into the exact inputs those functions already take — a second scoring implementation is precisely the drift #9262 exists to prevent, so the guarantee is mechanical: an equivalent binary case scored through this adapter and throughscoreBacktestdirectly produces the identical report, asserted as the suite's first test.The reuse works by borrowing the primitive's positive-class slot per action: for action A, the positive class is "the realized action was A" and the classifier answers "the agent proposed A". The
ruleIdslot carries the action name, which makes each per-action report self-labeling and makescompareBacktestScores' rule-mismatch throw double as a guard against accidentally comparing two different actions.The aggregation decision (issue requirement 2), recorded in the module header
Macro is the headline; micro is published alongside.
Under single-label scoring — one prediction per work unit — micro precision, micro recall and plain accuracy are all the same number, so it's reported once, honestly labeled, rather than three times as if independent. Being pooled, it's dominated by whichever action is most frequent, which here is overwhelmingly
merge: an agent that answers "merge" to everything scores 0.8 micro on the test corpus while never having made a decision. Macro weights each action equally and exposes exactly that agent through its floor-levelclose/request_changesnumbers — and since the benchmark's question is "can this agent make maintainer decisions", including the rare expensive ones, macro is the number that answers it. Both are published because a benchmark reporting only its headline invites the reader to reconstruct the other one wrongly.An action with no realized instances has null metrics and is excluded from the macro mean rather than counted as 0;
actionsScoredpublishes how many actions entered it, so "macro over 1 of 5 actions" can't be misread as "macro over 5".Coverage and denominators (#9215 semantics)
coverage = decided / (decided + abstained); abstentions lower coverage and never enter a confusion-matrix count. Silence and a declared{kind: "abstain"}are the same act and are scored the same way — scoring them differently would reward whichever one an agent's emitter happened to produce. #9261'sunresolvedunits leave the denominator before scoring starts, and proposals for units outside the snapshot are counted asunscorableProposals: ignored for scoring so padding inflates nothing, but visible, because a nonzero count means the agent is answering a different question than the one asked. Every rate isnullrather than0when its denominator is empty.Pareto floor across actions (requirement 4)
Extended to the multi-class case: any regressed action decides the verdict. The regression test uses a clean trade — the candidate buys perfect
mergerecall (0.33 → 1) by spendingcloserecall (1 → 0.33) with precision pinned at 1 on both sides — and asserts the verdict isregressed, not a wash.Tests (13 node:test cases, 100% of branches)
Anti-drift equality against the primitive (plus hand-checked TP/FP/TN/FN); all five actions one-vs-rest; the answer-merge-to-everything regression showing micro 0.8 against a floored macro; null-metric exclusion from the macro mean; abstention/coverage semantics with declared and silent abstentions treated identically; unresolved exclusion; unscorable proposals; last-proposal-wins resubmission; the empty-corpus null arms; report identity fields; the Pareto trade regression; improved/unchanged verdicts with per-action detail; and the cross-subject throw.
Closes #9262