feat(benchmark): maintainer action-space schema and proposal validator (#9260) - #9595
Merged
Conversation
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-28 22:37:09 UTC
Review summary Blockers
Nits — 5 non-blocking
Concerns raised — review before merging
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 0519426 | Commit Preview URL Branch Preview URL |
Jul 28 2026, 09:57 PM |
JSONbored
force-pushed
the
feat/benchmark-action-space-9260
branch
from
July 28, 2026 21:29
61fd12e to
7d29a65
Compare
Bundle ReportChanges will increase total bundle size by 6.38kB (0.08%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9595 +/- ##
==========================================
+ Coverage 89.85% 89.87% +0.02%
==========================================
Files 880 882 +2
Lines 111175 111488 +313
Branches 26473 26561 +88
==========================================
+ Hits 99891 100204 +313
Misses 9992 9992
Partials 1292 1292
Flags with carried forward coverage won't be shown. Click here to find out more.
|
This was referenced Jul 28, 2026
Merged
JSONbored
force-pushed
the
feat/benchmark-action-space-9260
branch
from
July 28, 2026 21:51
7d29a65 to
701257f
Compare
…the off-Worker submitter (#9277) The fourth, optional, Gittensor/SN74-audience anchoring backend. The on-chain SUBMISSION never runs in this Worker: a process on the operator's own node infrastructure (where its dedicated anchor-only hotkey lives as an operational secret, never in this repo) fetches the signed checkpoint, commits sha256(signingInput) via the commitments pallet's set_commitment(netuid, Data::Sha256), and reports the outcome back. This lands the validation boundary that report crosses: - GET /v1/public/decision-ledger/anchor-payload: the current tip's freshly signed checkpoint + the exact signingInput bytes to hash. Unauthenticated (it is the same payload Rekor/git already publish externally); no-store, since 'at' is minted per call. - POST /v1/decision-ledger/anchor-attempts: bearer-gated (LOOPOVER_LEDGER_ANCHOR_REPORT_TOKEN, fails closed unset -- the isAuthorizedIngest posture). Authentication alone cannot forge corroboration: an ok report must verify against a PUBLISHED anchor key AND its (seq, rowHash) must match the LIVE chain row; a failed report records without those checks, because a broken submitter is exactly what the public attempt log exists to make visible. - decision_ledger_anchors backend CHECK widened to 'bittensor' (migration 0201 rebuild); the public listing filter follows. Rows are byte-identical in shape to Rekor/git attempts. - backendRef carries the FULL historical-retrieval reference {netuid, blockNumber, blockHash, hotkey}: CommitmentOf is overwritten in place on-chain, so a verifier reads archive state at that block, never current chain state -- now documented step-by-step in what-you-can-verify.mdx, still clearly scoped as optional corroboration outside the default two-backend claim. Closes #9277
…backend filter The route/spec ratchet caught both new routes as undescribed -- correctly: they exist in createApp() with no OpenAPI operation, which is exactly what that check is for. - GET /v1/public/decision-ledger/anchor-payload: the signed checkpoint an off-Worker submitter commits, with the 404 arms (signing unconfigured, empty ledger) described as the honest degrades they are. - POST /v1/decision-ledger/anchor-attempts: the report ingest, with every refusal documented -- notably the 401 covering an unconfigured token (fails closed) and the 422 naming all four unverifiable outcomes, since those are what a submitter integrates against. - The anchors listing's `backend` filter gains 'bittensor', matching the widened CHECK constraint.
#9260) The frozen-repo benchmark's task format, fixed as validated types so two agents' outputs are comparable and the scorer downstream can assume well-formed input. - CLOSED action set (merge | close | request_changes | label | hold): an unrecognized action is a validation ERROR, never a silently-ignored or zero-scored entry. - Typed per-action parameters, enforced by a per-kind key allowlist, so a parameter that does not apply to the chosen action fails loudly ('labels: unexpected key for action "merge"') rather than being dropped into an answer the scorer grades differently than intended. close carries a reason CLASS (not free text, because realized history grades the class); request_changes carries a bounded concern; label carries a bounded, duplicate-free set; merge/hold carry nothing. - Abstention is FIRST-CLASS: declining a work unit is a recorded prediction feeding #9215's coverage metric, not a wrong answer, so a precise-but-narrow agent stays distinguishable from a broad-but-noisy one. An abstention carrying an action is contradictory and rejected. - The prediction horizon lives on the TASK, never the proposal, so an agent cannot pick a horizon that flatters its answer; a proposal echoes benchmarkId + snapshotRef + workUnitId so a submission cannot be replayed against a different snapshot. - Both validators are pure, never throw for ordinarily-invalid input, return one error per failing field path, and reject unknown keys -- the same contract as validateAttestationEnvelope, which they mirror deliberately. Exported from the engine barrel so a candidate agent author can validate locally against the exact code the harness runs. Closes #9260
JSONbored
force-pushed
the
feat/benchmark-action-space-9260
branch
from
July 28, 2026 21:54
701257f to
0519426
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The frozen-repo benchmark harness's task format (#9216 requirement 1), fixed as validated types. Without a closed, validated action space two agents' outputs are not comparable and the scorer has nothing stable to score against — so:
merge | close | request_changes | label | hold. Closed, not extensible-by-string: an unrecognized action is a validation error, never a silently-ignored or zero-scored entry.labels: unexpected key for action "merge") rather than being dropped into an answer the scorer would grade differently than the agent intended.closecarries a reason class (not free text — realized history grades the class);request_changescarries a bounded non-blank concern;labelcarries a bounded, duplicate-free, non-blank set;merge/holdcarry nothing.BenchmarkTask.horizonDays, fixed per benchmark), so an agent cannot pick a horizon that flatters its answer;horizonDayson a proposal is an unexpected key. A proposal echoesbenchmarkId+snapshotRef+workUnitId, so a submission cannot be replayed against a different snapshot or benchmark.{valid: true, …} | {valid: false, errors}with one error per failing field path, and reject unknown keys — the same contract asvalidateAttestationEnvelope, which they deliberately mirror. Exported from the engine barrel so a candidate agent author validates locally against the exact code the harness runs.Tests (10 node:test cases, 100% of branches)
Every action kind with exactly its own parameters plus abstention round-trips; the closed-set rejection; the inapplicable-parameter rejection for three action kinds; close reason-class bounds; request_changes blank/oversize/missing; the eight label rejection paths named by index (including duplicates); prediction-shape rejections; the full proposal envelope (version, stray keys, identity bounds, subject shape); a multi-defect proposal reporting one error per path rather than only the first; and the task validator's own envelope/horizon/
frozenAtrejection table.Closes #9260 — first of the #9216 benchmark chain, now unblocked by #9215's closed spec.