fix(review): stop publishing a reproducibility freeze point that commits to an empty corpus - #9675
Merged
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
loopover-ui | 29461aa | Commit Preview URL Branch Preview URL |
Jul 29 2026, 06:25 AM |
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Contributor
|
Warning ⏸️ LoopOver review result - manual review recommendedReview updated: 2026-07-29 06:41:04 UTC
Review summary Nits — 6 non-blocking
Decision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. Decision record
Visual preview
Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy. Scroll preview
A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
Bundle ReportChanges will increase total bundle size by 8.99kB (0.12%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: loopover-uiAssets Changed:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9675 +/- ##
==========================================
- Coverage 90.27% 89.45% -0.82%
==========================================
Files 904 904
Lines 113177 113179 +2
Branches 26840 26841 +1
==========================================
- Hits 102171 101249 -922
- Misses 9676 10842 +1166
+ Partials 1330 1088 -242
Flags with carried forward coverage won't be shown. Click here to find out more.
|
24 tasks
Owner
Author
|
Merge-order note: #9720 also edits |
Merged
15 tasks
…its to an empty corpus `/v1/public/stats` reported a `latestBacktestRun.corpusChecksum` of 4f53cda1…, which is `checksumCases([])` — SHA-256 over the canonicalized empty case list. The fairness page rendered it as a "Reproducibility freeze point", and `/v1/public/eval-scores` published two records committed to it with `trust.tier: "reproducible"` alongside decided=460/confirmed=287. A hash over zero cases is the same 32 bytes for every rule, every window and every deployment, so it points at nothing a skeptic can re-derive anything from. Treat it the same way as a missing run: `loadPublicRulePrecision` now reports `latestBacktestRun: null` for it, which clears the freeze point from the fairness page and, through the builder's existing guard, empties the eval-scores response. The record builder additionally refuses the value on its own, since it is the code that stamps the `reproducible` tier and should not depend on its caller to be honest. The scores are unaffected — they come from a different dataset (`signal.human_override:*` audit events), so an empty corpus never means the numbers are zero, only that they are uncommitted, which is exactly the state the record spec says must not be published. Docs: the walkthrough's step 4 referenced a bare `/v1/public/stats`, which resolves against loopover.ai and 404s — only the API host serves it. Fetch it absolutely, matching the convention in what-you-can-verify.mdx. Step 1 and the track-record command shell out to `wrangler d1 execute --remote` against the deployment's own database, so they need that deployment's Cloudflare credentials and are not runnable by a third party; each step is now marked with who can actually run it instead of the page claiming nothing needs a key. The freeze point covers whichever single rule the latest run backtested, not every rule on the report, and is described that way now.
JSONbored
force-pushed
the
fix/public-verifiability-gaps
branch
from
July 29, 2026 06:21
d1ca88e to
29461aa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GET /v1/public/statsreportedlatestBacktestRun.corpusChecksum=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945, which ischecksumCases([])— SHA-256 over the canonicalized empty case list (printf '[]' | shasum -a 256reproduces it). The/fairnesspage rendered that as a "Reproducibility freeze point", and/v1/public/eval-scorespublished two records committed to it withtrust.tier: "reproducible"alongsidedecided: 460, confirmed: 287.A hash over zero cases is the same 32 bytes for every rule, every window and every deployment, so it points at nothing a skeptic can re-derive anything from — a placeholder commitment wearing a real hash's clothes.
loadPublicRulePrecisionnow reportslatestBacktestRun: nullfor that value, treating it the same way as a missing run. That single change clears the freeze point from the fairness page (the page's existing conditional already handles null) and, through the builder's existing!precision.latestBacktestRunguard, empties the eval-scores response.buildEvalScoreRecordsFromRulePrecisionadditionally refuses the value on its own, since it is the code that stamps thereproducibletier and should not depend on its caller to be honest.The scores are unaffected.
decided/confirmedcome from a different dataset (signal.human_override:*audit events over a trailing 90 days), so an empty corpus never means the numbers are zero — only that they are uncommitted, which is exactly the state #9215 says must not be published.Docs
`/v1/public/stats`, which resolves againstloopover.aiand 404s — onlyapi.loopover.aiserves it. Now an absolutecurl, matching the convention already used inwhat-you-can-verify.mdx.wrangler d1 execute --remote, reading the deployment's own D1 with that deployment's Cloudflare credentials. The page claimed "nothing needs an API key" and the sibling page claimed "Anyone can do this". Each step is now marked with who can actually run it.Deliberately not in this PR
Publishing an anonymous corpus snapshot needs a redaction design first —
BacktestCasecarriestargetKey(repo/PR identity) and arbitrary firingmetadata, and this surface is documented to exclude exactly those. MakingcorpusChecksumper-rule and window-bound, and backfilling the missingrule_firedsignal, are likewise separate changes. All three are called out as out of scope on the issue.Closes #9636
Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
test:coverage:vitest run --coverageovertest/unit/public-rule-precision.test.ts,test/unit/eval-score-records.test.tsandtest/integration/public-eval-scores-route.test.ts, restricted to the two changed modules, reports 100% statements / branches / functions / lines on both (41/41,25/25,11/11,35/35). Both sides of the newchecksum !== EMPTY_CORPUS_CHECKSUMbranch are exercised, and the constant is re-derived independently fromsha256Hex("[]")in each module's own test file so it cannot drift from the exporter.test:workersandtest:mcp-packwere not run locally — this diff touches neither the Workers pool nor the MCP package — and are left to CI.npm run docs:drift-checkandnpm run coverage-boltons:checkalso pass (no bolt-on coverage files added; the new cases live in each module's existing test file).Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.Notes on the Safety boxes: no auth/CORS/session behavior changes here — the existing negative-path suites (
public-eval-scores-route,public-decision-ledger-routes,openapi-security-parity) still pass unchanged.ui:openapi:checkconfirms the OpenAPI artifact is unchanged: the response shape is identical, only whether a freeze point is present. The existingINVARIANT: the public payload never carries target keys, repos, confidences, or private termstest still passes.UI Evidence
Captured at 1280×800,
deviceScaleFactor: 2, dark build (the only theme). The fairness page itself is not screenshotted because no page code changed — it readslatestBacktestRunfrom the live API, so its freeze-point line disappears on deploy via the existing? :atfairness-report-page.tsx:313.verify-this-review— intro + audience calloutverify-this-review— step 4 (the 404'ing link)what-you-can-verify— accuracy claimNotes
The empty corpus itself is not fixed here, only the false claim about it.
.github/workflows/calibration-advisory.ymlexports--rule-id linked_issue_scope_mismatch, whosesignal.rule_firedevents are written on one path gated onlinkedIssueSatisfactionGateMode === "block"(default"off"), sobuildBacktestCorpusreturns[]. Until that rule fires or the workflow covers a rule that does,latestBacktestRunwill now correctly readnullrather than advertising a checksum that verifies nothing.