feat(engine): pure BacktestCase corpus builder from fired/override event pairs#8095
feat(engine): pure BacktestCase corpus builder from fired/override event pairs#8095RealDiligent wants to merge 1 commit into
Conversation
…ent pairs (JSONbored#8083) computeRulePrecision (JSONbored#7982) collapses a rule's fired/override history into one aggregate precision number; a backtest instead needs each PAIRED case kept as an individual labeled record. Add calibration/backtest-corpus.ts: the BacktestCase type and pure buildBacktestCorpus(ruleId, fired, overrides), which joins a RuleFiredEvent to its judging HumanOverrideEvent by (ruleId, targetKey). Pairing mirrors computeRulePrecision's "only the decided ones count" discipline: an unjudged firing is excluded, never emitted unlabeled. A re-fired, re-judged target pairs each firing with the override closest in time strictly after that specific firing, falling back to the most recent override when none strictly follows -- one case per fired event, never duplicates. The ruleId filter mirrors signal-tracking.ts's private overrideMatchesRule (deliberately not exported from there; this change is additive-only in a new file). metadata is omitted entirely when the fired event carries none, matching RuleFiredEvent's own optional-property discipline. Barrel export added directly after the signal-tracking line in index.ts, per the issue's placement requirement. Tests land in both suites: packages/loopover-engine/test/backtest-corpus.test.ts (the deliverable, mirroring signal-tracking.test.ts's node:test style against dist/) and test/unit/backtest-corpus.test.ts (root vitest, importing the source directly so the repo's coverage gate measures every line and branch: exclusion, single-pair labeling, nearest-following and most-recent fallback pairing, cross-rule/cross-target filtering, and present/absent metadata).
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8095 +/- ##
==========================================
- Coverage 92.01% 92.01% -0.01%
==========================================
Files 753 754 +1
Lines 77157 77182 +25
Branches 23330 23336 +6
==========================================
+ Hits 70998 71019 +21
Misses 5053 5053
- Partials 1106 1110 +4
Flags with carried forward coverage won't be shown. Click here to find out more.
|
loopover-orb
Bot
added
the
gittensor:feature
|
Caution 🛑 LoopOver review result - reject/close recommendedReview updated: 2026-07-22 22:09:22 UTC
Review summary Nits — 4 non-blocking
Why this is blocked
📋 Copy for AI agents — paste into your coding agentDecision drivers
Context & advisory signals — never blocks the verdict
Linked issue satisfactionAddressed Review context
Contributor next steps
Signal definitions
🧪 Chat with LoopOverAsk LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.
Full command reference: https://loopover.ai/docs/loopover-commands 🧪 Experimental — new and may change. 🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.
|
|
LoopOver is closing this pull request on the maintainer's behalf (Linked issue overlaps another open PR; duplicate of open PR #8093). This is an automated maintenance action — to pursue this change, please open a new pull request with the issues resolved. Closed PRs may be analyzed later to improve review accuracy, but they are not automatically reopened or re-reviewed. |
…8605) Live diagnosis on the self-hosted ORB box: metagraphed's secret_leak gate false-positive rate climbed from 38% to 56% over 6 hours, holding mergeable PRs. Root cause wasn't a pattern-match false positive -- it was the fail-closed "content exceeded the scan cap" block on regenerated OpenAPI/JSON-schema artifacts. metagraphed's openapi.json is ~1.9MB and api-components.schema.json ~514KB, both well past the old 512,000-char cap; three recent PRs (#8106, #8095, #8005) were all held for this reason and all merged anyway once manually verified clean. The underlying fetcher (grounding-wire.ts's makeGithubFileFetcher) already requests the raw+json media type specifically to bypass GitHub's Contents API ~1MB base64-JSON envelope ceiling, so the real limit was always this local constant, not GitHub's. Raising it only expands scan coverage -- more content becomes fetchable-and-scannable instead of being marked incomplete-and-blocked -- so this can't weaken detection on anything the old cap already caught. 4MB gives headroom above the largest observed real case (loopover's own openapi.json is already at 525KB, past the old cap too). Fetch count/concurrency stay capped separately, unaffected.
1 participant
Summary
computeRulePrecision(Extract a shared, deployment-agnostic calibration/signal-tracking module for ORB + AMS #7982) collapses a rule's fired/override history into one aggregate precision number; a backtest instead needs each PAIRED case kept as an individual labeled record. Addspackages/loopover-engine/src/calibration/backtest-corpus.ts: theBacktestCasetype (exact spec shape) and purebuildBacktestCorpus(ruleId, fired, overrides), joining aRuleFiredEventto its judgingHumanOverrideEventby(ruleId, targetKey).computeRulePrecision's "only the decided ones count" discipline: an unjudged firing is excluded, never emitted unlabeled. A re-fired, re-judged target pairs each firing with the override closest in time strictly after that specific firing, falling back to the most recent override when none strictly follows — one case per fired event, never duplicates, documented in the function's doc comment incomputeRulePrecision's own style. TheruleIdfilter mirrorssignal-tracking.ts's privateoverrideMatchesRule(deliberately NOT exported from there — this change is additive-only in a new file, per the issue's boundary).metadatais omitted entirely when the fired event carries none, matchingRuleFiredEvent's own optional-property discipline. Pure TypeScript: no IO, no DB, no imports fromsrc/,packages/loopover-miner, or any host adapter.signal-tracking.jsline inpackages/loopover-engine/src/index.ts, exactly where the issue requires.Scope
type(scope): short summaryConventional Commit format, for examplefix(api): restore profile access checks.CONTRIBUTING.mdand does not reintroduce GitHub Pages, VitePress,site/, orCNAME.Closes #123) — a linked open issue is required for every contributor PR.Validation
git diff --checknpm run actionlintnpm run typechecknpm run test:coveragelocally;codecov/patchrequires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.npm run test:workersnpm run build:mcpnpm run test:mcp-packnpm run ui:openapi:checknpm run ui:lintnpm run ui:typechecknpm run ui:buildnpm audit --audit-level=moderateIf any required check was skipped, explain why:
npm run build && npm run test --workspace @loopover/engine(614 tests green, including the newbacktest-corpus.test.tsmirroringsignal-tracking.test.ts's node:test style againstdist/), the roottest/unit/backtest-corpus.test.ts(8 tests green), and the full rootnpm run typecheck. Verified per-diff-line coverage via lcov: every changed line and branch inbacktest-corpus.tsis covered by the root suite (has-override vs no-override; single vs multiple overrides including the no-strictly-following fallback; matching vs non-matchingruleId; present vs absentmetadata), and simulated the scoped-CI shard condition with the exact CI invocation (--changed=origin/main --coverage.all=false): the lcov is non-empty and contains the changed instrumented files.actionlint/workers/mcp/ui checks are untouched surfaces; CI runs them all.Safety
UI Evidencesection below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.UI Evidence
Not applicable — engine-only pure-function addition (no UI, docs, or extension surface touched).
Notes
packages/loopover-engine/test/backtest-corpus.test.tsis the issue's deliverable (node:test againstdist/, thesignal-tracking.test.tsprecedent), andtest/unit/backtest-corpus.test.tsimports the source directly so the repo's Codecov patch gate measures every changed line and branch — the engine workspace's ownnode --testrun does not feed Codecov.