You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
packages/loopover-miner/lib/calibration-run.ts (#4248) is the "Phase 7 calibration runner" — its own header comment describes it as the module that finally connects two previously-built-but-unwired halves (the engine's computePhase7CalibrationLoop combine contract and the deterministic replay scorer). It exports runHistoricalReplayCalibrationCycle, recordCalibrationSnapshot, readCalibrationSnapshots, and latestCalibrationSnapshot, and is exercised thoroughly by test/unit/miner-calibration-run.test.ts and test/integration/miner-calibration-loop.test.ts.
None of these functions has a real caller anywhere in packages/loopover-miner/lib/** or bin/** outside their own test files — confirmed by grepping the whole package for runHistoricalReplayCalibrationCycle, recordCalibrationSnapshot, readCalibrationSnapshots, and MINER_CALIBRATION_SNAPSHOT_EVENT. loop-cli.ts (the miner's real attempt loop) has zero references to calibration. This means loopover-miner's Phase 7 calibration snapshot event is never actually written in any real deployment — the whole module is fully built and fully tested, but dead in production. (The closed issue #8185, which extended this module's snapshot payload with a backtest track record, itself assumed in its own body that "the MINER_CALIBRATION_SNAPSHOT_EVENT [is] written each cycle" and pointed at lib/loop-cli.ts as a resource — that assumption does not hold against the actual code.)
runHistoricalReplayCalibrationCycle already tolerates a missing replayRun (it degrades to historicalReplay: null, and the engine combine holds no_historical_replay_signal) — so a real caller does not need the still-unbuilt "replay executor" (replay-task-bridge.ts's own header comment names that executor as the genuinely unbuilt next step, out of scope here) to start producing real, meaningful snapshots today using just the pr_outcome signal.
The prOutcome input (PrOutcomeCalibrationInput = { mergeConfirmed, mergeFalse, closeConfirmed, closeFalse, hold?, observedAt? }, from @loopover/engine) is field-for-field identical to calibration.ts's existing CalibrationRow shape, which calibration-cli.ts's bare calibration command already computes per project via buildCalibrationReport. That report is sitting right next to the runner it should be feeding.
Requirements
⚠️ Required pattern. Mirror calibration-cli.ts's existing backtest-threshold/apply-min-rank/revert-min-rank subcommand shape exactly (parse args[0], open the event ledger, run, persist, print --json or text, return an exit code, close the ledger in finally) for a new calibration snapshot [--json] subcommand.
Add a calibration snapshot [--json] subcommand to runCalibrationCli (calibration-cli.ts).
For each CalibrationRow in the existing buildCalibrationReport(...).rows (already computed by the bare calibration command's own code path), build a PrOutcomeCalibrationInput by direct field mapping: { mergeConfirmed: row.mergeConfirmed, mergeFalse: row.mergeFalse, closeConfirmed: row.closeConfirmed, closeFalse: row.closeFalse, hold: row.hold }.
Call runHistoricalReplayCalibrationCycle({ prOutcome, repoFullName: row.project }, { eventLedger }) once per project row — do not aggregate across projects into one fleet-wide call; every other per-repo-scoped store/report in this package (the calibration report itself, recordCalibrationSnapshot's own repoFullName option, readCalibrationSnapshots's repoFullName filter) already keys on repoFullName, so per-project snapshots are the correct match for the existing read side.
Pass backtestTrackRecord into the snapshot too, not just prOutcome: runHistoricalReplayCalibrationCycle's snapshotPayloadFromResult call never populates meta.backtestTrackRecord, so the backtestTrackRecord field ams-calibration: aggregate backtest track record into the miner calibration report + snapshot event #8185 added to CalibrationSnapshotPayload is currently always null on every snapshot even after this issue's wiring lands. Extend runHistoricalReplayCalibrationCycle (or the new CLI call site — either is acceptable, but pick one and be consistent) to accept and forward a backtestTrackRecord meta value, and pass the bare calibration command's own already-computed computeAmsBacktestTrackRecord(readAmsThresholdBacktestRuns(eventLedger)) result into it (same value the bare command already prints).
Do not attempt to read .loopover-ams.yml's miner.calibration/calibration section for Phase7CalibrationConfig in this issue — no reader for that section exists yet anywhere in the package, and building one is out of scope here. Omit config entirely (the engine's computePhase7CalibrationLoop defaults to phase7LoopEnabled: false when config is omitted, so the resulting snapshot will legitimately show the loop disabled until a future issue adds config wiring — this is honest, not a bug, and must not be worked around by fabricating a config object).
calibration snapshot --json prints the array of per-project { repoFullName, snapshot } results (or an explicit "no decided predictions yet" message when the report has no rows, matching the bare command's own hasSignal handling); the text form prints one line per project.
Deliverables
calibration snapshot [--json] subcommand in calibration-cli.ts, following the backtest-threshold subcommand's exact shape.
backtestTrackRecord flows into every persisted snapshot's payload (no longer permanently null).
Per-project snapshot recording, using buildCalibrationReport's existing rows as the prOutcome source.
Test Coverage Requirements
packages/loopover-miner/** is not Codecov-gated, but npm run test:ci must stay green with full branch coverage in test/unit/miner-calibration-cli.test.ts and/or test/unit/miner-calibration-run.test.ts: cover a report with one project and one with multiple projects (one snapshot event per project, not one merged event), a report with zero rows (hasSignal: false, no snapshot attempted), the --json and text render paths, and a snapshot payload asserting backtestTrackRecord is populated (not null) when backtest runs exist and explicitly null when none do (mirroring normalizeBacktestTrackRecord's existing tolerant-degrade branch).
Expected Outcome
loopover-miner calibration snapshot actually persists calibration_snapshot ledger events from real prediction/outcome data, so readCalibrationSnapshots/latestCalibrationSnapshot (and any future consumer of the Phase 7 metric) have real history to read instead of a permanently-empty ledger type.
Context
packages/loopover-miner/lib/calibration-run.ts(#4248) is the "Phase 7 calibration runner" — its own header comment describes it as the module that finally connects two previously-built-but-unwired halves (the engine'scomputePhase7CalibrationLoopcombine contract and the deterministic replay scorer). It exportsrunHistoricalReplayCalibrationCycle,recordCalibrationSnapshot,readCalibrationSnapshots, andlatestCalibrationSnapshot, and is exercised thoroughly bytest/unit/miner-calibration-run.test.tsandtest/integration/miner-calibration-loop.test.ts.None of these functions has a real caller anywhere in
packages/loopover-miner/lib/**orbin/**outside their own test files — confirmed by grepping the whole package forrunHistoricalReplayCalibrationCycle,recordCalibrationSnapshot,readCalibrationSnapshots, andMINER_CALIBRATION_SNAPSHOT_EVENT.loop-cli.ts(the miner's real attempt loop) has zero references to calibration. This meansloopover-miner's Phase 7 calibration snapshot event is never actually written in any real deployment — the whole module is fully built and fully tested, but dead in production. (The closed issue #8185, which extended this module's snapshot payload with a backtest track record, itself assumed in its own body that "theMINER_CALIBRATION_SNAPSHOT_EVENT[is] written each cycle" and pointed atlib/loop-cli.tsas a resource — that assumption does not hold against the actual code.)runHistoricalReplayCalibrationCyclealready tolerates a missingreplayRun(it degrades tohistoricalReplay: null, and the engine combine holdsno_historical_replay_signal) — so a real caller does not need the still-unbuilt "replay executor" (replay-task-bridge.ts's own header comment names that executor as the genuinely unbuilt next step, out of scope here) to start producing real, meaningful snapshots today using just thepr_outcomesignal.The
prOutcomeinput (PrOutcomeCalibrationInput = { mergeConfirmed, mergeFalse, closeConfirmed, closeFalse, hold?, observedAt? }, from@loopover/engine) is field-for-field identical tocalibration.ts's existingCalibrationRowshape, whichcalibration-cli.ts's barecalibrationcommand already computes per project viabuildCalibrationReport. That report is sitting right next to the runner it should be feeding.Requirements
calibration snapshot [--json]subcommand torunCalibrationCli(calibration-cli.ts).CalibrationRowin the existingbuildCalibrationReport(...).rows(already computed by the barecalibrationcommand's own code path), build aPrOutcomeCalibrationInputby direct field mapping:{ mergeConfirmed: row.mergeConfirmed, mergeFalse: row.mergeFalse, closeConfirmed: row.closeConfirmed, closeFalse: row.closeFalse, hold: row.hold }.runHistoricalReplayCalibrationCycle({ prOutcome, repoFullName: row.project }, { eventLedger })once per project row — do not aggregate across projects into one fleet-wide call; every other per-repo-scoped store/report in this package (the calibration report itself,recordCalibrationSnapshot's ownrepoFullNameoption,readCalibrationSnapshots'srepoFullNamefilter) already keys onrepoFullName, so per-project snapshots are the correct match for the existing read side.backtestTrackRecordinto the snapshot too, not justprOutcome:runHistoricalReplayCalibrationCycle'ssnapshotPayloadFromResultcall never populatesmeta.backtestTrackRecord, so thebacktestTrackRecordfield ams-calibration: aggregate backtest track record into the miner calibration report + snapshot event #8185 added toCalibrationSnapshotPayloadis currently alwaysnullon every snapshot even after this issue's wiring lands. ExtendrunHistoricalReplayCalibrationCycle(or the new CLI call site — either is acceptable, but pick one and be consistent) to accept and forward abacktestTrackRecordmeta value, and pass the barecalibrationcommand's own already-computedcomputeAmsBacktestTrackRecord(readAmsThresholdBacktestRuns(eventLedger))result into it (same value the bare command already prints)..loopover-ams.yml'sminer.calibration/calibrationsection forPhase7CalibrationConfigin this issue — no reader for that section exists yet anywhere in the package, and building one is out of scope here. Omitconfigentirely (the engine'scomputePhase7CalibrationLoopdefaults tophase7LoopEnabled: falsewhenconfigis omitted, so the resulting snapshot will legitimately show the loop disabled until a future issue adds config wiring — this is honest, not a bug, and must not be worked around by fabricating a config object).calibration snapshot --jsonprints the array of per-project{ repoFullName, snapshot }results (or an explicit "no decided predictions yet" message when the report has no rows, matching the bare command's ownhasSignalhandling); the text form prints one line per project.Deliverables
calibration snapshot [--json]subcommand incalibration-cli.ts, following thebacktest-thresholdsubcommand's exact shape.backtestTrackRecordflows into every persisted snapshot's payload (no longer permanentlynull).buildCalibrationReport's existing rows as theprOutcomesource.Test Coverage Requirements
packages/loopover-miner/**is not Codecov-gated, butnpm run test:cimust stay green with full branch coverage intest/unit/miner-calibration-cli.test.tsand/ortest/unit/miner-calibration-run.test.ts: cover a report with one project and one with multiple projects (one snapshot event per project, not one merged event), a report with zero rows (hasSignal: false, no snapshot attempted), the--jsonand text render paths, and a snapshot payload assertingbacktestTrackRecordis populated (notnull) when backtest runs exist and explicitlynullwhen none do (mirroringnormalizeBacktestTrackRecord's existing tolerant-degrade branch).Expected Outcome
loopover-miner calibration snapshotactually persistscalibration_snapshotledger events from real prediction/outcome data, soreadCalibrationSnapshots/latestCalibrationSnapshot(and any future consumer of the Phase 7 metric) have real history to read instead of a permanently-empty ledger type.Links & Resources
packages/loopover-miner/lib/calibration-run.ts(runHistoricalReplayCalibrationCycle,snapshotPayloadFromResult,recordCalibrationSnapshot)packages/loopover-miner/lib/calibration-cli.ts(the file to add the subcommand to — mirrorrunBacktestThreshold)packages/loopover-miner/lib/calibration.ts(buildCalibrationReport,CalibrationRow)packages/loopover-engine/src/phase7-calibration-loop.ts(PrOutcomeCalibrationInput,computePhase7CalibrationLoop)backtestTrackRecordto the snapshot payload on the assumption the snapshot was already written each cycle)