Context
packages/loopover-miner/lib/metrics-cli.ts's collectPredictionMetricRows maps prediction-ledger rows onto the engine's MinerPredictionMetricRow shape but only ever sets conclusion, leaving the optional correct field unset on every row:
export function collectPredictionMetricRows(ledger: PredictionLedger): MinerPredictionMetricRow[] {
return ledger.readPredictions().map((entry) => ({ conclusion: entry.conclusion }));
}
The file's own header comment justifies this as deliberate: "the realized-outcome pairing (correct) is intentionally left unset: the miner has no outcome-join yet, so the correct/incorrect counters stay zero." That comment is now stale. packages/loopover-miner/lib/calibration-cli.ts (loopover-miner calibration, #4849) already implements exactly this join: toPredictionRecords projects PredictionLedgerEntry rows (keyed by targetId, which callers populate with the PR number) and toOutcomeRecords reduces the pr_outcome event stream to the latest outcome per (repoFullName, prNumber); buildCalibrationReport (lib/calibration.ts) joins the two on (project, targetId) and classifies each decided prediction as confirmed or false per merge/close.
packages/loopover-engine/src/miner-prediction-metrics.ts's MinerPredictionMetricRow type was explicitly designed for this: correct?: boolean | null ("true = matched, false = differed, null/undefined = not yet resolved"), and renderMinerPredictionMetrics already emits real loopover_miner_prediction_correct_total/loopover_miner_prediction_incorrect_total counters whenever any row carries a resolved correct value. The renderer and the join both exist and are both tested; they are simply never connected, so loopover-miner metrics's correct/incorrect counters are permanently zero in every real deployment.
Requirements
⚠️ Required pattern. Reuse calibration-cli.ts's existing join — toPredictionRecords, toOutcomeRecords, and buildCalibrationReport (or the same (project, targetId) matching logic) — to resolve each prediction row's realized outcome. Do not invent a second join implementation or change buildCalibrationReport's public contract.
collectPredictionMetricRows (or a new helper it delegates to) must resolve, for each prediction-ledger row, whether a realized pr_outcome exists for the same (repoFullName, targetId) and, if so, whether the prediction's conclusion (normalized the same way calibration.ts's normalizeDecision already normalizes merge/close/hold vocabulary) matches the realized decision.
- A prediction with no realized outcome yet (still pending) must set
correct: undefined (or omit the field) — never fabricate false for an undecided row.
- A
hold conclusion has no realized counterpart in pr_outcome (which only records merged/closed) and must always resolve to unresolved (correct unset), matching buildCalibrationReport's own existing treatment of hold predictions (tallied but never scored right/wrong).
runMetrics (lib/metrics-cli.ts) must read the pr_outcome event stream the same way calibration-cli.ts's bare calibration command does (open the event ledger via initEventLedger/resolveEventLedgerDbPath, read once, close in a finally) so metrics stays a strictly read-only, offline command with no behavior change to its existing --json-less, zero-positional-argument usage contract.
- Update the now-stale header comment in
metrics-cli.ts (the "the miner has no outcome-join yet" line) to reflect that the join now exists and is wired.
Deliverables
Test Coverage Requirements
Only src/** is measured by Codecov's codecov/patch gate (99% target, branch-counted, 0% threshold); packages/loopover-miner/** is not gated by Codecov, but npm run test:ci runs the full test/unit/**/test/integration/** suite and must stay green. Cover every branch of the new join logic directly in test/unit/miner-metrics-cli.test.ts — both the matched and mismatched outcome, the hold no-op path, the unresolved/pending path, and a malformed or missing pr_outcome event — mirroring the branch coverage calibration-cli.ts's own join tests already apply to toOutcomeRecords/buildCalibrationReport.
Expected Outcome
loopover-miner metrics's loopover_miner_prediction_correct_total/loopover_miner_prediction_incorrect_total Prometheus counters move as real predictions resolve against realized PR outcomes, instead of being permanently stuck at zero for every self-hosted miner.
Links & Resources
Context
packages/loopover-miner/lib/metrics-cli.ts'scollectPredictionMetricRowsmaps prediction-ledger rows onto the engine'sMinerPredictionMetricRowshape but only ever setsconclusion, leaving the optionalcorrectfield unset on every row:The file's own header comment justifies this as deliberate: "the realized-outcome pairing (
correct) is intentionally left unset: the miner has no outcome-join yet, so the correct/incorrect counters stay zero." That comment is now stale.packages/loopover-miner/lib/calibration-cli.ts(loopover-miner calibration, #4849) already implements exactly this join:toPredictionRecordsprojectsPredictionLedgerEntryrows (keyed bytargetId, which callers populate with the PR number) andtoOutcomeRecordsreduces thepr_outcomeevent stream to the latest outcome per(repoFullName, prNumber);buildCalibrationReport(lib/calibration.ts) joins the two on(project, targetId)and classifies each decided prediction as confirmed or false per merge/close.packages/loopover-engine/src/miner-prediction-metrics.ts'sMinerPredictionMetricRowtype was explicitly designed for this:correct?: boolean | null("true = matched, false = differed, null/undefined = not yet resolved"), andrenderMinerPredictionMetricsalready emits realloopover_miner_prediction_correct_total/loopover_miner_prediction_incorrect_totalcounters whenever any row carries a resolvedcorrectvalue. The renderer and the join both exist and are both tested; they are simply never connected, soloopover-miner metrics's correct/incorrect counters are permanently zero in every real deployment.Requirements
collectPredictionMetricRows(or a new helper it delegates to) must resolve, for each prediction-ledger row, whether a realizedpr_outcomeexists for the same(repoFullName, targetId)and, if so, whether the prediction'sconclusion(normalized the same waycalibration.ts'snormalizeDecisionalready normalizesmerge/close/holdvocabulary) matches the realized decision.correct: undefined(or omit the field) — never fabricatefalsefor an undecided row.holdconclusion has no realized counterpart inpr_outcome(which only recordsmerged/closed) and must always resolve to unresolved (correctunset), matchingbuildCalibrationReport's own existing treatment ofholdpredictions (tallied but never scored right/wrong).runMetrics(lib/metrics-cli.ts) must read thepr_outcomeevent stream the same waycalibration-cli.ts's barecalibrationcommand does (open the event ledger viainitEventLedger/resolveEventLedgerDbPath, read once, close in afinally) sometricsstays a strictly read-only, offline command with no behavior change to its existing--json-less, zero-positional-argument usage contract.metrics-cli.ts(the "the miner has no outcome-join yet" line) to reflect that the join now exists and is wired.Deliverables
collectPredictionMetricRows(or a new sibling helper) populatescorrect: true | false | undefinedusing the existingcalibration-cli.tsjoin primitives.runMetricsopens the event ledger, builds the outcome join, and passes resolved rows torenderMinerPredictionMetrics.test/unit/miner-metrics-cli.test.tscover: a merge prediction with a confirmingmergedoutcome (correct: true), a merge prediction with aclosedoutcome (correct: false), acloseprediction confirmed and disconfirmed the same way, aholdprediction (never resolved), and a prediction with no outcome at all yet (unresolved, not counted in either bucket).Test Coverage Requirements
Only
src/**is measured by Codecov'scodecov/patchgate (99% target, branch-counted, 0% threshold);packages/loopover-miner/**is not gated by Codecov, butnpm run test:ciruns the fulltest/unit/**/test/integration/**suite and must stay green. Cover every branch of the new join logic directly intest/unit/miner-metrics-cli.test.ts— both the matched and mismatched outcome, theholdno-op path, the unresolved/pending path, and a malformed or missingpr_outcomeevent — mirroring the branch coveragecalibration-cli.ts's own join tests already apply totoOutcomeRecords/buildCalibrationReport.Expected Outcome
loopover-miner metrics'sloopover_miner_prediction_correct_total/loopover_miner_prediction_incorrect_totalPrometheus counters move as real predictions resolve against realized PR outcomes, instead of being permanently stuck at zero for every self-hosted miner.Links & Resources
packages/loopover-miner/lib/metrics-cli.ts(the file to change)packages/loopover-miner/lib/calibration-cli.ts(toPredictionRecords,toOutcomeRecords— the pattern to reuse)packages/loopover-miner/lib/calibration.ts(buildCalibrationReport,normalizeDecision)packages/loopover-engine/src/miner-prediction-metrics.ts(MinerPredictionMetricRow,renderMinerPredictionMetrics)