Skip to content

fix(miner): metrics-cli's collectPredictionMetricRows never resolves correct/incorrect despite calibration-cli.ts already having the outcome join #8315

Description

@JSONbored

Context

packages/loopover-miner/lib/metrics-cli.ts's collectPredictionMetricRows maps prediction-ledger rows onto the engine's MinerPredictionMetricRow shape but only ever sets conclusion, leaving the optional correct field unset on every row:

export function collectPredictionMetricRows(ledger: PredictionLedger): MinerPredictionMetricRow[] {
  return ledger.readPredictions().map((entry) => ({ conclusion: entry.conclusion }));
}

The file's own header comment justifies this as deliberate: "the realized-outcome pairing (correct) is intentionally left unset: the miner has no outcome-join yet, so the correct/incorrect counters stay zero." That comment is now stale. packages/loopover-miner/lib/calibration-cli.ts (loopover-miner calibration, #4849) already implements exactly this join: toPredictionRecords projects PredictionLedgerEntry rows (keyed by targetId, which callers populate with the PR number) and toOutcomeRecords reduces the pr_outcome event stream to the latest outcome per (repoFullName, prNumber); buildCalibrationReport (lib/calibration.ts) joins the two on (project, targetId) and classifies each decided prediction as confirmed or false per merge/close.

packages/loopover-engine/src/miner-prediction-metrics.ts's MinerPredictionMetricRow type was explicitly designed for this: correct?: boolean | null ("true = matched, false = differed, null/undefined = not yet resolved"), and renderMinerPredictionMetrics already emits real loopover_miner_prediction_correct_total/loopover_miner_prediction_incorrect_total counters whenever any row carries a resolved correct value. The renderer and the join both exist and are both tested; they are simply never connected, so loopover-miner metrics's correct/incorrect counters are permanently zero in every real deployment.

Requirements

⚠️ Required pattern. Reuse calibration-cli.ts's existing join — toPredictionRecords, toOutcomeRecords, and buildCalibrationReport (or the same (project, targetId) matching logic) — to resolve each prediction row's realized outcome. Do not invent a second join implementation or change buildCalibrationReport's public contract.

  • collectPredictionMetricRows (or a new helper it delegates to) must resolve, for each prediction-ledger row, whether a realized pr_outcome exists for the same (repoFullName, targetId) and, if so, whether the prediction's conclusion (normalized the same way calibration.ts's normalizeDecision already normalizes merge/close/hold vocabulary) matches the realized decision.
  • A prediction with no realized outcome yet (still pending) must set correct: undefined (or omit the field) — never fabricate false for an undecided row.
  • A hold conclusion has no realized counterpart in pr_outcome (which only records merged/closed) and must always resolve to unresolved (correct unset), matching buildCalibrationReport's own existing treatment of hold predictions (tallied but never scored right/wrong).
  • runMetrics (lib/metrics-cli.ts) must read the pr_outcome event stream the same way calibration-cli.ts's bare calibration command does (open the event ledger via initEventLedger/resolveEventLedgerDbPath, read once, close in a finally) so metrics stays a strictly read-only, offline command with no behavior change to its existing --json-less, zero-positional-argument usage contract.
  • Update the now-stale header comment in metrics-cli.ts (the "the miner has no outcome-join yet" line) to reflect that the join now exists and is wired.

Deliverables

  • collectPredictionMetricRows (or a new sibling helper) populates correct: true | false | undefined using the existing calibration-cli.ts join primitives.
  • runMetrics opens the event ledger, builds the outcome join, and passes resolved rows to renderMinerPredictionMetrics.
  • The stale "no outcome-join yet" comment is corrected.
  • Unit tests in test/unit/miner-metrics-cli.test.ts cover: a merge prediction with a confirming merged outcome (correct: true), a merge prediction with a closed outcome (correct: false), a close prediction confirmed and disconfirmed the same way, a hold prediction (never resolved), and a prediction with no outcome at all yet (unresolved, not counted in either bucket).

Test Coverage Requirements

Only src/** is measured by Codecov's codecov/patch gate (99% target, branch-counted, 0% threshold); packages/loopover-miner/** is not gated by Codecov, but npm run test:ci runs the full test/unit/**/test/integration/** suite and must stay green. Cover every branch of the new join logic directly in test/unit/miner-metrics-cli.test.ts — both the matched and mismatched outcome, the hold no-op path, the unresolved/pending path, and a malformed or missing pr_outcome event — mirroring the branch coverage calibration-cli.ts's own join tests already apply to toOutcomeRecords/buildCalibrationReport.

Expected Outcome

loopover-miner metrics's loopover_miner_prediction_correct_total/loopover_miner_prediction_incorrect_total Prometheus counters move as real predictions resolve against realized PR outcomes, instead of being permanently stuck at zero for every self-hosted miner.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions