Skip to content

calibration: compute REGRESSED-verdict track-record statistics from ORB's persisted backtest results #8140

Description

@JSONbored

Context

#8105 (the Phase-2 merge-gating decision) explicitly requires real production data: how often a REGRESSED verdict fired, whether it was ever wrong, how often it was right. #8138 persists each backtest run's BacktestComparison result durably (via recordAuditEvent, from ORB's live review pipeline — not a CI job, see #8142), but nothing yet aggregates those individual records into the summary a maintainer actually needs to make that call.

Requirements

  • Add a pure function (e.g. computeRegressedVerdictTrackRecord) that takes an array of historical BacktestComparison results (calibration: Pareto-floor comparator between two BacktestScoreReports #8086) and returns a summary: total runs, count/rate of "regressed" verdicts, broken down per ruleId (a Map<string, { total, regressed, improved, unchanged }> or equivalent).
  • Add a thin CLI wrapper that reads the persisted audit_events rows Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138 wrote (via the same event type it uses, THRESHOLD_BACKTEST_EVENT_TYPE in src/services/threshold-backtest-run.ts) and calls the pure function, printing the summary.
  • The pure aggregation function must be fully testable against synthetic fixture data now — it does not need to wait for real production data to exist, since the aggregation logic itself doesn't depend on how much real data has accumulated. Cover: zero runs, all-clean runs, some-regressed runs, and the per-ruleId breakdown with more than one ruleId present.

Deliverables

  • The pure aggregation function + its type, in packages/loopover-engine/src/calibration/ alongside the other calibration primitives.
  • Tests covering the cases above.
  • A thin CLI wrapper reading real persisted data and printing the summary (no dedicated test required for the wrapper itself, provided it contains no logic beyond the read + the function call + printing — mirrors this epic's established pure-core/thin-IO exemption).

Test Coverage Requirements

99%+ patch coverage (branch-counted) on the pure aggregation function.

Expected Outcome

Once #8138 has been running for a while, a maintainer can run one command and get the real track-record summary #8105 needs to actually decide from, instead of manually reading through individual PR comments one at a time.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions