Skip to content

calibration: pure config-drift evaluator — does any alternative setting Pareto-dominate the live value on the trailing corpus? #8212

Description

@JSONbored

Problem

Epic #8211 track A. The single largest wrongness source identified by the #8170 retro analysis is STALE configuration: a threshold that was right when set, wrong after the repo's reality moved. The loosening evaluator (evaluateKnobLoosening, src/services/loosening-knobs.ts) already answers 'can we safely loosen?' — nothing answers the inverse operator question: 'is what is CURRENTLY live still the best-supported setting, in either direction?'

Requirements

⚠️ Required pattern. PURE module beside the registry, mirroring evaluateKnobLoosening's exact discipline: deterministic, no IO, same split primitive (splitBacktestCorpus with the knob's own seed/fraction), same Pareto floor (compareBacktestScores), same never-on-noise sample minimums from the registry entry. Anchor on evaluateKnobLoosening and the #8086 comparator as your ≥2 analogues.

  • evaluateKnobDrift(knob, cases, liveValue, nowMs) → a drift report or null: evaluates EVERY registry candidate plus the shipped value (not only loosenings — a TIGHTER alternative dominating live is exactly the stale-config signal), and reports the best strictly-dominating alternative with both split comparisons attached.
  • Distinguish direction in the result ("looser" | "tighter" | "shipped"), because the consumer's messaging differs: a looser winner duplicates the loosening loop's own proposal (informational); a tighter winner means live config is likely stale (actionable warning).
  • Null (never a guess) when the corpus misses the knob's sample minimums or nothing strictly dominates on the visible split while non-regressing held-out.
  • Report shape carries corpus sizes + both comparisons per the Epic: validated auto-loosening for auto-tune.ts's tuning advisor, gated on a clean backtest #8121 evidence-trail convention. Deterministic: same corpus + value ⇒ same report.

Deliverables

  • The pure evaluator + report type, exported beside LOOSENABLE_KNOBS
  • 100% line+branch unit coverage: dominance in both directions, no-dominance, sample-floor nulls, held-out regression rejection, shipped-value-is-best

Test Coverage Requirements

Mirror test/unit/loosening-knobs.test.ts's membership-probe seeding technique. Both sides of every ternary/nullish arm.

Links & Resources

#8211 (epic), src/services/loosening-knobs.ts, packages/loopover-engine/src/calibration/backtest-compare.ts

Boundaries

Pure evaluation only — no cron, no alert, no writes (the wire is a separate maintainer issue). No new dependencies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions