Should the classifier auto-tune its thresholds from your production traces?

[fitz2882]

Right now the trajectory thresholds are derived analytically — control theory plus standard statistical convention — not fitted to any dataset. That was a deliberate choice: thresholds fitted to a benchmark are thresholds overfitted to a benchmark, and I'd rather ship something whose behaviour you can reason about than something tuned to workloads that aren't yours. You can already override them per-domain via TrajectoryThresholds once you have traces.

The open question is whether LoopGain should do that fitting for you.

The case for: every workload has a different noise floor. A code-gen loop with a deterministic test count and a judge-scored prose loop have wildly different osc_std characteristics, and one global oscillation threshold is a compromise that's wrong for both. If you've got a few hundred of your own trajectories sitting in the dashboard, the data to calibrate is right there.

The case against: adaptive thresholds make behaviour non-reproducible. Same loop, same inputs, different stop decision depending on what the monitor has seen recently — that's a miserable thing to debug in production, and it undermines the main reason to prefer this over a heuristic.

A middle path I've been looking at: keep the decision rule fixed and static, but ship an offline calibration tool. Point it at your trace history, it reports what your thresholds should be and why, you review the diff and commit it. Explicit, versioned, reproducible — the tuning is a code change rather than runtime drift.

Which of those three do you want? And if you're already overriding TrajectoryThresholds, what did you change and what made you change it — that's the data I most want.