You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
BenchRunCfg.regression_overrides — per-variable regression specs. Maps result-variable name → a {method: threshold} dict drawn from percentage / adaptive / delta / absolute, or a bare number as shorthand for {'absolute': value} (a hard directional limit: minimize = ceiling, maximize = floor).
A listed variable is checked by exactly the methods in its spec instead of the benchmark-wide regression_method, so a threshold can be loosened or tightened per variable; an explicit {} opts a variable out of detection. Unlisted variables keep the benchmark-wide method, and names matching no result variable are skipped, so one override map can be shared across benchmarks.
Multiple entries run as independent checks — {'percentage': 15.0, 'absolute': 1.0} tracks the trend and holds a hard floor. absolute checks need no history and fire from the very first recording; the other methods skip until history exists.
Adaptive overrides: the threshold is the MAD limit (the dual-band percent gate still comes from regression_percentage); while history is too sparse for MAD the check skips instead of falling back to a percentage check, so a listed variable is never judged by a knob outside its spec.
Malformed specs never crash a run: unknown method keys and non-finite/bool thresholds are dropped with a warning, and a spec left with no valid checks keeps the benchmark-wide method (a typo can't silently disable detection). Bare-number shorthand accepts numpy scalars.
Motivating use case: CI tracking of flat health metrics (a success rate that must hold 1.0, an orphan-process count that must hold 0) next to per-metric trend thresholds on the same benchmark. Breaches flow through the existing machinery unchanged (has_regressions, regression_fail/RegressionError, report markdown, JSON, render_png).
First mutable (param.Dict) field on BenchRunCfg: every BenchRunner copy point deepcopies and the default is None; recorded in the copy-guard test's REVIEWED_MUTABLE_FIELDS allowlist with nested-dict isolation tests.
Changed
detect_absolute with OptDir.none now warns and returns None instead of recording a non-regressed "checked" result — a guard with no direction never ran, so it no longer appears to have passed.