Repository navigation
2.6.0: a metric says what its numbers mean
A metric says which way is good and what its values sit on.
Added
MetricRangeand avalue_rangefield onScoreandMetricValue.
Nothing about a float says what it is on:0.86is 86% if the metric runs
0..1 and 4.3 out of 5 if it runs 1..5. A consumer holding only the number
guesses from magnitude, which is how a run whose costs happened to stay
under a dollar gets rendered as percentages, and how the same metric gets
classified differently depending on the window you look at.- Built-in graders declare their own ranges. Deterministic checks are 0..1;
classificationseparates its confusion-matrix metrics (0..1) from
support_*anderrors(0..unbounded); judge metrics are 0..1, being
points over the scale, exceptdisagreement, a spread in raw points,
which is 0..scale-1. range:on anumericfield spec, as{min, max}or a two-element list.
That grader surfaces whatever the adapter put in the field, so it is the
one place the engine cannot know. It is a separate key from themin/max
bounds beside it, which are a pass/fail threshold rather than a domain: a
cost that must stay under a dollar can still cost five.maximum=Noneis a positive statement - unbounded above, so not a
fraction of anything - and is different from carrying no range at all,
which says only that nobody declared one. Nothing is inferred from
observed values.directionandhigher_is_betteronMetricDelta. A delta's sign and its
meaning are different questions -f1rising is an improvement, and
false_negative_raterising is a regression - and until now the engine only
knew the difference for the single win metric, viawin_higher_is_better.
Every other metric came out ofcompare()as a bare number, so anything
downstream that wanted to rank, colour or summarize deltas had to keep its
own list of which metrics are inverted, or get it wrong.thresholds.metrics, an optional map declaring polarity per metric:
lower_is_better,higher_is_better, orneutralfor one that moves
without either direction being a result.- Polarity is inferred from the guardrails when it is not declared, which
is what makes this useful without editing a single existing suite. Amax
ormust_not_increaserule is only ever written about a metric you want
low, andmin/must_not_decreaseabout one you want high, so a gated
metric has already stated its direction. Precedence, strongest first: an
explicitmetrics:entry, thenwin_higher_is_betterfor the win metric,
then the guardrails. A metric none of them mention defaults to
higher-is-better; a metric fenced in on both sides by a band resolves to
neutralrather than falling through to that default, since a band says
neither direction is the good one.
Changed
_evaluate_winderives the win verdict through the same call that fills in
eachMetricDelta.direction, soComparison.winand the win metric's own
delta row cannot disagree about the same number.