Skip to content

2.6.0: a metric says what its numbers mean

Choose a tag to compare

@scottpmiller scottpmiller released this 13 Sep 00:53
· 3 commits to main since this release
2.6.0
9c7d8c2

A metric says which way is good and what its values sit on.

Added

  • MetricRange and a value_range field on Score and MetricValue.
    Nothing about a float says what it is on: 0.86 is 86% if the metric runs
    0..1 and 4.3 out of 5 if it runs 1..5. A consumer holding only the number
    guesses from magnitude, which is how a run whose costs happened to stay
    under a dollar gets rendered as percentages, and how the same metric gets
    classified differently depending on the window you look at.
  • Built-in graders declare their own ranges. Deterministic checks are 0..1;
    classification separates its confusion-matrix metrics (0..1) from
    support_* and errors (0..unbounded); judge metrics are 0..1, being
    points over the scale, except disagreement, a spread in raw points,
    which is 0..scale-1.
  • range: on a numeric field spec, as {min, max} or a two-element list.
    That grader surfaces whatever the adapter put in the field, so it is the
    one place the engine cannot know. It is a separate key from the min/max
    bounds beside it, which are a pass/fail threshold rather than a domain: a
    cost that must stay under a dollar can still cost five.
  • maximum=None is a positive statement - unbounded above, so not a
    fraction of anything - and is different from carrying no range at all,
    which says only that nobody declared one. Nothing is inferred from
    observed values.
  • direction and higher_is_better on MetricDelta. A delta's sign and its
    meaning are different questions - f1 rising is an improvement, and
    false_negative_rate rising is a regression - and until now the engine only
    knew the difference for the single win metric, via win_higher_is_better.
    Every other metric came out of compare() as a bare number, so anything
    downstream that wanted to rank, colour or summarize deltas had to keep its
    own list of which metrics are inverted, or get it wrong.
  • thresholds.metrics, an optional map declaring polarity per metric:
    lower_is_better, higher_is_better, or neutral for one that moves
    without either direction being a result.
  • Polarity is inferred from the guardrails when it is not declared, which
    is what makes this useful without editing a single existing suite. A max
    or must_not_increase rule is only ever written about a metric you want
    low, and min/must_not_decrease about one you want high, so a gated
    metric has already stated its direction. Precedence, strongest first: an
    explicit metrics: entry, then win_higher_is_better for the win metric,
    then the guardrails. A metric none of them mention defaults to
    higher-is-better; a metric fenced in on both sides by a band resolves to
    neutral rather than falling through to that default, since a band says
    neither direction is the good one.

Changed

  • _evaluate_win derives the win verdict through the same call that fills in
    each MetricDelta.direction, so Comparison.win and the win metric's own
    delta row cannot disagree about the same number.