2.6.1: the facts 2.6.0 added reach a results store

The facts 2.6.0 added reach a results store.

Shipped as a patch. It changes what published rows contain, so the 1.0.0
policy would call it a minor; it goes out as 2.6.1 as a deliberate exception,
because it completes 2.6.0 rather than adding to it. Same call 2.4.2 made, for
the same reason.

Added

• score_rows exports the direction and range 2.6.0 introduced. That release
  gave every metric both and then kept them in memory: the row shape this
  module flattens a run into carried neither, so nothing reached a results
  store and no consumer downstream could use them. Five columns on every row
  shape, including the failed-invocation row that names no metric at all.
  • value_range (none / bounded / unbounded_above /
    unbounded_below / unbounded), value_minimum and value_maximum.
    The discriminator is what makes "declared unbounded" and "never declared"
    different answers, which two nullable floats cannot express; it is the
    idiom metric_kind = 'none' already uses beside a filled-in value. All
    five members are spelled out because MetricRange can express all five -
    range: {max: 5} on a numeric field produces unbounded_below - and a
    consumer storing this as an enum pays a migration to add a member later.
  • metric_direction and higher_is_better. metric_direction carries the
    same four values gate_win does, none among them, meaning never
    computed: an ungated run has no baseline, and neutral would report
    "measured, and it did not move" about a number nothing was measured
    against. higher_is_better is 'true'|'false'|'unknown', and unknown
    is narrower than it looks - compare() defaults a metric nothing
    mentions to higher-is-better, so it reads unknown only when a suite
    declared the metric neutral or there was no comparison at all.

Changed

• _passed is now _tristate, taking the spelling of its third state as an
  argument. passed keeps 'null', which it has published since the row
  shape existed; higher_is_better uses 'unknown', which reads as "nobody
  declared one" rather than as a missing value. Private; no caller outside
  this module.

2.6.0: a metric says what its numbers mean

A metric says which way is good and what its values sit on.

Added

• MetricRange and a value_range field on Score and MetricValue.
  Nothing about a float says what it is on: 0.86 is 86% if the metric runs
  0..1 and 4.3 out of 5 if it runs 1..5. A consumer holding only the number
  guesses from magnitude, which is how a run whose costs happened to stay
  under a dollar gets rendered as percentages, and how the same metric gets
  classified differently depending on the window you look at.
• Built-in graders declare their own ranges. Deterministic checks are 0..1;
  classification separates its confusion-matrix metrics (0..1) from
  support_* and errors (0..unbounded); judge metrics are 0..1, being
  points over the scale, except disagreement, a spread in raw points,
  which is 0..scale-1.
• range: on a numeric field spec, as {min, max} or a two-element list.
  That grader surfaces whatever the adapter put in the field, so it is the
  one place the engine cannot know. It is a separate key from the min/max
  bounds beside it, which are a pass/fail threshold rather than a domain: a
  cost that must stay under a dollar can still cost five.
• maximum=None is a positive statement - unbounded above, so not a
  fraction of anything - and is different from carrying no range at all,
  which says only that nobody declared one. Nothing is inferred from
  observed values.
• direction and higher_is_better on MetricDelta. A delta's sign and its
  meaning are different questions - f1 rising is an improvement, and
  false_negative_rate rising is a regression - and until now the engine only
  knew the difference for the single win metric, via win_higher_is_better.
  Every other metric came out of compare() as a bare number, so anything
  downstream that wanted to rank, colour or summarize deltas had to keep its
  own list of which metrics are inverted, or get it wrong.
• thresholds.metrics, an optional map declaring polarity per metric:
  lower_is_better, higher_is_better, or neutral for one that moves
  without either direction being a result.
• Polarity is inferred from the guardrails when it is not declared, which
  is what makes this useful without editing a single existing suite. A max
  or must_not_increase rule is only ever written about a metric you want
  low, and min/must_not_decrease about one you want high, so a gated
  metric has already stated its direction. Precedence, strongest first: an
  explicit metrics: entry, then win_higher_is_better for the win metric,
  then the guardrails. A metric none of them mention defaults to
  higher-is-better; a metric fenced in on both sides by a band resolves to
  neutral rather than falling through to that default, since a band says
  neither direction is the good one.

Changed

• _evaluate_win derives the win verdict through the same call that fills in
  each MetricDelta.direction, so Comparison.win and the win metric's own
  delta row cannot disagree about the same number.

2.5.0

What's Changed

A suite can declare the modules it needs imported.

Added

• plugins: on a suite: a list of module paths the runner imports before it looks up any type, so a suite that names a custom adapter or grader resolves it without a flag at the call site. Registration is an import side effect and nothing in the engine imports a consumer's module on its own, so until now every entry point had to remember --plugins my.graders (CLI) or a bare import my.graders (Python API) - and the two could disagree. A suite is now self-contained: the same file runs from the CLI, from a consumer's own run_eval.py, and from a test with nothing to remember.

  --plugins is unchanged and still the way to add a module without editing the suite, which would change suite_hash.

  The import happens when a run starts, never in load_suite. Parsing, hashing, diffing or reporting on a suite executes no consumer code, so a suite you have not decided to run is still only data. compare and report therefore do not import a suite's plugins - they do not need the registries.

  A module that cannot be imported raises ConfigError naming it, rather than the unknown-type error one lookup later.

Fixed

• examples/quickstart/run_eval.py called JsonlOutboxExporter.export(), removed when 2.2.0 named the exporter seam, so just example-api had been failing with AttributeError since. It now exports score rows only, which is the one grain the store has: a scorecard is a read-time aggregation over those rows, so exporting it too would persist something derived that could disagree with them.

Changed

• examples/quickstart declares its own plugins: and no longer needs --plugins on the command line, nor the import ... # noqa: F401 that three of its entry points carried to force registration. graders.py no longer imports adapter.py for the side effect either. The example is the same eval; it just stops demonstrating the workaround.

Upgrading: nothing breaks. Existing suites have no plugins: key and keep working exactly as before, with --plugins or an explicit import. Adding the key to a suite changes its suite_hash, which is correct - it changes what code runs - so expect one re-baseline on the suites you convert.

2.4.3

What's Changed

Live Anthropic judges work on current Claude models again.

Fixed

• The Anthropic judge and pairwise clients no longer send temperature=0. temperature (with top_p/top_k) was removed from the Claude request surface at Opus 4.7, and sending it at all is a 400 there and on every model after it - so a judge or a pairwise comparison pinned to claude-opus-4-7, claude-opus-4-8, claude-opus-5, claude-sonnet-5 or claude-fable-5 failed every call. The OpenAI clients still send it; that API still takes it.

Changed

• Live judge max_tokens defaults are now 8192 (from 1024 on the rubric judge, 512 on pairwise). Thinking is on by default from Opus 5 and Sonnet 5 onward and max_tokens bounds thinking plus reply together, so a 1024-token budget could be spent on reasoning before the forced tool call landed - which surfaced as error scores rather than as an error. It is a ceiling, not a spend: a model that does not think generates the same handful of tokens it did before.

Upgrading: a judge on a model that still accepts temperature (Sonnet 4.6, Opus 4.6, the 4.5 line and older) now samples at the API default instead of 0, so its scores are no longer pinned run to run - expect more variance in a rubric dimension or a win-rate than before, and re-baseline if a gate sits close to its threshold. A judge on a thinking model also now bills thinking tokens on every call. Pass max_tokens= to a client to keep the old budget.

2.4.2

What's Changed

Every llm_as_judge row now describes what it measures.

Shipped as a patch. It changes what published rows contain, so the 1.0.0 policy would call it a minor; it goes out as 2.4.2 as a deliberate exception, alongside 2.4.1's judge-pin fix that it completes. No schema change — the columns already exist and were empty.

Fixed

• A judge's dimension row (<grader>.<dimension>) carried no judge information at all: no judges.*, and judge_scale of 0. So 0.400 on a row could not say who scored it, on what scale, or that the raw number was 2 out of 5 — the scale lived only on <grader>.overall, and recovering the raw point meant joining back to it.

  Each dimension row now carries the verdict for the dimension it measures, with judges.points scoped to that dimension:

  metric                    value   scale  judges.score  judges.points
  quality.accuracy          0.8     5      [0.8]         [{accuracy: 4}]
  quality.empathy           1.0     5      [1.0]         [{empathy: 5}]
  quality.overall           0.9     5      [0.9]         [{empathy: 5, accuracy: 4}]
  

  Carrying the full map on a dimension row would put irrelevant numbers on it and invite reading the wrong one. <grader>.overall is unchanged.

  This establishes one invariant across every judge row, dimension rows included:

  value == mean(judges.score)
  

  With one judge they are equal; with a panel, value is the mean and judges.score shows the spread — so per-dimension disagreement is readable from the dimension row instead of by unpacking the map on .overall.

Upgrading: notEmpty(judges.name) now matches every judge row rather than only <grader>.overall, so a six-case run goes from 12 judged rows to 48. Anything aggregating over judges must filter to .overall or it multiplies by the dimension count. That is a change in results, not an error.

2.4.1

What's Changed

The judge's identity is now in its provenance pin.

Shipped as a patch. Both changes below alter a published value, so the 1.0.0 policy would call this a minor at least; it goes out as 2.4.1 as a deliberate exception, because the old pin was answering a provenance question wrongly and the sooner it stops the fewer runs are affected.

Fixed

• Scorecard.judge_version now includes the judge's model: anthropic:claude-sonnet-4-6@v1 rather than judge@v1.

  It was key@judge_version. Swapping the judge's model while leaving the declared judge_version alone produced a byte-identical pin, so any provenance check reading that field passed a comparison against a baseline scored by a different model:

  before the swap: judge@v1
  after  the swap: judge@v1     ← same pin, different judge
  

  Now:

  before the swap: anthropic:claude-sonnet-4-6@v1
  after  the swap: anthropic:claude-opus-4-1@v1
  

  The model is the thing most likely to change and the thing a declared version is most likely to miss.

Changed

• A single judge's key defaults to its provider (anthropic) instead of the literal judge. A panel already defaulted to key or provider, so the one-judge case was the odd one out, and judge named nothing the grader column did not already say. It appears in the judges.name column.

Upgrading: every suite with a judge re-baselines once, because judge_version is part of what identifies a comparable run — the intended behaviour, just paid all at once. Queries filtering judges.name = 'judge' need the provider instead. Set key: on a judges: entry to pin a name of your own.

2.4.0

What's Changed

A run without a baseline can now say whether it passed.

Added

• RunResult.checks, a models.ThresholdCheck the runner fills in: the suite's absolute guardrails (min/max) measured against this run. Most of a suite's rules are absolute and answerable from one run; only the win metric and the relative rules need a baseline.

  There is nothing new to call. compare is for comparing, and a single run was never a comparison — so the runner does it, the same way it already records graders.

  verdict: pass
    [ok] errors - 0.0000 ok
    [ok] length_ok - 1.0000 ok
    [skipped] cta_mentions_offer - must_not_decrease not evaluated: needs a baseline
  

• compare.check_thresholds(scorecard, thresholds), which the runner uses.

Changed

• Breaking: GuardrailResult.passed is now bool | None. None means the rule could not be evaluated — a must_not_increase / must_not_decrease with no baseline. Test passed is False for a breach; not passed now also catches the skipped case. The Markdown and HTML reporters render it as skipped rather than a breach.

• store.score_rows reports run.checks on the gate columns when no Comparison is passed: gate_verdict and the per-metric guardrail become real, while gate_win stays 'none'.

  That split is the point. gate_win is what marks the three win_* fields as never computed rather than measured at zero, so an ungated run reports a real verdict without claiming a comparison happened. A rule that could not run reports guardrail = 'none' with the reason in guardrail_gap, never a false pass.

• A Comparison supersedes the run's own checks: same rules, plus a baseline, so it gets the relative ones too.

Upgrading: if you read GuardrailResult.passed, change not g.passed to g.passed is False.

2.3.0

What's Changed

A run now describes its own scores.

Added

• RunResult.graders, a map of grader name to models.GraderInfo (category, scale), filled in by the runner from the graders it built.

  A Score names the grader that emitted it and nothing else, so a run could not previously answer for itself. A run.json written by one CI step and published by a later one needed the suite file alongside it; without one, every grader reported unknown and a suite with a judge could not be published at all. The artifact looked complete and wasn't.

  Run-grain rather than score-grain: one entry per grader, not two fields repeated across a couple of hundred rows.

• models.GraderInfo.

• register also records the category on the decorated class as grader_category, so a grader instance can answer for itself. The runner reads it the same way it already collects judge_version.

Changed

• store.score_rows takes the grader category and judge scale from run.graders:

  exporter.export_scores(run, comparison)   # no lookups needed

  grader_types and judge_scales still override it, and are how a run written before this release - whose map is empty - publishes correctly. grader_lookups is unchanged.

Upgrading: nothing to do. Passing the lookups still works and still wins.

2.2.0

What's Changed

Added

• store.ScoreExporter, a runtime_checkable Protocol naming the seam JsonlOutboxExporter already occupied. store.py has always said to "replace this class with a database client implementing the same export_scores method", but the contract was a docstring sentence and **kwargs, so an implementation had to duck-type a private shape.

  An exporter belongs in the package that owns the store it targets - it is the store that knows its own column types, null policy, and transport. Swapping one for another is a constructor line at the call site, so an offline run and a live one share a code path:

  exporter = store.JsonlOutboxExporter(path)   # offline
  exporter = KafkaOutboxExporter(brokers)      # live, from another package
  exporter.export_scores(run, comparison, grader_types=..., judge_scales=...)

Fixed

• examples/quickstart/graders.py still used the one-argument register and raised TypeError on import, so 2.1.0 shipped with its own bundled example broken. just test does not run the example; just test-all does.
• The README's Python API example called a .export() that does not exist and passed RunResults to compare.compare and render_scorecard, which take Scorecards. It now runs verbatim, and shows grader_lookups feeding the exporter.

Upgrading: nothing to do. Both additions are backward compatible.

2.1.0

What's Changed

A grader declares what kind of check it is at registration, so a consumer plug-in is categorised the same way a built-in is. Previously any grader outside the seven built-in types reported grader_type = 'unknown' on every row it produced.

Shipped as a minor despite the signature change below. register is public, so the 1.0.0 policy would call this a major; it goes out as 2.1.0 as a deliberate exception, because the break is a one-line edit per grader that fails loudly at import.

Added

• graders.GraderType, a StrEnum over the closed set the results store's grader_type column accepts: unknown, heuristic, statistical, llm_as_judge, trajectory, human. A StrEnum so it needs no serializer of its own on the way to a row.
• graders.category_of, the registry lookup behind grader_lookups.

Changed

• Breaking: graders.base.register takes a required second argument, category, a graders.GraderType. Every @base.register('foo') becomes @base.register('foo', base.GraderType.HEURISTIC) or whichever member applies; omitting it is a TypeError at import. Required rather than defaulted on purpose - it is the only source of a row's grader_type, and a default would be the value every grader forgets to override.
• store.grader_lookups reads the category from the registry instead of a closed table of built-in type names, so a plug-in that declares HEURISTIC reports heuristic where it used to report unknown. Rows for consumer graders change value in the grader_type column; nothing about the row shape changes.

Removed

• store._GRADER_TYPES, the private table the categories used to live in. Keeping it alongside the registration argument would mean two sources for one fact and a precedence rule between them.

Upgrading

Add the category to every @base.register(...) in your plug-ins:

from evalcore.graders import base

@base.register('my_grader', base.GraderType.HEURISTIC)
class MyGrader:
    ...

Pick the member that describes the check, not the grader's shape - an AggregateGrader computing a rate is still HEURISTIC, not STATISTICAL.