evaluation-harness v0.4.0

See your runs, not only tables of them. evh catalog builds static pages over a project's
runs, and evh catalog-serve runs an inspector that tests chosen arms against a baseline and
shows one item's outputs beside its reference. A project on the harness's layout needs no code;
one with its own layout writes one adapter. Goals record what an evaluation is for; items carry
the typed fields a project declares, and can live at a pinned URL. The paired test now lives in
one place, evaluation_harness.stats, exact at 30 items or fewer.
Upgrading can move a result only where pair-test or family-test ran on 30 items or fewer,
in a wins count or an order on a lower-is-better metric, or where --exclude-dataset named a
dataset whose items are not .txt files under data/sources/<id>/ (all below), and adds no
dependency. Design:
ADR-0002,
ADR-0003;
how to upgrade: UPGRADING.

This repository now deploys the harness on its own example experiments:

• the catalog and run inspector are public at
  catalog.evals.humancopilot.eu;
• the public site, the harness and its findings, moves from GitHub Pages to the same server;
• the docs move to docs.evals.humancopilot.eu, and old
  /docs/ links redirect there.

The deployment files are in infra/ (runbook: infra/README.md). They are not part of the
package.

The example experiments keep one record per arm, the latest: the rescored and re-parsed runs
are folded into data/runs as derived runs, and DailyMail's bart_s is its fp32 run
(coverage 0.4048; the fp16 run read 0.4039). CNN's bart_s stays the fp16 run, the one the
judge's verdicts were made on. The reports state the current figures only; their correction
logs are gone (git keeps them). Two published statements were stale and are restated:
SciFact's qwen_s e5-first against the e5-first leader is +0.0178, p 0.10 (it read +0.0092,
p 0.36, from the pre-fix run), and DBpedia's 56-item dev slice, over today's 28 arms, puts
bert_base_fy alone at the top rather than 13 arms in a tie.

Added

• The rest of the inspector and catalog in the terminal. Every page and view now has a
  command:

  • evh search TEXT is the catalog's search box, ranked the same way, with the command that
    shows each hit;
  • evh inspect --items is the per-item table (each arm's score and its Δ against the
    baseline);
  • evh inspect --analysis [--x METRIC] gives the analysis charts as numbers: spread per run,
    the difference from the baseline, and each arm's mean against a second metric.

• pair-test --round rN tests within a round. A defaults to the round's leader, B to the
  baseline the round started from, and the metric to the goal's measure. An arm that is not in
  play is refused.

• On the goal page:

  • each round has an inspect button that opens its arms against its leader (evh inspect --round);
  • every suggested command has a copy button;
  • a retired arm is greyed out.

• Experiment rounds, optional (ADR-0004).
  Experiments go round on round: try some arms, keep what wins, drop what lost, change something,
  compare again. evh round-open --goal G --dataset D [--from rN] records what a round sets out to
  learn, what changed, the arms carried in unchanged (--keep) and those dropped with the reason
  (--retire ... --reason); arms join by naming it in their config (round: rN, recorded on the
  run). evh rounds --goal G reads the evidence the way the inspector does — the round's arms
  against its leader, its best against the previous round's best and against the baseline it
  started from, its spend against budget_usd — and suggests the next decision, each as the
  command that makes it: evh round-retire, evh rank-stability, evh round-close --conclusion ... --promote <run> (which promotes the baseline with the conclusion as its reason). A closed
  round refuses new runs; validate (V14) checks the records; leaderboard and family-test take
  --round. The goal page shows the rounds, newest first, with the open round's suggestions. The
  catalog stays read-only: decisions are commands. A project that never opens a round sees none of it.

• A goal page leads with its answer. First the promoted baseline on each of its datasets and
  why (or "no decision yet", with the command that makes one), then what the goal is for, then on
  each dataset its references and its arms in groups by what they are scored against: gold,
  silver, a judge. Arms are ranked only within a group, and a group's leader is never an arm graded
  against its own outputs. A dataset page shows its baseline and reason, and the baseline's row is
  tagged. evh goals --goal and evh datasets --dataset-id say the same.

• Everything the catalog and the inspector show is a command. evh datasets (one dataset:
  its versions, references and arms), evh references (one: who wrote it, from which run, what
  is scored against it), evh metrics, evh goals (one: its arms per dataset) and
  evh inspect --runs … (the inspector's verdicts, conditions, --where/--by on fields,
  --item for one item's outputs beside the reference). Same adapter, same computation: the
  numbers are the pages'. Each is also a make target.

• The catalog says what every number is measured against. A metrics page (what each metric
  measures, which way is better, where it is recorded, which goals decide on it) and a page per
  reference (who wrote it, from which run, every arm scored against it); every metric and
  reference on the pages links there. Each arm's row says what it is scored against, and rows
  scored against silver, by a judge, or against the reference the arm wrote itself are tinted; a
  judged arm reads "judge ", no longer "gold". A goal page flags the arm that wrote the
  reference it is scored against, so it is never the leader. The inspector's charts section is
  "analysis", and its verdict rows carry the same tints. Actions are compact and tinted.

• Long tables page. Every catalog table and the inspector's per-item table show 10 rows, with
  "show 10 more" and "show all"; the filter and the sort work over every row, and a link to a row
  opens the table at it. The verdict tables stay whole: their Holm thresholds are read together.

• Dataset versions keep their lineage. evh dataset-create --dataset-id x_v2 --from x_v1
  records where a version came from and the item ids it adds, removes and changes
  (derived_from in the dataset file); validate V2 fails a version whose parent is gone. The
  catalog links versions both ways with what changed, and a version no arm has run on yet gets
  a page when it is linked to one that has.

• The inspector says whether the compared runs ran under the same conditions, from their
  fingerprints: the machine, library versions, the harness, the scorer (adapter code), the
  items and the reference's bytes. One line when all match; otherwise the fields that differ,
  arm by arm, with a warning when the scorer or the reference bytes differ (different
  measurements) or latency is compared across machines. A dataset page names each arm's machine
  when its arms ran on more than one.

• METRIC_DESCRIPTIONS in an adapter, one sentence per metric, is recorded on every run
  (metric_descriptions in metrics.json) and refreshed by evh rescore; the catalog reads the
  adapter's current descriptions over the recorded ones, so describing a metric later needs no
  new run. The harness describes its own metrics (cost, tokens, latency, judge scores).

• A run becomes a reference: evh reference-create --from-run <run_id> (make reference-create RUN=<run_id>). The usual way to get references is to run the expensive
  model as an arm, read its results, and score the cheaper arms against what it wrote. Until now
  that took a second paid pass (--config), whose outputs could differ from the run you read,
  and the reference did not say which run it came from. --from-run copies a finished run's
  outputs and calls nothing; the manifest records the run, its config, fingerprint, build and
  cost. It refuses what run-promote refuses (a dirty tree, an unknown build), an unfinished,
  derived or other-dataset run, and --tier gold (gold is what a person wrote or checked).
  With --not-redistributed only the manifest is committed: it records the command that rebuilds
  the texts from the run, and that command, on a checkout with the manifest alone, writes them
  again and checks them against its checksums without touching it.

• reference-create refuses an empty output, and outputs that stopped at max_tokens, by
  either path (--allow-cut-off for outputs that end at the cap by design). A frontier arm at
  max_tokens: 80 wrote 19 of 20 summaries cut off mid-sentence and one empty reply, and they
  were frozen as the reference with no word.

• The leaderboard marks the arm that wrote the reference it is scored against (arm *, and
  a note): that row is graded against its own outputs.

• reference-create says when gold exists (new runs keep scoring against gold; how to rescore
  against the new silver), and to commit the reference before the next run: an uncommitted
  reference makes every later run dirty.

• A newcomer scenario for every release
  (SCENARIO): a fresh
  project, the published docs only, a summariser and a classifier with small CPU models, cheap
  hosted LLMs, a frontier reference and a judge, browsed in the catalog, under $1. RELEASING
  runs it before the merge.

• A goal weighs cost, quality and speed, and recommends
  (ADR-0006).
  Cost (USD per item, tokens beside it) and speed (latency p50 and p95) are built in;
  success_criteria.axes adds limits, another measure, role: ignore, or the project's own
  per-item metric in one line. A strategy says who is tied with the quality leader
  (balanced, the default: not separated by the paired test and within a margin, 2% of the
  leader unless the goal sets one; quality_first; frugal) and tie_break (default cost,
  then speed) picks among them. Limits come first; flagged and retired arms are never
  recommended, nor is an arm measured on fewer items than the rest of its group; identical
  scores are always tied; a missing number is "not recorded", and an axis a tied arm does not
  record cannot break the tie; speed is not compared across machines the runs recorded. Some
  groups show their leader and never recommend: a dataset marked decides: false, a silver
  group beside a gold one, and a group measured by unvalidated judges ("Shown, not
  recommended", with the reason). The helpers are public as evaluation_harness.axes.
  The goal page shows the decision with its reasons, an axis column per arm and quality
  against each axis as charts; the home page's glance a recommended column with cost and
  speed against the baseline; the dataset and arm pages cost and speed; the inspector's scatter
  cost and tokens. evh goals --goal and evh overview print the same. Guardrails are shown
  on the goal page and in the CLI. When the recommendation is not the promoted baseline, the
  goal page and evh goals --goal say by how much and whether the test separates them; with no
  eligible arm, they count the reasons. The goals page and evh goals count configs filed and
  show each goal's recommendation. Cost and speed metrics are their own kinds, with the goals
  that weigh them, and costs print to six places; chart names no longer overlap.

• Reference manifests and V16. Every reference carries a manifest.json with the checksum of
  each item's text (schemas/reference.schema.json); evh reference-manifest writes it for gold a
  script builds, reference-create for the references it makes. It is committed even when the texts
  are not redistributed: evh validate (V16) checks a rebuilt copy against it, and the catalog lists
  such a reference as fetched. The manifest names the dataset (dataset_id), so a second gold for
  one dataset can have a name of its own (reference-manifest --dataset-id, e.g.
  gold/ag_news_200_adj1 for ag_news_200). This repository's gold references now have one.

• round-retire --runs-dir retires one copy of a config (its runs in another runs directory,
  such as a re-timing on another machine) and leaves the arm in the project's runs directory in play.

Changed

• A config with a key written into its params is refused (api_key, access_token,
  secret, ...): it would be copied into every run record and onto the arm page. Name the
  variable with api_key_env instead; the catalog shows any such param as "(hidden)".
• success_criteria.secondary is removed: its entries are axes (UPGRADING); a goal that
  still has it fails V14.
• Documentation never moves a fingerprint (fingerprint version 4). Code is hashed by what it
  does: its syntax tree without docstrings, comments, formatting or METRIC_DESCRIPTIONS
  (assigned, or imported from a shared module). That
  holds for the adapter, the harness's measuring modules, judge_score's own hash and the examples'
  code_digest. A judge scorer YAML is hashed by its parsed content, and a reference manifest's
  caveat is outside the reference digest. Adding metric descriptions to an adapter used to make
  run-compare and the inspector call every earlier run "not the same conditions". The tree is
  serialised without empty fields, so one file hashes the same on Python 3.11, 3.12 and 3.14
  (ast.dump does not). v3 and v4 digests are not comparable. The committed v3 runs here were
  migrated: each recorded hash was matched to the code or reference state it came from (git
  history, unreachable objects included) and recomputed. No score changed.
• import-run hashes the scoring code itself: --scorer-file replaces --scorer-sha256.
  Give the file (or several, repeated in a fixed order) and the harness hashes it by the same
  rule, so a docstring in your scorer is not a new scorer. The hash is now the run's
  instrument.adapter.sha256, inside the fingerprint hash: before, it sat only in imported,
  outside the hash, so two imports scored by different versions of the code looked like the same
  conditions.
• One install calls hosted models. openai and anthropic are core dependencies; the
  [llm] extra is gone. A newcomer's first hosted arm failed on the import, and only one RUNBOOK
  section named the extra.
• The core times every call. latency_ms is recorded for any adapter that does not report
  its own (the attempt that answered, not the retries' backoff); only the bundled adapter used to
  time itself, so a project's adapter recorded no speed. This can move a result: a project's
  runs gain a latency column, and the leaderboard its cost/speed frontier.
• experiment-run warns when outputs stop at max_tokens, by the provider's finish_reason
  (the bundled OpenAI-compatible and Anthropic providers now record it) or tokens_out reaching the
  cap, naming the items: they are scored as if complete.
• A run that stops mid-pass says what to do: which run, after how many items, the exact
  --resume command, or delete it; V7's failure says the same.
• validate V5 no longer fails two different arms that are both perfect. On a saturated binary
  metric their per-item scores agree too; different outputs now tell two measurements from a copy.
• The inspector's item view shows the item itself (what every arm read), folded when long; the
  local inspector only. evh inspect --item prints it.
• Bookkeeping fields (resumed, the judge's self_family_*) stay in the record but are listed as
  metrics nowhere (leaderboard, runs-list, the spread, the inspector).
• The leaderboard's version note fires only for one arm at two versions, not for different arms at
  different versions (judge-score's too); "P(top 1)" is no longer printed beside "P(1st)" when they
  are the same; the rounding note names the scorer as what rounds.
• evh init's .gitignore ignores __pycache__/; experiment-run ends "wrote N run(s) to …".
• Exact p-values at 30 items or fewer. pair-test and family-test now count the
  two-sided sign-flip p over all 2^n sign patterns when a pair shares 30 items or fewer, instead
  of estimating it from 20,000 seeded permutations. Above 30 items the estimate is unchanged,
  bit for bit, including family-test's single stream across opponents. Near a threshold the
  estimate could land on either side of it; the count cannot. This can move a result:
  • in this repository's reports: none moved. All 200 report claims re-derive unchanged,
    because every published pair and family test is on more than 30 items;
  • in the tutorial (6 items): pair-test reads 0.0312 (was 0.0330), and the family test
    0.0938 against lead1 and 0.3125 against author (lead2_noisy's is unseeded and moves
    with every run). The verdicts are the same (getting started step 7, tutorial 3 step 4);
  • in your project: re-run pair-test / family-test on any pair with 30 or fewer
    shared items. Whether a verdict near its threshold moves depends on your data.
• pair-test says which it did: (exact, all 2^n sign patterns) or (20000 permutations).
• family-test's cache key includes stats.py, so a cached answer from 0.3.0 is recomputed.
• validate V4 no longer passes what it did not check. A materialized copy that has lost
  an item file now fails; it used to be skipped. A dataset with no materialized copy on the
  machine (materialized data is not committed, so a CI job has none) is named as not checked,
  and the ok line says how many were checked, e.g. "3 of 14 datasets materialized here". A
  project whose materialized copy is incomplete goes red where it was green.
• The bundled adapter names its primary metric (PRIMARY_METRIC = "overlap_f1"). New runs
  on it record primary_metric: overlap_f1. The leaderboard already ranked them by it, first
  by name; it no longer prints that the metric was "chosen by sort order", and the catalog and
  the inspector open on it. Runs made before keep what they recorded.
• docs/GUIDE.md is getting started, in your own project: install the package, evh init --tutorial, then the loop with evh (a dataset, arms, a reference from a run, a comparison, a
  choice). It used to run inside a checkout of this repository, with make, and to hold every
  step. The rest is in five tutorials (docs/tutorials/): a local model against a baseline, a
  hosted LLM with your own key (which variable each provider reads), a frontier model's run as
  the reference with gold and judging, experiment rounds and dataset versions, and the catalog
  and the inspector with the command behind each view. make guide-check runs getting started
  and the three offline tutorials, each from where getting started ends. The README and the
  RUNBOOK lead with evh.
• The default metric is the one your runs recorded. pair-test, family-test, holdout
  and rank-stability defaulted to coverage, the summarisation example's metric: in any
  project that does not score it they found no runs, or crashed. Without --metric they now
  read the primary metric the runs on the dataset recorded, and refuse, naming the recorded
  metrics, when none did.
• evh self-test in a project checks the installed harness: the paired test's own suite,
  then the tutorial end to end in a scratch folder (dataset with fields, runs, pair-test,
  rescore, validate, catalog). It used to run this repository's development suite, which
  checks this repository's reports and failed 52 checks in every project; in a checkout it
  still does.
• A metric's direction is inferred, not remembered by a flag, by one rule for every command
  and the inspector: a flag, else the goal the runs name, else the adapter's METRIC_DIRECTIONS
  (new, recorded on the run) or import-run --asc (a field nothing read until now), else the
  name (cost, latency and token metrics go down); V14 fails a goal that contradicts its runs.
  So lower is better without --asc for pair-test's "A wins", family-test's wins
  and "ahead", the leaderboard's order and significance block, holdout and rank-stability.
  leaderboard --sort latency_ms ranked the slowest arm first. p-values do not change (the
  tests are two-sided); --asc and --desc override. This can move a wins or "ahead"
  count, and an order, on a lower-is-better metric; no published claim is on one (200/200
  re-derive).
• validate checks the project's own run directories. V7, V8 and V9 read this
  repository's six directory names, so a project's data/runs-adjudicated was never checked;
  V10's scorer sources were this repository's example paths. They are project policy in
  <data>/validate.json (committed, floors, scorer_sources, duplicate_ok); without it
  every git-tracked runs* directory is checked (validate_floors.json is gone).
• The package holds the harness, not this repository's tests. The wheel shipped ~14,500
  lines of tests and report checkers; they are in harness/tests/ and harness/tools/ in the
  checkout. self-test's own suite (test_stats.py) still ships. A run's code_sha256 hashes
  the code that can move a number (the run-writing commands and what they import, read from
  the source), so editing a test no longer shows in run-compare as a confound; 0.4.0 runs
  record a different value from 0.3.0 runs. The deploy installs the inspector from the wheel
  too.
• Refusals and arguments. Every command refuses with ERROR: … on stderr and exit 1 (four
  printed to stdout); a command that takes no arguments refuses one with exit 2 (validate,
  self-test and five checkers ignored them). RUNBOOK: exit codes and every environment
  variable.
• The bundled adapter refuses an unknown provider: (a typo fell back to echo and
  recorded the echoed input as the model's output), and records what an OpenAI-compatible
  provider billed when the response carries it. One price rule for it and the judges.
• One run record, one format; a derived run says what it was derived from. rescore,
  judge-score and the SciFact re-parse write derived: {kind, from, at, with}, their own
  created_at, and source_created_at. rescored_from, rescored_with, judged_from,
  judged_with, judge_label, reparsed_* and primary_metric_asc are gone, and nothing
  reads them: a 0.3.0 derived run is read as a measured one. Every field the harness writes
  is declared in metrics.schema.json, and V1 fails a run record with any other (0.3.0's
  import-run --asc wrote primary_metric_asc, which nothing read; 0.4.0 writes
  metric_directions, which every command reads). EVAL_GOALS_DIR reads goals from outside
  the data root; latency_s is a speed metric, lower is better, as latency_ms is. A goal may name no dataset (one checked by a gate) and may set
  direction: non_inferior, a guard that ranks nothing (no significant drop against a baseline). rescore no longer records a metric it kept as
  dropped (rescore_dropped_metrics named whichever key a loop ended on).
• --resume-unverified is gone. experiment-run --resume refuses rows written before
  0.4.0 (no _extra); experiment-run writes _extra on every row, so its own rows resume.
• The integration seam is adapter.py at the project root. An arm whose config names no
  adapter: uses it, recorded as adapter.py, and the bundled adapter only when there is
  none. The docs lead with the pin and evh init rather than a copied harness/.
• Importing the package changes nothing. .env is read when a harness command starts,
  not on import, and the project's paths resolve on first use, so an adapter's tests or a
  notebook can set EVAL_HOME after the import. evaluation_harness.catalog_adapter reads
  no path at import, and its load_adapter is load_catalog_adapter.
• Configs, judge scorers, judging sets and verdicts have schemas, and validate V1 checks
  them. A config key other than config_id, dataset_id, params, adapter, goal and
  format fails (a mistyped key was read by nothing and passed).
• A baseline carries what its number means: run-promote records the run's reference
  tier, adapter and fingerprint hash, runs-list --baselines prints the tier and adapter, and
  V6 fails a baseline that no longer matches its run. A 0.3.0 baseline fails V6 until it is
  promoted again. Promoting replaces the baseline: the copy archived to
  data/baselines/superseded/ and the supersedes field are gone, and git keeps the old one.
• A command run from a project's subfolder finds the project: without EVAL_HOME, the
  root is the nearest directory up with a data/ folder (it was the current directory, so
  evh validate in a subfolder read an empty project).
• The judge panel is a command, evh judge (it sat beside the package and could not run
  from a pin); the tutorial kit ships tutorial/judge_panel.yaml.

Fixed

• V9 fails when git cannot list the committed runs. A git ls-files that refused (a committed
  directory behind a link that leaves the repository) listed nothing, and V9 read that as nothing
  missing: a run with no rows passed. It now fails and quotes git's error; a project that is not a
  git repository has nothing committed, and V9 says it was not checked.
• silver-calibrate no longer prints a ceiling it did not measure. With one set of runs
  serving as both references and arms, it correlated a ranking with itself and printed "rho =
  +1.000 … from two different sets of runs". The ceiling now counts only arms whose two sides are
  different runs, and says "not measured" when fewer than three are.
• From an external review of the core and the UI surfaces (0.4.0 candidate):
  • A run scored against references for only some items says so. experiment-run took a
    reference directory as covering the dataset, so gold for 20 of 200 items gave a 20-item mean
    recorded as a 200-item gold run. It now counts coverage before any call. With no file for any
    item it refuses. With some, it warns, records reference_coverage on the run (schema), and
    the leaderboard names the run. This can move a result: a run that was refused now stops.
  • A pair opened from a goal page's group is held to the group's family. "test vs leader"
    passes family=<group size − 1>, so it is not tested as if it were the only pair. The
    inspector refuses a family smaller than the arms tested, and a threshold above alpha or
    below what the family allows, instead of obeying a URL.
  • rescore and judge-score keep a project adapter's call-time extras (named in the row's
    _extra) and resumed; they were dropped and reported as metrics the scorer dropped.
  • The adapter contract:
    • INTEGRATION gives lower_is_better(metric, dataset), as the code calls it;
    • catalog_adapter re-exports load_fieldset and fieldset_problems, so an adapter needs
      no private _fields;
    • two new optional members: arm_of(dataset, config_id) (rounds, inspect --round; it
      follows EVAL_RUNS_DIR) and reference_free(metric) (from the metric's kinds, not its
      name);
    • the structured view kinds (insights, nodes, entities) are documented;
    • evh rounds reads the project's catalog adapter;
    • the inspector's default baseline is the promoted baseline's arm.
  • A public inspector publishes text only when told. catalog-serve --public served every
    arm's output and the reference text for any item, while INTEGRATION and the adapter contract
    said that text never leaves the machine. It now serves numbers, adds outputs or reference text
    only for what --publish outputs,references names, and never serves the item itself; the item
    view says what it withholds. This can change a deployment: one that relied on the old
    behaviour adds --publish. This repository's own publishes outputs (already public in the
    repository), not references.
  • The catalog opens on the whole project, and every name is a link.
    • Home is a dashboard: counts (goals, datasets, configs, arms, references, metrics, runs,
      USD billed, tokens), each goal's answer at a glance (its baseline, the leader of the gold
      group and the paired test between them), activity and spend side by side (spend in USD or
      in tokens, so a free local model's use is tracked too), what each dataset covers, and
      everything recorded; each for the last 14 days of activity or all time. evh overview
      prints the same.
    • New pages: Datasets (what the home page listed) and References in the menu; a page per
      arm (its metrics, runs, config without credentials, conditions, the same config scored other
      ways, its rounds; evh arms --arm) and per metric (what it measures and how, which way is
      better, its kind, the goals it decides, every arm recording it with a strip chart;
      evh metrics --metric).
    • Links everywhere: arms, metrics, datasets, references and goals link to their pages
      from every page and table, a reference's author to its arm, a round's configs to theirs; in
      the inspector, arm names and the metric open their pages.
    • Metric explanations: the harness explains its own metrics at length; a project's
      METRIC_DESCRIPTIONS value may be {"summary": ..., "details": ...} for its metric pages.
    • The catalog adapter gains optional arm_runs, arm_config and spend.
    • From the second newcomer run: the inspector warns when the baseline wrote the reference
      the arms are scored against; a judged arm's item shows each judge's score beside the output;
      the goal page says what "filed" means.
  • One provider layer for the systems under test and the judges, keys by role. The bundled
    adapter, judge-score's judges and the judge panel call hosted models through one module,
    _providers: openai (OpenAI or any compatible endpoint), anthropic, openrouter and
    litellm, each taking model, base_url, api_key_env, temperature and max_tokens.
    • A judge goes where its provider: says. Judges ignored it and went to a LiteLLM
      proxy if LITELLM_BASE_URL was set, else to OpenRouter with an alias looked up in
      models.json, while the record said the declared provider. The scorer schema now
      requires provider on every judge. This can change a setup: re-running the
      committed scorers (provider: litellm) needs a LiteLLM proxy.
    • Judging can have its own keys. A judge reads every variable as EVAL_JUDGE_<NAME>
      first, then <NAME>, so it runs on separate keys, budgets and spend logs, or shares the
      experiments' with no setting. evh env-check shows what each role reads.
    • A key is not identity. api_key_env is left out of a run's fingerprint and a judge's;
      no committed hash moved (all 2,800 committed verdicts keep their judge_fp).
    • judge-score names each judge's route and key, dry run included, and stops before any
      call when a key or address is missing.
    • The judge panel's judges are declared like judge-score's (id, provider, model,
      prices). It called only Anthropic, priced Sonnet in code, and ran the fake judge when
      ANTHROPIC_API_KEY was missing; the fake judge now runs only with --use-fake-judge.
    • Anthropic takes base_url and api_key_env like the others. Arms pass reasoning,
      reasoning_effort and provider_routing to the request as judges did, and record
      reasoning_tokens (descriptive, as every committed run declares it), the model and the
      host that answered.
  • The harness imports itself as a package. Its modules imported each other as bare
    top-level names (import stats) with their own directory on sys.path, so a project's own
    stats.py first on the path replaced the harness's paired test inside
    evaluation_harness.judging. Under evh, the harness's adapter, stats or rounds could
    also hide a project's module of that name. They now import evaluation_harness.*, and nothing
    of the harness goes on sys.path. In a checkout harness/evaluation_harness links to
    scripts/, and python scripts/x.py still runs. A project script that relied on the
    harness's directory being on the path writes from evaluation_harness.adapter import Result,
    as INTEGRATION documents.
  • The public server trusts no client header unless told (--client-header NAME); a
    spoofed CF-Connecting-IP no longer buys a fresh rate bucket. The deployment names
    Cloudflare's header (origin lock).
  • The rest of the page features have commands: evh inspect --no-rescore (the inspector's
    box), its "Average only" table, and the dataset page's fields block in
    evh datasets --dataset-id.
  • Statistics, consistently:
    • family-test marks an estimated p (~) when a family mixes exact and estimated rows;
    • pair-test separates at p <= threshold, as Holm and the inspector do;
    • the bootstrap ties ranks with the same tolerance as the within-item ranks;
    • runs that disagree on a direction or a primary metric are named;
    • rescore refreshes metric_directions;
    • the inspector and evh inspect show each row's 95% interval of Δ (exact up to 30 items, a
      t interval above, as its docstring now says).
  • Accessibility: a sortable column's header works from the keyboard and states aria-sort;
    every chart mark with a tooltip carries an SVG <title>.
  • Messages: rescore's missing-reference message no longer sends a project to
    examples/; RFC-002 tells _extra (adapter extras) from _meta (provider metadata).
• From the second newcomer run (/work/evh-scenario-040; SCENARIO, 0.4.0 candidate):
  • A baseline promoted from one of an arm's repeats records every repeat and their mean
    under repeats, and run-promote prints the two side by side. The catalog, the inspector
    and the tests read the mean; scores stays the run's own (V6).
  • A judged run keeps its source's cost and speed kinds, so the judged leaderboard has its
    frontier. That leaderboard also prints the judging pass's agreement and its position and
    noise checks, from summary.json.
  • The arm that wrote a silver reference is left out of the ordering test and the frontier,
    and the leaderboard says so (its row measures agreement with itself).
  • A judge reply that is not committed (not bare JSON, or repeating the source) keeps its
    text in a git-ignored .withheld/ beside its record, and the log names the file. An
    unparsed reply's cause can then be read; the record keeps only the hash (V12).
  • The dataset page and evh datasets sort arms by what they are scored against (gold,
    silver, a judge, nothing) before the mean. A silver and a gold mean no longer interleave.
  • The goal page and evh goals --goal show a judge's group, read on its own score. evh goals --goal also lists the rounds and marks retired arms.
  • One p format everywhere (stats.fmt_p: four decimals, <0.0001 below that).
    family-test says whether p is exact or a seeded estimate. Above 30 items it draws one
    random stream for the family, so pair-test and the inspector, which estimate one pair at a
    time, can differ in the third decimal. Its wins column is labelled A wins.
  • Round messages:
    • round-open --keep says carried arms need no change and no new run;
    • evh rounds does not suggest promoting the run that is already the baseline;
    • leaderboard --round does not warn about two versions of an arm in a round that compares
      them.
  • evh cost-report sees the bundled adapter's bills. A provider's bill (OpenRouter, or a
    LiteLLM proxy passing it through) was the run's cost but was not kept where cost-report reads
    it, so every run read as "price table only". It is now kept in the item's _meta.usage.cost.
  • evh inspect --item prints which run each output is from and, for a repeated arm, that it
    is the latest of N repeats beside their mean (as the page does). evh datasets counts configs.
  • Smaller items:
    • experiment-run prints what the command spent, and its repeat spread calls cost and speed
      what they are, not "deterministic";
    • judge-score says its spend is the provider's bill where one is reported, else the price
      table;
    • silver-calibrate explains rho and sibling lift;
    • the catalog counts the configs behind its arms;
    • "filed" reads "filed under this goal";
    • the item view says it shows the latest of N repeats beside their mean;
    • judged arms keep their version in the inspector;
    • a heading scrolled to is not hidden under the header;
    • env-check names the example arms only in a checkout;
    • the tutorial scorer points at tutorial 3.
• One reference per table and per arm. The leaderboard averaged an arm's runs whatever they
  were scored against: a silver 0.90 next to a gold 0.24, in one table labelled silver. It now
  prints one table per reference, gold first, each with its own tests. The catalog, the
  inspector and evh rounds read an arm against the reference of its latest run (by
  created_at), and they name the earlier runs they leave out. A judged run is grouped by its judge,
  not by its source run's reference. evh rounds reads a round against the reference most
  of its arms use and names any arm scored against another. This repository's data, the
  trial and the podcast project have no such arm.
• provider: openrouter uses OpenRouter. It read OPENAI_API_KEY and called OpenAI's
  address unless the arm set both base_url and api_key_env. It now reads
  OPENROUTER_API_KEY and calls https://openrouter.ai/api/v1, and an arm's own values still
  win.
• evh inspect --round reads the round as evh rounds does: on the goal's measure, against
  the round's leader. It defaulted to the first arm, so an arm with no score became the
  baseline and the metric fell back to cost_usd. Arms with no score are named and left out.
• The bundled adapter stores overlap_f1 as computed. It rounded the value to 6 decimals,
  which made ties that are not there (the leaderboard warned about its own example). This can
  move a result only where two arms tied at the sixth decimal.
• Hints and messages:
  • with no reference, experiment-run names reference-create --from-run first;
  • evh --help describes reference-create as "from a run or with a config";
  • env-check and the litellm provider no longer tell a pinned project to copy a
    .env.example it does not have;
  • run-promote no longer prints resumed and self_family_* among the scores (the baseline
    file still keeps them);
  • inspector notes are no longer all labelled "Re-scoring";
  • the bundled adapter and dataset-create --help describe what the code does now, not its
    history.
• A judge scorer must state its position and noise checks. ADR-0001 makes them part of the
  pass, not optional, but a scorer that left protocol.position_check_fraction or
  noise_check_fraction out ran with them off. Both are now required. The noise check must be
  above 0, and so must the position check whenever a call sees more than one output (a pointwise
  scorer has no order to check). judge-score and V1 refuse a scorer that breaks this.
• A round is read against the arm that wrote its silver reference. evh rounds and the goal
  page now compare the leader with the previous round's best, the baseline and, new, the
  frontier arm whose outputs became a silver reference of the dataset. Cost per item is shown
  for each of those arms. That arm is the yardstick, so it is never suggested for retirement.
• The inspector reads only arms and references the catalog lists. It asked the
  adapter's arms_of about any path a request named; an adapter that maps a path to itself
  (the podcast repo's did) let compare read outside the data tree (runs=../../.git). A
  request may name an arm, or a path that selects arms (a run directory), but only arms the
  adapter's arms() reach, and a reference references() lists.
• A catalog nav entry may carry a CSS class as a fourth element (hide-sm hides it on a
  phone), and the inspector page names the command that serves it (serve).
• evh rescore and evh silver-calibrate in a project that pins the package. A run
  made with no adapter: uses the bundled adapter and records it as scripts/adapter.py, a
  path that exists only in a checkout, so both failed with "adapter not found" in every pinned
  project on the default adapter (0.2.0 and 0.3.0 alike). They now score with the package's
  own copy, the one that made the run, through one shared lookup. Found by following the GUIDE
  in a brand-new project.
• The "next, run this" hints name a command you have. Every one said make …, a target
  only a checkout of this repository has; through evh they now say evh … (the first was
  GUIDE step 1's "Next: make dataset-materialize"). Two were wrong in a checkout too: an empty
  baselines list suggested make promote (the target is run-promote), and an empty runs list
  suggested make demo, which a project without the bundled sample data cannot run.
• evh catalog-serve on a port another program holds says so and suggests --port,
  instead of a Python traceback.
• rescore onto another reference no longer claims the references changed. Moving runs
  from silver to gold (--reference-id) compared the gold files' hash with the silver files'
  and, as they always differ, wrote references_changed_since_measurement: true into every
  re-scored run and printed "the reference files have CHANGED". The move is recorded as
  before (reference_id_at_measurement); the flag and the warning are kept for the same
  reference's files changing. No committed run in this repository carries the false flag.
• Commands that assumed a checkout, each found by running every evh command in a
  project that pins the package:
  • rank-stability crashed (set.intersection()) instead of saying no arm had the metric;
    on a lower-is-better metric it named the worst arm the leader; with too few items for two
    halves it now says so.
  • --exclude-dataset (pair-test, family-test, holdout, bootstrap-test) read
    data/sources/<id>/*.txt instead of the dataset file: wrong for any item that is not a
    .txt, any dataset from a manifest, any project keeping sources elsewhere, and "nothing
    excluded" was then reported as true.
  • The leaderboard's metric kinds, run-compare's, and the classification report's label
    parser looked for a run's adapter in the checkout only; they now find the bundled one in
    the package, as rescore does.
  • bootstrap-test on a gold set of free text (summaries) treated every summary as a class
    and would have reported a macro-F1; it now refuses, and its "too few items" refusal names
    the threshold and the opponents.
  • catalog-serve printed its address without flushing, so under a log file or a process
    supervisor the line did not appear; --build-dir outside the project was a traceback,
    now a refusal; self-test --help ran the whole suite instead of printing help.
• The catalog counted a dataset's items from the harness's own path, not the adapter's,
  so a project with its own layout got "the most items any arm scored". It now asks the
  adapter (dataset_items, optional).
• The bundled adapter's anthropic provider failed every call with the anthropic SDK 1.x,
  which dropped the temperature= keyword: a TypeError before the request was sent. The
  temperature now travels in the request body, so the request is the one every SDK version
  sends. A self-test checks the call against the installed SDK's own signature. Found by mypy.
• A manifest URL could read a local file. dataset-create --manifest fetched any scheme
  urlopen accepts, so a file:///… row was copied into the item cache and pinned, although
  a dataset's schema allows only http(s). Every fetch now refuses other schemes before reading
  anything. Found by bandit.
• The judge panel judged placeholder text. Every finalist, real runs included, was scored
  on "Synthetic item i for ". It now reads each finalist's stored outputs, finds runs
  where every command does (EVAL_RUNS_DIR, the project), ranks by their primary metric, and
  refuses a lower-is-better one rather than promote the worst.
• rescore could record the bundled adapter's metric kinds in a project's run: two
  registries named loaded adapters differently, and the lookup fell back to the bundled one.
  One loader (_adapter.py), keyed by the resolved file.
• import evaluation_harness.judging failed in an installed package (No module named 'stats'), and importing catalog_adapter beside it loaded _common twice.
• holdout's significance block had no direction, so on a lower-is-better metric it ranked
  the worst arm first.
• Checks that could not fail, or did not run: self-test's cost-cap control turned any
  failure into a skip; check_terminology was failing and nothing ran it (it is in make test
  now); make -C harness docs was a no-op (eight targets were not .PHONY).

Added

• evaluation_harness.stats. Standard library only, and the one place the paired test
  lives:

  • exact_p and sign_flip_p;
  • holm, with a declared family larger than the p-values passed;
  • exact_ci, an interval for the mean difference that agrees with the test: none at 5
    items or fewer, where no shift can be rejected;
  • compare, every arm against a baseline with verdicts in words;
  • compare_by, the same test within each group of items, Holm-corrected over every group
    and arm together;
  • ranks, pearson and spearman (average ranks for ties; a constant input returns
    constant, NaN unless you pass another), which the judging, rank-stability and
    report-check scripts share instead of keeping their own copies.

  It has its own test suite, scripts/test_stats.py, run by make test.

• The exact count is ~4x faster (0.03 s at 30 items, was 0.11 s): the sign-pattern sums are
  built by doubling a list instead of enumerating each pattern. The same sums, so the same p.
  The exact interval bisects to ~1e-9 of its span, not ~1e-14: 1.4 s at 30 items, was 11 s.
  It is shown to 3 decimals.

• evh catalog and evh catalog-serve (make catalog, make catalog-serve).

  • The catalog: static pages over a project's runs, by dataset, built into
    .build/catalog/. Each dataset page lists every arm with its headline number, and links
    to the inspector for one arm, or for all the arms on one metric scored against one
    reference.

  • The run inspector: a server on 127.0.0.1 that tests any arms on one dataset against
    a baseline with stats.py, so it gives the same p as pair-test on the same pair. It
    shows the verdicts, charts of the per-item scores and differences, and, for one item,
    each arm's output beside the reference.

  • The same rules as everywhere else:

    • Arms on different dataset_ids are refused.
    • On a metric scored against a reference, an arm scored against another reference than
      the baseline's (silver against gold, say) is named and not tested against it; cost,
      latency, tokens and length compare across references.
    • The static pages carry counts and means only, never an item's text.
    • The server answers only to a 127.0.0.1 Host. It serves only the catalog and the
      data tree, and no dot-folder in either.

  • An adapter is the only project code needed. catalog_adapter.Adapter lists the
    questions both tools ask. The built-in HarnessAdapter reads the harness's own layout
    (runs*/, runs-judged/, references/): an arm is one config on one dataset, its
    repeats averaged per item. A project with another layout names its own adapter in
    config.yaml (catalog: {adapter: path/to/file.py:Class}) or passes --adapter.

  • A public mode (evh catalog-serve --public HOST), for a deployment behind a reverse
    proxy that serves the static pages:

    • it answers only the named host;
    • each client gets --rate requests a minute;
    • at most --max-arms arms in one comparison;
    • two computations at once, every answer cached;
    • errors never show the server's internals;
    • --warm computes, at start, every comparison the built catalog links to.

    evh catalog --link LABEL=URL adds the deployment's other sites to the nav.

  • CSP-clean: no inline script or style. The theme script is a file, and the legend
    colours are set from JS.

  Tests: tests/test_catalog.py (54 assertions), run by make test.

• Goals: what an evaluation is for (data/goals/<goal_id>.yaml, schemas/goal.schema.json).

  • What a goal says: who uses the output, the decision it informs, the measure that
    decides (and which direction is better), the output formats it accepts, its datasets
    and their roles, and optionally min_items, the fewest items a winner may rest on.

  • Configs name their goal (goal:) and may declare the format they ask for.

    • experiment-run refuses an unknown goal before it runs.
    • The run records both, outside the fingerprint: they change what a run answers, not
      what it measured.
    • run-promote carries both onto the baseline.

  • validate V14 fails:

    • a goal that breaks its schema, or whose goal_id is not its file name;
    • a config, run or baseline naming a goal that does not exist, or a format the goal
      does not accept.

    A dataset smaller than a goal's min_items gets a note.

  • Optional until a project opts in. Set required_from: YYYY-MM-DD in
    data/goals/policy.yaml, and every run created from then on must name a goal. A
    project upgrading changes nothing until it sets the date.

  • In the catalog: a goals page, and a page per goal. On each of the goal's datasets
    it ranks the arms that record the goal's measure. An arm under min_items or in a format
    the goal does not accept is flagged and never the leader. Each arm links to the
    inspector, tested against the leader.

  • This repository's six experiments have goals (data/goals/), and so does the
    tutorial (tutorial/goals/tickets_summary.yaml). No date is set here, so nothing
    already committed is held to one.

  Tests: tests/test_goals.py (14 assertions), run by make test.

• Item fields (data/fields/<fieldset_id>.yaml, schemas/fieldset.schema.json, ADR-0003).
  The harness defines no field: a project declares its own in a field set, and any number of
  its datasets name it.

  • Types: text, number, integer, boolean, date, duration (seconds; h:mm:ss accepted),
    url, and choice with its values; a field can be required and public.
  • dataset-create --fieldset ID --fields FILE attaches values from a CSV or JSONL, one
    row per item_id, and refuses an undeclared column, a mistyped value, a value outside a
    choice, an unknown item and an empty required field. --update-fields corrects values on
    an existing dataset.
  • Not the identity. A run's fingerprint hashes item ids and bytes only, so a corrected
    field invalidates no run; the dataset records fields_sha256 beside them.
  • validate V15 fails a field set that breaks its schema, fields with no field set or an
    unknown one, an undeclared or mistyped value, a missing required value, and a
    fields_sha256 that does not match (a hand edit).
  • In the catalog: each field with its coverage, and a summary of its values only when it
    is public. In the inspector: an item's fields beside its outputs; a filter on any
    field; a breakdown by a choice or yes/no field, tested per group with stats.compare_by.
    A public deployment answers with public fields only.
  • The adapter protocol gains an optional item_fields(dataset).

• Items fetched from a pinned URL. dataset-create --manifest FILE takes one row per item:
  item_id, source_url and source_sha256 (empty: fetched and pinned now), and field
  columns. dataset-materialize downloads each into .cache/items/ and refuses bytes that
  hash to anything else, as it refuses a changed file. A transient failure is retried like a
  provider call; a host that needs a header gets it from EVAL_FETCH_HEADERS (that host only,
  never on a redirect elsewhere, never written into a dataset). With --update-fields, an
  empty cell clears a value.

  Tests: tests/test_fields.py (47 assertions), run by make test; the catalog's in
  test_catalog.py.

• evh init [DIR] [--tutorial] starts a project: data/sources, data/configs,
  data/goals, data/fields and a .gitignore for what the harness derives. --tutorial
  copies the GUIDE's kit (tickets, their fields, gold, arms, a goal, a judge scorer and its
  prompt, the judge panel's config) into tutorial/; the kit ships inside the package. It never overwrites a file.

• make guide-check (part of make check) runs docs/GUIDE.md in a fresh project, as a
  reader would, and fails when a quoted output no longer matches what the command prints.
  Lines that the unseeded lead2_noisy arm moves are marked in the GUIDE and not compared.

• The tests that run evh install the real wheel (tools/_wheel.py), built from
  pyproject.toml, instead of copying folders. A file the package data leaves out now fails
  them, where a copy would have included it.

• make browser-check drives the catalog and the inspector in headless Chromium: a
  tutorial project with fields, the main paths and the field controls, the phone-width layout,
  and no console error. A development extra (requirements-browser.txt, make browser-install), never a dependency of the package; CI runs it as its own job. The root
  make browser-check also drives the public site, served with production's headers.

• tests/test_installed_commands.py (in make test) runs every evh command in a
  project that pins the package; a command without a case fails it.

• tests/check_commands.py (in make test) checks every make target, evh command
  and flag shown in the current docs still exists: 274 of them across 27 docs today.

• The deploy's health check also asks the inspector the question readers send most, a
  group of arms on one metric (infra/deploy/probe_link.sh), and rolls back without verdict
  rows.

• Code-quality gates, make quality at the repository root and a CI job of the same name:
  ruff (lint and format), mypy, bandit, pip-audit, vulture, interrogate, codespell, shellcheck
  and actionlint, with radon's complexity report. Tools pinned in requirements-dev.txt
  (make quality-install), never a dependency of the package. What they found, and how ruff
  compares with the black, isort and flake8 that podcast_scraper runs:
  research/internal/QUALITY_TOOLING_EXPERIMENT.md. Every public function, class and method
  has a docstring (interrogate at 100%, the bar podcast_scraper sets). Logic kept in two or
  more copies (Spearman, Holm, the sign-flip test, module loading, the test scripts' check(),
  cost sums, file hashing) now lives in one place each, and every report claim re-derives
  unchanged; a test script that runs no check now fails.

• This repository's worked examples run on patched libraries (Linux): transformers 5.20.2,
  sentence-transformers 6.2.0, pillow 13.0.0, aiohttp 3.14.5, nltk 3.10.4, certifi 2026.10.19,
  clearing 58 published advisories in their locks; few-nerd stays on transformers 4.55.4,
  because span-marker has no transformers 5 release. make audit-locks audits every lock per
  platform. Every local arm was re-run against the committed record: every published finding
  holds, and all but four arms reproduce item for item (why, per arm, in the record).
  Not part of the package.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.4.0#subdirectory=harness"