evaluation-harness v0.7.1

Fixes from the fifth newcomer scenario (the first on a release, 0.7.0) and from releasing 0.7.0. A judge that
returns empty replies no longer leaves a tree that fails evh validate; the commands say more about what they
did and what they spent; the public MCP server answers a leaderboard at once. No committed run's scores move;
three figures read differently (Changed).

Changed

• family-test --round tests the round's leader against every other arm in play, without --a and
  --against (as pair-test --round picks its pair), and reads a re-score in place of the run it re-read, as the
  round does. A family test of a round can give a different p than 0.7.0's runs-only read.
• The billed figure counts arm smokes. evh overview's billed tile and the catalog's include what
  evh arm-smoke spent (recorded in the model records), and the spend lists name it as "arm smokes".
• runs-list --baselines shows the baseline's mean over its repeats, as every other view does; --json
scores is that mean and a repeats column says over how many.
• --json fields: the leaderboard's frontier gains dominated_by; evh goals --goal's last column is named
  decision (it was ""); the dry run gains input_tokens_counts; arm-smoke has --json.

Fixed

• An empty judge reply is withheld like any other unparsed reply. A reasoning judge that spent its whole
  max_tokens thinking returned nothing; judge-score stored it as text and evh validate's V12 then failed on
  the harness's own record. Now it is withheld (its hash kept), V12 names the command that rewrites one stored
  before 0.7.1 (evh judge-score … --retry-unparsed), each unparsed reply says why (empty, cut off at the token
  limit, not JSON, not the expected shape), a pass whose replies were cut off ends with a HINT (raise
  max_tokens, or reasoning: {enabled: false}), and its progress lines arrive as they happen through a pipe.
• evh arm-smoke prints each reply (its first 300 characters; --replies for all) and what it spent, and
  times a call that the adapter did not time.
• evh run says whether "spent" is the provider's bill or the config's price table, prints no "warmed up in
  0ms" for an arm with nothing to load, and counts the config's prompt in the dry run's input estimate.
• rescore ends with a whole sentence that says when a re-score stands in for a run, and warns about outputs
  cut off at max_tokens. score-preview says when a goal holds no measured run to re-score.
• silver-calibrate says what --ref-match and --arm-match take, with an example from your runs.
• Leaderboards fit the terminal (the ranking metric first, the rest named; --wide for every column), and
  "dominated" says what it means (as good on all three, better on at least one) and by which arm.
• inspect shows a model name whole in its conditions, and --round names the round's leader as such;
  evh goals --goal's headers line up; cost-report names each repeat; evh rounds prints a round's
  conclusion whole.
• The catalog and the inspector: the inspector names the silver reference's author and marks its row; an
  item's cards show each arm's score; a goal page's "test every arm against its leader" leaves out retired arms;
  a judged group below the goal's min_items says so; a recommended dataset id may break, so the goals table
  fits; home tiles keep their number on one line at 1400 px.
• The public MCP server answers leaderboard for each dataset at once: evh mcp --write-answers writes it
  ahead per dataset, and the leaderboard's permutation test is kept in the p cache (EVH_P_CACHE) with the
  generator state the rank intervals draw on, so every number is the same. A first call took 47-60 s.
• The website's on-this-page line names each section in a word or two and leaves out "Who built this".
• Deploys and CI: the release renders while CI runs, and the site's build check runs beside verify.
• Docs: tutorial 3 shows the whole judge scorer and what to do when a reasoning judge returns nothing;
  tutorial 5 lists the inspector's URL parameters; the RUNBOOK has a complete goal file; the newcomer scenario
  smokes each arm first, says what each step writes (all committed), and seeds pip in a uv venv.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.7.1#subdirectory=harness"

evaluation-harness v0.7.0

Agents as users. evh init writes an AGENTS.md, three skills teach the loop, and the ranking and the tests answer
in --json, so an agent with a shell needs neither the catalog nor MCP. The features and fixes the first beta
agents asked for are in, among them crosstabs of per-item scores, a hypothesis recorded before a run, pair-test
across two versions of a dataset, and capture-restore. Upgrading can change a verdict: a leader the tool picked
is now corrected for the pick, and a round no longer promotes on a tie (Changed). No committed run's scores or
fingerprint move.

Changed

• A leader the tool picked is corrected for the pick. pair-test --round takes A from the table, so its p is
  Bonferroni-corrected for the comparisons A could have been (every arm in play but B); a round's promotion gate
  corrects the same way (p × K over the round's arms). A verdict that was significant uncorrected can read
  not significant. A pair you name yourself is unchanged, and still uncorrected.
• A round promotes nothing when its leader is tied, or its arms were scored differently. Before, it promoted
  the first of the tied arms. The hold names the arms and says how to break the tie (the goal's tie-break picks
  among them; evh goals --goal names the arm it recommends). A closed round suggests nothing.
• A round's baseline is read re-scored. It was read as first scored, so the rule "a leader must beat the
  baseline" did not run on re-scored rounds; it does now, and can hold a promotion it let through.
• The leaderboard's JSON lists no distinguishable pairs without an arm effect. Pairs are read only after the
  global (Friedman) test finds one: without it, pairwise_valid is false and distinguishable_pairs is empty.
• dataset-from-capture --from replaces an item with the capture's record of the same id, counted as changed
  in the lineage. Before, the earlier version's body was kept and the lineage read "0 changed". --fieldset
  alone keeps every field it declares; before, it kept none without --field.
• A run is never written into another run's directory. A run id is the config and the start time to the
  second, so two runs of one config started in the same second shared a directory and the second wrote over the
  first, a committed run included. The second now takes the next second's id; --run-id naming an existing run
  is refused.
• evh run --dry-run no longer calls its estimate an upper bound. It counts one call per item, or an adapter's
  CALLS_PER_ITEM, and says more calls or a growing conversation cost more. Where the model record has an
  arm-smoke of the config, it shows the measured cost per item projected over the run. --json adds
  estimate_is_upper_bound: false, calls_per_item and measured_usd_per_item.

Added

• AGENTS.md (and CLAUDE.md importing it) in every new project from evh init: the rules an agent must not
  break, the loop from nothing to a decision, the --json command for each question, and what an agent may do
  (run arms, label and promote when its operator asked, under EVAL_MAX_COST_USD; closing a round stays the
  operator's).

• Three skills, and evh skills-install: evh-evaluate-a-system, evh-compare-and-read and
  evh-promote-with-evidence, shipped with the package and copied into .claude/skills/ (or --target). Every
  evh block in them runs as written in the tests.

• --json on leaderboard, pair-test, family-test, run-compare, rank-stability, runs-list and
  rounds: one envelope, {"command", "version", "result": {"blocks", "values"}}, with raw numbers. The text
  output is unchanged. evh run prints each run id it wrote (values.runs under --json), and its progress as
  one [n/N] item outcome seconds line per item while it runs (on stderr under --json).

• Five more MCP tools: leaderboard, pair-test, family-test, rank-stability and run-compare (23 tools,
  22 with --public).

• A hypothesis recorded before the run (ADR-0025):
  a hypothesis: block in an arm config (which way, against which arm, on which measure) is recorded in each run
  with whether it was committed before the run. pair-test and evh goals --goal show it; an arm with runs made
  before it reads "not named in advance". evh validate V20 checks it. Not in the fingerprint.

• evh inspect --runs … --crosstab A,B [--group FIELD [--group-match RE]]: two per-item scores crossed, per
  value of a field of the run's prediction rows, the items off the diagonal named. --items-out: each run's
  whole output and scores beside the reference and labels (--disagree: only the items the runs decided
  differently).

• evh goals --goal G --arms PATTERNS --top N: the rows shown filtered; the decision is still made over every
  arm, and the baseline, the leader and the recommended arm are always shown. Each row has its paired test
  against the leader.

• pair-test --b-dataset D2: one arm on two versions of a dataset, paired by item id. Refused unless the
  reference is the same text for each shared item; it says it is cross-version, what that establishes and what
  not, and names the runs of each side (run_ids_a, run_ids_b).

• evh capture-restore --capture C [--query KEY=VALUE]: a capture's records, and the item files of every
  dataset made from it, read again from the recorded importer and query (a key replaced, e.g. the same files in a
  download folder), a recorded scrubber run again; every record and item must match its recorded sha256, or
  nothing is written.

• The triage example's v2 (orrery_triage_v1_labelled_v2): the same 65 owner-labelled decisions with the
  rebuild's hindsight taken out (probe answers that stated an outcome, release and last-seen times after the
  decision), as revision v2 of the dataset on Hugging Face; examples/triage-fleet/fetch.py --revision v2
  restores it. Round r4 of triage_orrery runs on it.

Fixed

• An importer path in a capture works from the project root as well as from the directory it was given in.
• The triage example's fetch restores its capture again: the importer's newer fields come only when the query
  asks (prompt_fields, v2_fields), so a past capture's records hash as recorded.
• evh inspect --items is no longer read as --items-out, and --crosstab refuses a name that is not a
  per-item score, pointing at --group.
• A label set is written whole or not at all: to a temporary folder renamed into place.
• pair-test --b-dataset refuses a measure stored only in re-scored runs as a missing metric (naming
  evh rescore), not as "no shared items".
• family-test --against takes a list; --family below 1 is refused; a mixed family's n is its range; a goal's
  --goal cannot name a file outside data/rounds; round-open --keep's help says it accepts any arm already run.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.7.0#subdirectory=harness"

evaluation-harness v0.6.2

Faster answers from evh and its MCP server. No result moves.

Added

• evh mcp --write-answers FILE and evh mcp --public --answers FILE: a release's answers written ahead. Each
  call to the MCP server runs a fresh evh process, several seconds of CPU; a deployment that limits the server's CPU
  pays it in tens of seconds (the evals server's first goals after each start, and after each hour, took 33 s at
  half a core). --write-answers writes the answer of every tool that takes no argument and does not depend on the
  date (not overview or activity); a public server started with --answers serves them without running the
  command, for as long as it runs. A line another harness version wrote is not read.

Fixed

• A view and a catalog build read each arm's scores once. evh goals read 1,054 arms' scores for 686 arms; that
  reading was 79% of its CPU. The answers are byte for byte the same; goals, arms and models take about a
  quarter to a third less CPU.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.6.2#subdirectory=harness"

evaluation-harness v0.6.1

Fixes to 0.6.0, from two reviews of what it shows. Six can move a number or a verdict a project sees (Changed): the
sheet's win rates and verdicts, a goal's recommendation and the places in its field, and the catalog's run and
spend figures. One keeps a hidden dataset's labels and notes off public surfaces (Security). The rest restore what
0.6.0 documented or make the catalog say what a number means beside it (Fixed).

Changed

• A goal recommends from the group it decides on, once per dataset. A judged group (a grade, a pick, a pairwise
  pass) or another measure of a scorer is shown with its leader, never recommended: 0.6.0's CNN goal page printed
  four "Recommended:" verdicts, three of them from judging passes the goal does not decide on.

• A goal's field places one arm per config, without the arms a round retired (a re-timing, a second run, a
  paired side run). A config run twice took two places: AG News anthropic_l read "#8 of 33" in the catalog where
  the site reads 5th of 30, and the Models page put Gemma 4 31B first on SummEval through its paired side run.

• Runs re-scored or re-parsed in place count in every period figure, dated when they were measured: 0.6.0 left
  Few-NERD's and SciFact's 54 runs and $5.58 out of the activity, the datasets page and the home tiles. A copied
  re-score (judged, re-scored, silver, adjudicated) keeps its run's id too and is still never counted.

• The sheet's verdict compares only the arms scored on the metric: an arm with no score on it reads "no
  in this run", where 0.6.0 named the other arm the leader and recommended.

• The sheet's win rate on a pick or pairwise verdict is each arm's share of what the judge gave over the rows it
  judged (its judge_won), so with every arm shown it is the judging pass's own win rate. 0.6.0 gave a shown arm a
  whole row when it held the largest share among the arms shown, however small: on the CNN pick pass anthropic_l
  read 0.646 in the sheet against the pass's 0.402.

• The sheet's win rate on a metric is over the rows each arm was scored on, "—" for an arm scored on none. 0.6.0
  divided by every row with a winner, so an arm with no value on a row counted as beaten there, and one never scored
  read 0.000.

Security

• A hidden dataset's labels and notes stay off public surfaces (RFC-019 §4: per-item labels, adjudication notes).
  0.6.0 showed its label sets' confusion matrices, whose rows and columns are the label values, on the catalog's
  dataset page and in a public server's labels; listed a person's sheet notes on its items in a public server's
  labels; and copied those notes into a deployment's data. The counts, Cohen's kappa and Krippendorff's alpha stay.

Fixed

• The installed evh reads the project's .env. 0.6.0's evh, as pip installs it, loaded none (env-check:
  "loaded 0 var(s)"), so a key or a cost cap put there was not in force; python -m evaluation_harness and a
  checkout were unaffected. Found by the newcomer scenario.
• reference-promote adopting a hand-made folder keeps its files' bytes. It added a newline to each, so the
  reference digest every run scored against the folder had recorded no longer matched (rescore warned the
  reference had changed), where UPGRADING said no run's reference moves.
• Fetching the examples' public data falls back to the Hub's parquet files when the Hugging Face
  datasets-server's /rows keeps failing (its outage of 2026-11-02 answered HTTP 503 for every dataset): the same
  rows and row numbers, read with pyarrow, now in each example's environment; every item is still checked against
  the frozen dataset. The installed harness does not depend on it.
• inspect --sheet --wrong-against NAME refuses a name two datasets' sets share on this dataset's items (labels
  follow an item into every dataset holding it), naming each as DATASET/SET; it read whichever it found first.
• A round's best is read as its goal decides: on the deciding reference, never the arm that wrote it (CNN round
  r1's best read anthropic_l 1.000, the silver's author scored against itself, beside a conclusion naming bart_l).
• Gold reference sets are dated by when their labels came to be: the dataset's freeze, or the adjudication. The
  0.4.0 migration had stamped every one with its own time, so the references page listed sets a month old as new.
  No run's reference digest moves (the digest leaves the date out).
• evh deploy-sheet and the goal page say a key production never recorded "cannot be checked", where they read
  "prompt_sha 0 match; not recorded: 233".
• The catalog says each thing once, and what a number means beside it. A goal page names a re-score's scorer
  once, counts a long list of contenders, folds a conclusion its answer already shows, and explains its groups and
  strategy once. Dataset pages open on the deciding measure, give one inspect button per judging pass, name the
  machine in one form ("served on", "run on"), say how much of a period's spend was judging, and show a rise past
  doubling as a multiple. The answer column no longer scrolls off at 1280 px; chart names never overlap; a cost every
  arm shares draws no axis; a metric's strip leaves out the arm scored against what it wrote; ids break between
  words at phone width; the sheet's rows stay short and name arms by what sets them apart; probes read yes or no.
  The triage datasets and fleet_dedup_v1 have their profiles (evh dataset-profile).
• Site: SciFact leads with the compared field (the best reranker +0.025, not significant) and names the extra
  +0.07 run with its caveats; the method page describes fingerprint v5 with 0.6.0's own run-compare output; the
  use page counts its steps; the XSum control's caveat and the judging spend's passes are carried where they are
  quoted.
• Docs: tutorial 5 and the RUNBOOK named the old fixed ports (8765, 8766) where 0.6.0 gives each project its own;
  the home page's MCP section names what 0.6.0 added and that the public server never answers with an item's text.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.6.1#subdirectory=harness"

evaluation-harness v0.6.0

What you get: datasets made from what production did (capture, scrub, split, dataset-from-capture); labels that say
who decided what is right, and reference sets promoted from them; production's own decisions scored as an arm;
models onboarded before they are run; scoring that evolves without re-running an arm; judges that grade, pick the
best or compare two; and one sheet of every item and arm, filled in Excel or another tool and read back. Upgrading
moves no committed score. Two formats (fingerprint v5, model records) and the old references are migrated by a
command, and a few names change to the field's words with no alias (evh run, significant, contenders):
UPGRADING has the steps.

Added

• Labels, and the reference sets promoted from them (RFC-023, PRD-002, ADR-0022). A label says what is right
  for one item and who said it: a person, a model or a rule. Labels come in sets, never edited, under
  data/labels/<dataset>/<set>/, and follow an item by its sha256 into any dataset holding the same bytes.
  • evh label-from-run: a finished run's outputs as a model's labels, no calls, the refusals a reference's
    provenance needs (a dirty or unknown build, an unfinished or derived run, an empty output, one cut off at
    max_tokens unless --allow-cut-off). --not-redistributed keeps the texts out of git (a .gitignore beside
    them) and records the command that rebuilds them, checked against the set's checksums.
  • evh label-sheet writes a sheet of the items to label (unlabelled, all, where runs disagree, or chosen),
    each run's decision and a proposal beside an empty label; evh label-import reads it back, or a folder of
    <item_id>.txt, as a person's set (yes takes the proposal; notes kept). The sheet is CSV that Excel opens
    and saves (UTF-8 with its mark, , or ;); --choices puts the allowed answers in the label column's
    header and the import refuses any other; the local inspector downloads it. evh label-from-capture turns a
    capture's recorded outcomes into a rule set by the rules reference-from-outcomes read. evh labels lists
    the sets, what each covers and how two agree: the same answers, sharing one, Cohen's kappa, Krippendorff's
    alpha and (--confusion) the confusion matrix; free-text labels are not compared as text (an MCP tool).
    V19 checks every set.
  • A label can accept more than one answer, when several are right for an item: {"accept": ["escalate", "file"]}, sorted and without repeats, made by the sheet's also right column, a rule's label: [file, escalate] or a folder's text, shown "escalate or file". A decision is wrong against a label when it is none of
    them; two labels agree only when they accept the same answers. An adapter's score reads a reference with
    answers(reference) (evaluation_harness.adapter); the bundled one keeps the best overlap. The triage gold's
    23 file-or-escalate labels (and their reference) are written so now, by labels_from_verdicts.py byte for byte:
    every one of the 18,460 item scores of the 284 runs against it is the same replayed old and new. The outcome
    rules give the same two answers, so gold and the rules accept the same answers on 34 of the 34 items both label
    (kappa 1.00).
  • evh reference-promote makes a reference set of a label set, at the tier you choose, with a reason and who promoted
    it; the manifest records where it came from (for a model's set, the run and model that wrote it). No rule ties
    the tier to the source. Every existing reference set records how it was promoted (20 here; 6 copied byte for byte
    into label sets, 209 labels), every run's fingerprint unchanged.
  • Scoring against a label set without promoting it: evh rescore --labels SET, evh run --labels SET. The run records reference_id: labels/<dataset>/<set> and lands in data/runs-labels-<dataset>-<set>/, an
    arm of its own; a goal groups such arms under the set, the arm that made a model's labels flagged.
  • The catalog: a tier is a pill in its own colour, a label set a source chip, wherever either is named; a
    Labels page; a dataset's label sets and how often two agree; where each reference set was promoted from.
• Evaluating what production did (RFC-023, PRD-014, ADR-0023). A capture's decisions are evaluated against
  labels; the governed datasets grow only by a version you make:
  • evh arm-from-capture makes production's recorded decisions an arm (provider: recorded): each output the
    capture's record of the same id, read by the adapter's recorded_output(record) (else --field), scored,
    tested and compared like any arm, cost 0. The catalog badges it recorded, its cost and speed "recorded, no
    call".
  • Agreement: the adapter's same(a, b) says when two outputs are one decision (default: the same text;
    the triage adapter compares dispositions). evh inspect --disagree and the inspector's "only where they
    disagree" keep the items where the arms decided differently; label-sheet --items disagree reads the same hook.
  • The sheet, evh inspect --sheet --dataset-id D --runs A,B,… and the inspector's 05 sheet (the dataset
    page's "Open its sheet"): the items down, a column per arm across with what it wrote and every score it has
    (its run's metrics, and a judging pass's where one covered the run; --metric M or SCORER/M chooses the one it
    is read on; --scores-only leaves the outputs out), every label set and reference set, the winner per item (on
    that metric, and per pick or pairwise judge), the signals wrong (--wrong-against SET), disagree,
    split, new (--new-vs D: a field value it never had), unlabelled, each a filter beside --where and
    --by, and a footer per arm: its mean, its win rate and its verdict (the goal's decision, else the paired test
    against the leader). On a pick or pairwise verdict only an arm the judge preferred wins a row; a row it gave to
    an arm not shown is nobody's win. The item column stays in place as the arms scroll; the rows you tick give the evh pick
    and evh label-sheet commands. --csv FILE writes it (UTF-8 with the mark Excel needs), and the inspector
    downloads the same (/api/sheet, &format=csv). Local only: a public server refuses it.
  • A sheet to fill, and evh sheet-import (make sheet-import): --csv FILE --fill label adds label, also right and note (--choices); --fill pick or scores writes the outputs as blind columns A, B, C… shuffled
    per row (--seed), a pick or a score: X column per output (--scale LO..HI), the rubric as the first row
    (#rubric; --rubric FILE) and its key beside the sheet (sheet.key.json). evh sheet-import FILE --dataset-id D --by WHO reads back what was filled: labels as a label set (--set, --kind person|model), picks or scores
    as judged runs under data/runs-judged/<scorer>/ (--scorer), judge_won_unvalidated or
    judge_score_unvalidated per arm per item, through the key. A row or text the key does not have, a letter not on
    its row, a score off the scale and a name already used are refused, with nothing written.
  • Picks: evh pick marks a production item for the next version of one or more governed datasets (targets,
    tags, a note) in data/picks.jsonl, append-only; evh picks lists what waits (an MCP tool), and the target
    dataset's page shows them with a box per row. evh dataset-extend V1 --dataset-id V2 --picks (or --pick)
    makes V2 of V1's items and the picks chosen, each checked against the bytes picked, extended_from recorded;
    labels follow the items; nothing is promoted.
  • evh deploy-sheet --config ARM [--capture C]: what production must match to run what you promoted (the
    model, its record and deployment, every setting sent and a record's setting that was not, what each item
    reported, the adapter), and with a capture, whether each record says production ran it (the adapter's
    production_ran(record)); exit 1 unless every record matches. On the promoted arm's page, and folded beside a
    goal's baseline. The triage importer keeps the ledger's prompt_ver and prompt_sha.
• Several projects on one machine (ADR-0024): catalog-serve and mcp --http take a port of the project's own
  (18000–18999, 19000–19999), the same every time, the next free one if taken; a local catalog names its project;
  every command notes a stray EVAL_HOME; a model record's lock is keyed by the record.
• Datasets from production: the capture lane (RFC-019, PRD-014, ADR-0019). A project keeps what a
  production system did as a capture, then makes datasets and labels from it:
  • evh capture reads records through an importer (--importer jsonl|tsv|github|langfuse, or a
    project's own path.py:Class) into data/captures/<id>/: each record's id, body, fields, time, source
    and outcome, hashed one by one (capture.json); --since-capture continues from an earlier capture.
    evh importer-check tests an importer against the contract, including that it keeps to since and
    until; evh captures lists them, and the catalog's Captures page shows them with the datasets made
    from each. A capture keeps the window it records: a record outside it is named and nothing is written
    (--late keeps the ones older than since that the earlier capture never saw). Newer outcomes of the same
    records are a fresh capture, labelled with label-from-capture --capture (tutorial 6 §5).
  • evh capture-scrub runs a project's own scrubber over a capture's records and re-hashes the ones
    it changed, keeping their earlier hashes.
  • **...

evaluation-harness v0.5.0

Two things to know before upgrading: the package now needs Python 3.10 or later, and it depends on the
official MCP SDK (mcp>=2.3,<3). On 3.9, stay on v0.4.4. What you get: evh mcp, the project's read
commands as tools for any agent (stdio on your machine, or --http for a team or the public), and a dataset page
that shows what is inside a dataset without republishing its text. UPGRADING has the steps.

Added

• evh mcp: the project's read commands as MCP tools, for an agent (stdio, the official SDK,
  RFC-018). Thirteen
  tools: overview, datasets, dataset-items, arms, goals, references, metrics, activity,
  search, inspect, rounds, runs-list, validate. Each call runs the command as a child process, so a tool
  answers what the command prints; each is annotated read-only; no tool writes project data, spends or runs a
  model (the one file a call writes is the p cache in the project's ignored .cache/, below). For Claude
  Code, in the project: claude mcp add evh -- evh mcp.

• evh mcp --http: the same server for a team, or for anyone. Streamable HTTP, stateless, JSON responses:
  clients on every protocol revision since 2025-03-26 call tools without a session. The Host is checked against DNS
  rebinding (--allowed-host is required to bind beyond the loopback); --public leaves out the options that
  answer with an item's text, and the validate tool; --rate, --concurrent and --timeout bound the load; a
  missing required argument is refused before anything runs; --hooks takes a deployment's error reporting,
  tracing, metrics and analytics, in the inspector's shape. Each call is logged as one JSON line: the tool, its
  arguments' names (never their values), how it ended, its time; never the caller's address or user agent.
  --rate counts calls a minute per caller (an IPv6 caller by its /64), --rate-total for all callers together;
  over HTTP a caller has one call running at a time.

• EVH_P_CACHE: the goals' paired tests kept on disk. Every evh command is a fresh process, so a view that
  shows the goals drew the same tests again each time (about 50 s on this repository's runs). With the variable
  naming a file, each estimated p is stored and read back exactly; evh mcp sets it to the project's ignored
  .cache/. goals, overview and arms answer in under two seconds after the first.

• This repository's MCP server, https://evals.humancopilot.eu/mcp (the deployment, not the package): the six
  worked examples for any agent, read-only, without sign-in.

• --json on the catalog's view commands (overview, datasets, arms, goals, references, metrics,
  activity, search, inspect): one object, {"command", "version", "result": {"blocks": [...]}}, each table as
  columns and rows of values, every other line as text. It is collected from the same calls that print the text, so
  the two cannot disagree. 0.7.0 gives each command its own result schema and retires this generic form: do not
  build on blocks.

• What is inside a dataset, on its page and as a command. evh dataset-profile --dataset-id D,
  run where the dataset is materialized, writes data/profiles/D.json (schemas/profile.schema.json,
  checked by validate V1): the number of items and their words, the shape of each gold reference
  (labels and how many of each, entries per item, a record's fields, or words per reference), and
  per item its length, its gold and, when known, where it can be read at its source. A gold of labels
  (a few short values that repeat) is listed per item; any other gold is summarised, never listed. It
  holds no item text, which a dataset's licence may not let you redistribute, unless written with
  --with-text, which is refused unless git has committed every item's file (a project's own synthetic data). Commit the profile and the dataset page shows "what's
  inside" wherever the catalog is built; evh dataset-items --dataset-id D [--item ID] prints the
  same, and an item's text when it is materialized here.

• An item's link to its source comes from the dataset: an item's source_url, or a
  data/sources/<dataset_id>.rows.json that a fetch or import step writes ({"source", "item_url", "rows": {item_id: {"row", "sha256"}}}), matched by the item's bytes, so an id that recurs in
  another dataset never takes its link. This repository's Hugging Face fetchers write one, so each
  item opens at its row in the dataset viewer.

• inspect.html?…&item=ID opens that item once the comparison has loaded; a dataset page's
  inspect link uses it.

• Adapter: dataset_profile(dataset), optional (RFC-003).

• Every catalog page names its command, in its footer (evh goals --goal G, the inspector's from the arms
  picked). evh catalog --mcp URL names the same as an MCP tool call, with the server and how to connect
  (--mcp-howto); this repository's catalog passes its /mcp, and its header links to the site's MCP section.

• evh dataset-profile --json and evh dataset-items --json print the profile inside the same envelope as
  the views, {"command", "version", "result": <the profile>}; the Makefile targets dataset-profile and
  dataset-items.

Changed

• Python 3.10 or later. The official MCP SDK needs it, and every feature comes with one install, so the
  package's floor moves from 3.9 to 3.10 (CI tests 3.10, 3.11 and 3.14). On 3.9, pin v0.4.4.
• A new dependency: mcp>=2.3,<3, which brings anyio, httpx2, mcp-types, opentelemetry-api,
  pydantic>=2.12, pyjwt[crypto], python-multipart, sse-starlette, starlette, typing-extensions,
  typing-inspection and uvicorn. No result moves.
• A dataset id is a name, never a path. Every command that reads or writes data/…/<id>.json refuses an id
  that is not letters, digits, ., _ and - (or that holds ..); before, ../x read or wrote outside data/.
• A param shown on a catalog page or by evh arms --arm hides a URL's user and password and a nested key
  named like a credential (headers: {Authorization: …}), as it already hid a top-level api_key.

Security

From two independent reviews before this release, each reproduced before it was fixed:

• The inspector served any .txt under its data tree through /api/item?item=../../… (an item id was never
  checked; on main since 0.4.0). An item id is now a file's name, refused otherwise, in the server and the adapter.
• The inspector's answers were cached by count, not size: a hundred distinct 40-arm questions took a 256 MiB
  container past its limit, and --warm alone did on this repository's data. The cache is bounded by the bytes it
  keeps (32 MB), keeps no refusals, takes no question over 4 KB, and --warm stops when it is three quarters full;
  the deployment's inspector has 384 MB (it peaked at 228 of 256 under load). Its rate key is an address, never a header's
  text.
• evh mcp --http: a few parallel 4 MiB bodies killed the server; held GETs and ping floods cost memory no rate
  saw. Now --max-body, --max-connections, 8 open requests per caller, a 10 s deadline on a request's body
  (eight callers holding slow bodies took every connection), --request-rate, GET is 405, refusals do not count toward
  --rate-total, and --public answers are kept and their errors carry no paths. Values with NUL bytes or over 512
  characters are refused.
• The deployment: Caddy refuses a body over 64 KB on /mcp (and its handle line now adapts); nginx limits
  each visitor (20 requests a second, a burst of 200, 32 connections); the deploy checks the inspector is up and
  under 80% of its memory once --warm is done, and rolls back otherwise; the MCP hooks no longer export the
  SDK's own spans; the containers'
  /tmp caps files as well as bytes; render.sh's check for reference texts in a release could pass with 1,657 of
  them (a pipe's SIGPIPE) and refused the five tracked ones: it now compares against what git tracks; a deploy keeps
  two releases (the live one and the rollback), prunes at its start as well as its end, and refuses to start short of
  2 GB or 300,000 inodes; the upload hard-links unchanged files again (no file ever linked: each release took ~69k new
  inodes).

Fixed

• The deployment's release ships data/models.yaml (render.sh, and the item fields and profiles with it):
  from 0.4.2 it was never copied, so the deployed arms page called every model's weights unknown.
  tests/test_render_data.py fails when a part of data/ the adapter reads is not copied.
• The CNN/DailyMail and SummEval fetchers deleted the committed gold manifest.json on every refetch; they
  now clear only the item files.
• --hooks files with a module-level @dataclass failed to load (the module was not registered before it
  ran); they load as any other module now.
• A page links only to http(s) URLs: a source_url or rows map that held javascript:… was written into
  an href.
• The p cache reads an unreadable or non-UTF-8 file as no cache instead of failing the command, starts a new
  line after a torn one, and keys each p by how it was computed.
• evh dataset-profile --with-text counts a file as tracked only once it is committed, not when it is
  merely staged.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.5.0#subdirectory=harness"

evaluation-harness v0.4.4

The catalog says what it is and links out: --about puts a deployment's own paragraph on the
overview, a goal's links are shown, and --repo turns project paths into links. The arms page
now carries the deployment's options like every other page. No result moves.

Added

• evh catalog --about FILE: a deployment's own paragraph on the overview, under its
  description. It says which project the catalog shows and where its findings are. None by
  default.
• A goal's links on its page and in evh goals --goal G. The field was in the goal
  schema but shown nowhere. A web address opens; a path in the project is named as text, because
  the static pages cannot know where the project is hosted.
• evh catalog --repo URL: where the project's files can be read (e.g. a GitHub blob/main/
  address). A project path in a goal's links, or at the start of a guardrail's source, then links
  there. Without it, the path stays text.

Changed

• The overview says what the catalog is: "pages the harness builds from this project's
  committed runs", and that the inspector tests any arms against a baseline, item by item.

Fixed

• evh catalog built the arms page without the deployment's options. Run as a command, the
  builder is __main__, and the arms page was rendered by a second copy of it with the
  defaults. So arms.html said "eval catalog" whatever --brand was, and had no --link
  switcher, no --head and the default --api. Every page now carries them.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.4.4#subdirectory=harness"

evaluation-harness v0.4.3

A security fix to the inspector: evh catalog-serve no longer serves files through /api/.
Upgrade if you run it where anyone else can reach it. No result moves.

Fixed

• Nothing is read through /api/. A path under /api/ skipped the server's path checks
  and fell through to the file server, so /api/../.env, encoded or not, returned files from
  the project root, keys included. This happened in local and in --public mode. Now a ..
  anywhere in a path is refused first, and only the API's three endpoints (/api/health,
  /api/compare, /api/item) answer under /api/. Any other /api/ path is 404. In
  --public mode no path outside the API serves a file.
• --hooks see one route for every unknown /api/ path. It is named /api/other, so
  someone asking for many made-up paths cannot create a metric series for each.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.4.3#subdirectory=harness"

evaluation-harness v0.4.2

A catalog that says what changed, and where: every count on the home page has a trend and opens a
page that shows the rows behind it, for the same period. An activity page and an arms and configs
page. No result moves.

Added

• A period on every page, counted from today. Four chips (14 days, 30 days, this month, all
  time) and a list with the rest (7 and 90 days, this and last week, last month, this year); the
  browser works it out from the viewer's date, so a catalog built days ago still shows this week.
  A trend compares with the same days of the period before. The period travels in the link
  (datasets.html#p30d). On the goals, datasets, arms, references and metrics pages a row made
  in the period has a new badge, chips keep only the new or the active rows, and in-period
  columns give the runs and spend with their trend (▲ 12%, ▼ 31%, ▲ from 0) and a line of runs
  per day. The goals page lists what happened to each goal (baselines, rounds).
• The activity page (activity.html, evh activity): the home page's charts at full width,
  and the period's runs, spend and tokens by dataset, by arm and by kind of call, the busiest
  first. The measured runs, billed and tokens tiles open it.
• The arms and configs page (arms.html): every arm, saying how it was measured (measured,
  rescored against a reference, judged by a scorer, re-parsed, run again), grouped by dataset,
  config, model, provider or what it is scored against, filtered to local or hosted arms or a
  price tier, with its role in each goal's decision; tick arms on one dataset to open them in the
  inspector. Every config across the datasets it ran on, with its rank on each goal. What changed
  for arms in the period, and each goal's trade-off charts. The arms tile opens it; the configs
  tile opens it grouped by config.
• --period P on every list: evh datasets, goals, references, metrics, arms print
  the page's period columns. evh arms lists the arms page's rows (--by, --where, --tier)
  and --configs the configs. Each page's period numbers come from one builder the page and the
  command both read.
• Adapter: dataset_created(dataset), optional: the day a dataset counts as new (else its
  first run). spend() events may name a measured run's arm, which the arms and activity pages
  bill it to; HarnessAdapter does.
• Grouping on the datasets page (by goal) and the metrics page (by role: decides, weighs,
  describes), as the arms page groups; one script (group.js) groups and filters all three.
• The arms page filters on quality and on weights: ties the leader or below it (the arm's place
  in a goal's decision), and open weights or proprietary. evh arms --quality ties|below and
  --weights open|proprietary list the same.
• data/models.yaml (optional, schemas/models.schema.json, validate V1): each model's
  weights, open or proprietary, and licence, keyed by the model a run records. The adapter's
  optional model_facts() reads it.
• A derived arm says how far it moved from the config's measured arm on the same dataset
  (rescored, re-parsed, run again: "0.712 → 0.745 (+0.033) vs measured", in evh arms too).
• An arm a goal leaves out for a limit it breaks is badged "over a limit", not "not eligible".
• evh catalog-serve --hooks FILE.py|MODULE: a deployment's error reporting, tracing or
  metrics, on a generic hook point (start, request, error, extra routes such as
  /metrics). The harness ships no tracker and depends on none; without --hooks nothing
  changes. This repository's deployment uses it (infra/inspector/hooks.py).

Changed

• The home page's tiles carry trends. Measured runs, billed and tokens show the period's
  total, ▲/▼ % against the period before and a line per day; goals, datasets, configs, arms,
  references and metrics show the total and how many are new. The tiles are grouped and coloured
  by subject (goals; data; experiments; work and spend), with a legend; never red or green for a
  direction. The charts and the feed follow the period.
• The menu is grouped by the same subjects: Overview, Goals, Data (Datasets, References),
  Experiments (Arms, Configs, Metrics), Activity, Inspector; a phone's menu lists every page.
• evh overview --period P prints the tiles and trends; --all is gone (--period all).
• The activity page and evh activity list what moved most first against the period before
  (up or down), then the busiest; for all time, the busiest.
• "What changed in the period" on the arms page folds three or more arms of one dataset
  re-measured the same way on one day into one line ("12 arms on <dataset>").
• A period select never makes a page wider than a phone, however long its options.

Fixed

• evh bootstrap-test and this repository's report tools refuse when the gold references are
  not on the machine (the repository commits their checksums only): they name the fetch step.
  The extraction and retrieval reports printed tables of zeros instead, as if they were results,
  and the classification report died with a traceback.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.4.2#subdirectory=harness"

evaluation-harness v0.4.1

A catalog that works on a phone and sits inside a deployment's own site, and an inspector that
no longer runs out of memory. No result moves.

Added

• evh catalog --head FILE: HTML added to every page's <head>, for a deployment's own
  analytics or tags. The harness adds none of its own.
• evh catalog --api PATH: where the inspector's API answers, for a catalog served under a
  path (default /api/; e.g. /catalog/api/).
• evh catalog --brand NAME: the product's name beside the logo (default "eval catalog").

Changed

• --link LABEL=URL is a switcher between the parts of a publication, not extra nav entries:
  the links head every page in order, the catalog marked (LABEL=. places it; without one it
  comes first as Catalog), and the logo opens the first. The catalog's own pages stay in its nav
  (a ☰ menu at the left of the bar on phones), and its front page is called Overview. The search
  box reads "Search ( / )", and on a phone it is a round search button in the bar that opens it below; a long
  count on the overview shows in short form on a phone (19.5M). The bar's buttons are SVG icons.
• Tables on a phone: a cell breaks between words, never inside a name, so a narrow screen
  scrolls a wide table sideways; running text keeps a readable width; a long item id is
  shortened in the inspector's per-item table (the whole id is its title); no catalog or
  inspector page is wider than the screen.
• No inline styles: the charts' legend colours are classes, so a deployment's content
  security policy can forbid style attributes.

Fixed

• The inspector no longer runs out of memory. The paired test kept the signs it draws for
  every item count it met (~31 MB each), so --warm on a large catalog grew past a 256 MiB
  container limit and the server was killed every few minutes. At most two sign tables are kept
  now, stored as bytes, and at most 4096 estimates; a warmed inspector settles at ~150 MB. The
  p-values are the same, bit for bit.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.4.1#subdirectory=harness"