evaluation-harness v0.6.0

What you get: datasets made from what production did (capture, scrub, split, dataset-from-capture); labels that say
who decided what is right, and reference sets promoted from them; production's own decisions scored as an arm;
models onboarded before they are run; scoring that evolves without re-running an arm; judges that grade, pick the
best or compare two; and one sheet of every item and arm, filled in Excel or another tool and read back. Upgrading
moves no committed score. Two formats (fingerprint v5, model records) and the old references are migrated by a
command, and a few names change to the field's words with no alias (evh run, significant, contenders):
UPGRADING has the steps.

Added

• Labels, and the reference sets promoted from them (RFC-023, PRD-002, ADR-0022). A label says what is right
  for one item and who said it: a person, a model or a rule. Labels come in sets, never edited, under
  data/labels/<dataset>/<set>/, and follow an item by its sha256 into any dataset holding the same bytes.
  • evh label-from-run: a finished run's outputs as a model's labels, no calls, the refusals a reference's
    provenance needs (a dirty or unknown build, an unfinished or derived run, an empty output, one cut off at
    max_tokens unless --allow-cut-off). --not-redistributed keeps the texts out of git (a .gitignore beside
    them) and records the command that rebuilds them, checked against the set's checksums.
  • evh label-sheet writes a sheet of the items to label (unlabelled, all, where runs disagree, or chosen),
    each run's decision and a proposal beside an empty label; evh label-import reads it back, or a folder of
    <item_id>.txt, as a person's set (yes takes the proposal; notes kept). The sheet is CSV that Excel opens
    and saves (UTF-8 with its mark, , or ;); --choices puts the allowed answers in the label column's
    header and the import refuses any other; the local inspector downloads it. evh label-from-capture turns a
    capture's recorded outcomes into a rule set by the rules reference-from-outcomes read. evh labels lists
    the sets, what each covers and how two agree: the same answers, sharing one, Cohen's kappa, Krippendorff's
    alpha and (--confusion) the confusion matrix; free-text labels are not compared as text (an MCP tool).
    V19 checks every set.
  • A label can accept more than one answer, when several are right for an item: {"accept": ["escalate", "file"]}, sorted and without repeats, made by the sheet's also right column, a rule's label: [file, escalate] or a folder's text, shown "escalate or file". A decision is wrong against a label when it is none of
    them; two labels agree only when they accept the same answers. An adapter's score reads a reference with
    answers(reference) (evaluation_harness.adapter); the bundled one keeps the best overlap. The triage gold's
    23 file-or-escalate labels (and their reference) are written so now, by labels_from_verdicts.py byte for byte:
    every one of the 18,460 item scores of the 284 runs against it is the same replayed old and new. The outcome
    rules give the same two answers, so gold and the rules accept the same answers on 34 of the 34 items both label
    (kappa 1.00).
  • evh reference-promote makes a reference set of a label set, at the tier you choose, with a reason and who promoted
    it; the manifest records where it came from (for a model's set, the run and model that wrote it). No rule ties
    the tier to the source. Every existing reference set records how it was promoted (20 here; 6 copied byte for byte
    into label sets, 209 labels), every run's fingerprint unchanged.
  • Scoring against a label set without promoting it: evh rescore --labels SET, evh run --labels SET. The run records reference_id: labels/<dataset>/<set> and lands in data/runs-labels-<dataset>-<set>/, an
    arm of its own; a goal groups such arms under the set, the arm that made a model's labels flagged.
  • The catalog: a tier is a pill in its own colour, a label set a source chip, wherever either is named; a
    Labels page; a dataset's label sets and how often two agree; where each reference set was promoted from.
• Evaluating what production did (RFC-023, PRD-014, ADR-0023). A capture's decisions are evaluated against
  labels; the governed datasets grow only by a version you make:
  • evh arm-from-capture makes production's recorded decisions an arm (provider: recorded): each output the
    capture's record of the same id, read by the adapter's recorded_output(record) (else --field), scored,
    tested and compared like any arm, cost 0. The catalog badges it recorded, its cost and speed "recorded, no
    call".
  • Agreement: the adapter's same(a, b) says when two outputs are one decision (default: the same text;
    the triage adapter compares dispositions). evh inspect --disagree and the inspector's "only where they
    disagree" keep the items where the arms decided differently; label-sheet --items disagree reads the same hook.
  • The sheet, evh inspect --sheet --dataset-id D --runs A,B,… and the inspector's 05 sheet (the dataset
    page's "Open its sheet"): the items down, a column per arm across with what it wrote and every score it has
    (its run's metrics, and a judging pass's where one covered the run; --metric M or SCORER/M chooses the one it
    is read on; --scores-only leaves the outputs out), every label set and reference set, the winner per item (on
    that metric, and per pick or pairwise judge), the signals wrong (--wrong-against SET), disagree,
    split, new (--new-vs D: a field value it never had), unlabelled, each a filter beside --where and
    --by, and a footer per arm: its mean, its win rate and its verdict (the goal's decision, else the paired test
    against the leader). On a pick or pairwise verdict only an arm the judge preferred wins a row; a row it gave to
    an arm not shown is nobody's win. The item column stays in place as the arms scroll; the rows you tick give the evh pick
    and evh label-sheet commands. --csv FILE writes it (UTF-8 with the mark Excel needs), and the inspector
    downloads the same (/api/sheet, &format=csv). Local only: a public server refuses it.
  • A sheet to fill, and evh sheet-import (make sheet-import): --csv FILE --fill label adds label, also right and note (--choices); --fill pick or scores writes the outputs as blind columns A, B, C… shuffled
    per row (--seed), a pick or a score: X column per output (--scale LO..HI), the rubric as the first row
    (#rubric; --rubric FILE) and its key beside the sheet (sheet.key.json). evh sheet-import FILE --dataset-id D --by WHO reads back what was filled: labels as a label set (--set, --kind person|model), picks or scores
    as judged runs under data/runs-judged/<scorer>/ (--scorer), judge_won_unvalidated or
    judge_score_unvalidated per arm per item, through the key. A row or text the key does not have, a letter not on
    its row, a score off the scale and a name already used are refused, with nothing written.
  • Picks: evh pick marks a production item for the next version of one or more governed datasets (targets,
    tags, a note) in data/picks.jsonl, append-only; evh picks lists what waits (an MCP tool), and the target
    dataset's page shows them with a box per row. evh dataset-extend V1 --dataset-id V2 --picks (or --pick)
    makes V2 of V1's items and the picks chosen, each checked against the bytes picked, extended_from recorded;
    labels follow the items; nothing is promoted.
  • evh deploy-sheet --config ARM [--capture C]: what production must match to run what you promoted (the
    model, its record and deployment, every setting sent and a record's setting that was not, what each item
    reported, the adapter), and with a capture, whether each record says production ran it (the adapter's
    production_ran(record)); exit 1 unless every record matches. On the promoted arm's page, and folded beside a
    goal's baseline. The triage importer keeps the ledger's prompt_ver and prompt_sha.
• Several projects on one machine (ADR-0024): catalog-serve and mcp --http take a port of the project's own
  (18000–18999, 19000–19999), the same every time, the next free one if taken; a local catalog names its project;
  every command notes a stray EVAL_HOME; a model record's lock is keyed by the record.
• Datasets from production: the capture lane (RFC-019, PRD-014, ADR-0019). A project keeps what a
  production system did as a capture, then makes datasets and labels from it:
  • evh capture reads records through an importer (--importer jsonl|tsv|github|langfuse, or a
    project's own path.py:Class) into data/captures/<id>/: each record's id, body, fields, time, source
    and outcome, hashed one by one (capture.json); --since-capture continues from an earlier capture.
    evh importer-check tests an importer against the contract, including that it keeps to since and
    until; evh captures lists them, and the catalog's Captures page shows them with the datasets made
    from each. A capture keeps the window it records: a record outside it is named and nothing is written
    (--late keeps the ones older than since that the earlier capture never saw). Newer outcomes of the same
    records are a fresh capture, labelled with label-from-capture --capture (tutorial 6 §5).
  • evh capture-scrub runs a project's own scrubber over a capture's records and re-hashes the ones
    it changed, keeping their earlier hashes.
  • evh capture --sample-rate R --seed S keeps a share of the records, by a hash of each id, as they are
    read: the same rule draws the same records again, an importer may sample at the source, and a continued
    capture keeps the rule.
  • evh dataset-from-capture freezes a selection (--where, --dedup, --one-per),
    typed fields from a field set, and --tag K=V labels; only the body becomes the item.
  • What happened afterwards becomes labels by a small rule set (when, unless, exists, absent, range,
    min_age_days): evh label-from-capture, above.
  • evh dataset-split makes <D>_validation and <D>_test by hash (stable across versions, a group
    kept whole with --key) or by time.
  • V17 checks every capture's records against their hashes; a capture whose records a project keeps
    out of git (fetched from where it publishes them) is named as not fetched, not failed.
• text: hidden on a dataset: none of its text, nor anything derived from it, on any public surface:
  the catalog's pages, the inspector and evh mcp in public mode, and a deployment's copy of the data
  (_release.py copy-data, which also refuses a release carrying reference texts or run outputs git does
  not track). dataset-from-capture --hide-text sets it.
• Dataset tags on the views: the catalog's datasets page groups by each tag key and shows where a
  dataset came from (a capture and its selection); evh datasets --tag K=V filters the same way.
• Two examples from a production fleet: examples/triage-fleet/ (an agent's real triage decisions,
  replayed through its own code; the operator's verdicts as gold; the model's own decision scored beside
  the fleet's, before and after its gates) and examples/signal-fleet-dedup/ (a regression guard for the
  fleet's de-duplication functions; no model). Their data is published on Hugging Face and fetched by a
  pinned revision.
• A share card on every catalog page: its title and its opening line as description, og:* and
  twitter:card, and with evh catalog --card URL the image a shared link shows. The pages stay
  noindex.
• Onboarding models, once, for every eval (RFC-020). A project keeps a model record per model,
  data/models/<model_id>.yaml: its aliases, weights and licence, what it can do, the settings that hold
  whatever the task (reasoning_effort; device, precision, batch size for an ML model) with why, its history,
  and its deployments (each place and build it runs: machine or host, server and version, digest, revision
  or hash, the route for a hosted one), each onboarded on its own.
  • evh model-onboard <id> --deployment <d> probes an LLM through the deployment's endpoint: the digest the
    server reports, whether it reasons at the default and with the record's settings, JSON, determinism and
    speed (the deployment's, not the record's); a different build than the record's is flagged and kept in its
    history. An ML model or a baseline through an arm (--config). ONBOARD PASS or ONBOARD CHECK.
  • evh arm-smoke --config <arm> runs any arm on a few items before its full run and writes no run: runs,
    valid output for the scorer, a projection of the full run's time and cost, build pinned, determinism; for an
    LLM every model call (reasoning, slow, long, cut off, no answer, from a provider's _meta or an adapter's
    meta["calls"]; a reply is long past half the arm's reply cap, or with no cap against the arm's own median,
    so neither a summarising arm nor an agent's short probe calls beside its answers make a long reply).
    SMOKE PASS or SMOKE CHECK, written into the record.
  • An arm names a record and a deployment (model_record:, deployment:): the record's settings go under
    its params, which add the task's own (its reply cap, its call limit: a record refuses them, since they would
    reach every other eval's arms); the run records both. V18 checks the records, and the arm configs in data/configs and those validate.json's configs globs name; validate notes an arm never smoked, a
    reasoning model with no max_tokens, a tag that is a different build on two deployments.
  • The catalog's Models page and a page per model, a smoke badge on the arms page, a models tile; evh models for the same (and over evh mcp). A model is vetted when an arm that names it passed its smoke;
    its onboarding alone makes it onboarded.
• Scoring that evolves (RFC-022). A round records the scorer it is judged by, and a scoring change is a
  step of its own:
  • evh score-preview --goal G [--round rN] re-scores the goal's arms (or the round's) with the scorer in
    the working tree, in a temporary directory, and prints each arm before and after and the decision before
    and after; it writes nothing and spends nothing.
  • evh rescore --goal G --round rN --why "…" re-scores a round's arms; the round records the change
    (scoring: when, the scorer, why, where).
  • evh round-open --from rN --keep all --scoring "…" opens the next round judged by the current scorer:
    every carried arm another scorer judged is re-scored first ($0), each one's measure printed before and after.
  • Re-scores land in one place the tool designates, data/runs-rescored/<scorer>/<source>/, read by the
    catalog, the goals, the rounds and validate; --out still names any other. Rounds read an arm from its
    re-score as the goals do, and evh rounds and the goal page show a round's scorer and scoring changes.
  • evh round-open --adopt records a goal's first round from the arms already run for it.
  • A goal axis's limit: baseline: an arm is out when the paired test finds it worse than the group's
    baseline arm on that measure.
  • validate notes a round whose arms one adapter scored in more than one version.
  • evh init's .gitignore ignores the throwaway data/runs-smoke*/ and data/runs-tune*/.
• call_timeout_s, an arm's call limit, for the built-in providers: the SDK client's timeout (its own
  default is 10 minutes). The RUNBOOK told arms to set a call limit only the triage adapter honoured.
• The triage example's adapter takes reasoning_effort and max_tokens per arm, reports every model call
  for arm-smoke, and records a call that fails the same way twice at temperature 0 as the fleet's own "no
  terminal decision" (no_decision, deferred). With reply_in_tool_call: <name> it reads an answer a server
  returns as a tool call to that name, with no text: Ollama 0.40 does that to every gpt-oss answer, and the fleet,
  which reads only the text, got nothing.
• No key for a local server: the openai provider asks for no OPENAI_API_KEY when its address is on this
  machine (localhost, 127.x, ::1) or on Ollama's port, 11434; Ollama takes any key. Any other address still
  needs one, and a key that is set is used.
• Tutorial 7, set up a model before you test it: a model record, onboarding it where it runs, an arm that
  names it, a smoke before the full run; on your own machine with Ollama.
• Judging modes (RFC-005, PRD-003 FR5): a scorer's mode says what a judge is asked: grade (the default,
  as before), pick (the best of an item's outputs, in one call: outputs_per_call must hold every arm) or
  pairwise (the better of two, every pair of arms; outputs_per_call 2). Pick and pairwise project won per arm
  per item (judged runs' primary judge_won) and report each arm's win rate and Bradley-Terry strength with a 95%
  interval from resampling the items, and the position and noise checks as the share of verdicts that held. A judge
  is validated in its mode (validation.mode must be mode). The SummEval example validates gemma_l and
  qwen_s in both (mode: pick, scored top1 beside top1_chance; mode: pairwise, scored pair_agreement
  with n_pairs; a random floor each): picks 0.364 and 0.263 against a random 0.070, pairwise 0.825 and 0.813
  against 0.473; the prompts ask for the JSON first and nothing after it. judge-summ-pick-v1 and
  judge-summ-pairwise-v1 are the summarisation eval's judges in those modes; the pick pass is run (29 arms in
  one pick: the order moved about half the picks, see the SummEval README), the pairwise one not yet.
• An adapter's optional primary_metric(params): it names an arm's primary metric from the arm's params,
  beside PRIMARY_METRIC (its fallback); run and rescore record it. The SummEval adapter's is tau, top1 or
  pair_agreement by mode.

Changed

• evh experiment-run is evh run, and an experiment is an eval: industry words (OpenAI evals,
  inspect_ai, Weave, Langfuse); what Braintrust and LangSmith call an experiment is a run here. The site's
  Experiments page is Evals (evals.html); the old address redirects.

• Dataset roles are validation and test (were dev and heldout): evh dataset-split --test SHARE (was
  --heldout) writes <D>_validation and <D>_test, and a split dataset records split.test and part
  validation or test. A goal's role: says validation, test or pilot. The reports' dev slice is the pilot slice,
  a small slice inside the test slice (not a validation split), and their measurement slice the test slice.
  evh holdout keeps its name.

• The paired test's verdict is significant (was separated): significant: better, significant: worse,
  not significant, everywhere they are printed or shown (evh output, the catalog, the inspector, the site,
  the reports); family_test and bootstrap_test print SIGNIFICANT, pair-test SIGNIFICANT or
  NOT-SIGNIFICANT. A project reading the old verdict strings reads the new ones.

• A goal's arms that may win the tie-break are its contenders (were tied): the decision's key is
  contenders (was tied), evh arms --quality contender|below (was ties|below), the arms page's facet
  "a contender", and the why-lines say "Contenders beside it" and "Still undecided after the tie-break". A tie
  is still what it was in a paired test: an item where two arms score the same.

• A promoted reference is a reference set (the governed collection under data/references/<tier>/<name>/);
  a reference is the expected answer for one item. The catalog's References page is Reference sets. Commands,
  paths and keys (reference-promote, reference_id, reference_tier) keep their names. REFERENCE.md maps
  every term to what inspect_ai, lm-evaluation-harness, HELM, OpenAI evals, Braintrust, LangSmith, Langfuse and
  Hugging Face call it.

• reference-create and reference-from-outcomes are removed (ADR-0022): a reference set is made only by
  promoting labels. A run's outputs are label-from-run's (with the same refusals and --not-redistributed),
  outcomes label-from-capture's (the same rules). A reference set made from outcomes has no rule of its own any more:
  promoted, it is a silver like any (with gold beside it, gold still decides; ADR-0020 is superseded). The one
  reference set here that named reference-create as its rebuild says where its texts come from instead.

• validate fails a reference set with no manifest (V16), where it noted one: texts copied into
  data/references/ by hand are a reference set nobody promoted. Import them as labels where they are and promote
  them: reference-promote adopts the folder when its texts are the labels' (writing its manifest; a folder whose
  texts differ is still never overwritten). Tutorial 1, the newcomer scenario, evh self-test and the tests make
  gold that way.

• evh reference-manifest is a promotion: it takes --labelled-by, --reason and --by (and --kind), and
  records them as promoted_from and the reason, so every reference set says where it came from; the reference schema
  requires both. This repository's reference sets are aligned: the caveats name where their labels came from (no
  "human-authored" or "model-generated" tier wording), the two that named a removed command as their rebuild name
  the 0.6.0 ones, and from_outcomes and cost_usd, which only the removed commands wrote, are gone. No run's
  fingerprint moves (none of these is hashed).

• The sheet shows each arm's decision, the signals first: an adapter's optional decision(output) (the
  triage example's: its disposition) is what an arm's column shows, the output itself otherwise; with no
  same(a, b), two outputs agree when their decisions do. The browser's columns follow the terminal's (UXS-002).
  An adapter's optional title(item) names each item (the triage example's: the signal's alert name, not the
  probes that open it).

• evh inspect --items is gone, and so is --rows; the sheet replaces both (no aliases): evh inspect --sheet --dataset-id D --runs … shows every item's scores per arm and the rows view's signals in one table. The
  inspector's endpoints /api/rows and /api/label-sheet are gone; /api/sheet serves the sheet, and its
  &format=csv the download. The MCP tool inspect offers the sheet's options but not the ones that write a file
  (--csv, --fill and theirs).

• The triage example's write_gold.py is labels_from_verdicts.py: a source's labels from verdicts.jsonl,
  one file per item, for evh label-import; the tier is the promotion's. Its owner labels rebuild the committed
  gold set byte for byte.

• catalog-serve serves on the project's own port, not 8765 (--port overrides; --public keeps 8765), and
  mcp --http on another; a bookmark of 127.0.0.1:8765 moves.

• The release leaves the pick list out: picks are working notes that name production items, so a published
  catalog shows none waiting.

• The tutorial kit's desk export records where the desk's router sent each ticket (routed): tutorial 6 scores
  that decision against what happened, and picks the misrouted tickets for a dataset it governs.

• The goal page explains, then answers, then shows how it got there: what it is for first; then the answer
  so far, led by the latest round's conclusion when one closed after the baseline was promoted (the promotion's
  reason folded under it); then its rounds, one folded section each, headed by its row (round and status, the
  conclusion, what it set out to learn, arms and best), and inside it a heading per part (what changed, arms
  with retirements grouped by reason and the carried ones folded, scoring, evidence, suggested next, notes). The
  rounds table is gone.

• The Models page has one line per cell: the first build and "+k", the deployments' ids, "N goals" with the
  first named, the single best placing with "+k more"; vetted records first. A model's page has one folded
  section per deployment, holding its build and its own onboarding with the flags it accepts and why; its arms
  table names each arm's deployment and reads a re-scored arm as the goals do.

• The datasets table shows each tag as a chip.

• Fingerprint v5 (RFC-021): what each part identifies. instrument (the harness and the adapter's code),
  data, deployment (the system under test as it executed: from the arm's model record and deployment, the
  adapter's hook, or the params; the weights, the server, where it ran; in process only when the arm calls
  nothing elsewhere) and settings (how it was asked: the params that change behaviour, and what a hook declares
  about its prompt), each hashed, and hash over the four. The machine evh ran on is runner, recorded outside
  the hash: the same deployment run from two machines is the same measurement. evh run-compare names the part
  that moved; the inspector's speed warning tells a deployment that differs from a runner that does. Every
  committed run is migrated by evh fingerprint-migrate (idempotent, --dry-run); v4 fingerprints are not kept.

• data/models.yaml is replaced by the model records (data/models/): its 41 entries are migrated one record
  each, and the arms page's open / proprietary filter reads the records.

• CI runs once per merge, not twice: a squash onto main whose tree is its pull request's head,
  tested green and up to date with main, skips the workflows that already passed on it; the deploy
  no longer waits ~13 minutes for a second run of the same files. Anything else runs them again.

• An item id may contain :, @, + and = (a production record's id, e.g. a fingerprint and its
  time), on the inspector's item view too; still never a path.

• A deployment copies its data through _release.py copy-data, not a list in render.sh: the same
  directories, plus each capture's capture.json (never its records), with the text: hidden rules
  applied and the untracked-text refusal checked before anything is written.

Fixed

• Every evh command is a make target again, as the READMEs and the RUNBOOK say: the 22 added in 0.5.0 and
  0.6.0 had none (make labels, make reference-promote ARGS="…", …).
• The home page's "recorded" feed shows the period's events. Its period filter read the date from the table
  cell, not the span inside it, so it hid every event and said "Nothing recorded in this period" (since 0.4.2).
• Coverage counts an arm scored against a label set as labels, on the home page and in evh overview; it
  counted as none.
• The home page's glance reads a goal's baseline from its re-score (RFC-022), as the goal page does: a baseline
  promoted before a re-score showed no value and no test.
• A frontier chart's names stay inside the chart: a long name on a dot in the middle ran past the right edge;
  it goes to the side it fits, shortened when neither does, never over the y axis.
• The label-set and agreement tables keep their promoted and agreement columns on screen on a desktop: long set
  ids may break there (a phone still scrolls the table sideways).
• evh deploy-sheet says "against no reference" for a run scored against none, not None (None).
• The leaderboard's silver note no longer says "model-generated": a silver reference set can be a person's or a
  rule's labels.
• A retired arm read through its re-score is out of the goal's contenders. A round's retirements name the arm's
  config, and an arm read from runs-rescored/<scorer>/ kept the prefix, so it was never matched as retired and
  stayed among the contenders (the triage goal's: 76 arms, 51 once read right).
• Sorting a page's later table sorts it by the column clicked. Its header's index was counted over every sortable
  header on the page, so a second table sorted by a column it does not have and threw.
• evh mcp --public sees only public fields in inspect. The tool runs the command in its own
  process, where the inspector's public rules were never set, so a --where or --by on a field not
  marked public was accepted and its values could be read through the counts (0.5.0). A public
  server now runs its commands as public (EVH_PUBLIC), and such a filter is refused by name. This
  repository's own deployment declares no field sets, so it exposed nothing.
• A command's own refusal through evh mcp (exit 1, its ERROR: line, no traceback, such as a
  dataset id that is a path) is told to --hooks as refused: "command", not as an error: a
  deployment's metrics count it as refused and its analytics send no event for it.
• The site's links to its home page point at /, never index.html (analytics counted the two
  apart, search engines saw two addresses for one page), and each of its pages names its own address
  as <link rel="canonical"> and og:url, checked by site/build.py --check.
• validate checks every git-tracked runs directory, those committed in validate.json with their floors
  and every other one: a directory re-allowed in .gitignore but not added to the list was never checked
  (a set of 65 re-scored triage runs, before RFC-022 gave re-scores one place).
• The local catalog reads what is committed. It read every data/runs* directory on disk, so a smoke
  pass in an ignored runs-smoke* showed as a result locally and was gone from a clone and the published
  catalog. Directories git ignores are no longer read (outside a git checkout everything is). A measured arm
  the goals read from its re-score says so on its page, on the arms page and in evh arms.
• Runs written where git does not keep them are named. EVAL_RUNS_DIR pointed at any
  data/runs-* directory that .gitignore does not re-allow wrote runs that were never committed
  and that no catalog showed, without a word: 35 self-hosted triage runs sat in data/runs-local
  that way. run and rescore now print a WARNING when they write to such a directory
  (throwaway runs-smoke* and runs-tune* stay silent), or outside the project's data, where nothing
  reads them, and evh validate notes one that holds
  finished runs. The RUNBOOK's scratch example is data/runs-smoke, the throwaway name git already
  ignores.
• One row per arm when a scorer changed. A goal listed an arm twice when its runs had been
  re-scored (evh rescore) into a derived set, once per directory, and dropped the measured runs
  whose older score lacked the goal's measure: 37 triage arms were listed twice, 15 only from
  the re-score. A re-score against the same reference now stands in for the run it re-read on the
  goal pages, in evh goals and in the recommendation, which says how many arms it read that way.
  evh goals --goal also prints a dataset's notes, as its page did.
• A deployment the adapter's hook declares keeps the server it was called on. Its server was
  always null, so a self-hosted run whose adapter has a fingerprint() hook lost its Ollama or vLLM
  URL. The 130 triage runs are backfilled with the fleet they replayed (tools/backfill_triage_fleet.py:
  the fleet's code hash and commit f07efa9, the prompt's hash, each checked against what the runs'
  own items recorded) and re-sealed.
• Onboarding a model on several machines at once keeps every result. model-onboard and arm-smoke read the
  record when they started and wrote all of it back when they finished, so the slower erased the other machine's
  result. They re-read it under a lock and write only their own deployment or smoke.
• A LiteLLM address given with /v1 is not given a second one (http://host:4000/v1 became /v1/v1, a 404),
  and a reasoning setting sent through LiteLLM is named in allowed_openai_params: most hosted model groups refused
  reasoning_effort without it (UnsupportedParamsError).
• A record can accept a flag as the model's nature (accepted: {reasoning: "<why>"}): onboarding and arm smokes
  report it as accepted and pass (gpt-oss reasons whatever it is asked). A deployment accepts kinds of its own
  beside the record's: a hosted gateway's build cannot be pinned, and its repeats at temperature 0 vary.
• A round with more than 40 arms is compared. Its evidence asked the inspector for one comparison of every
  arm, which refuses past 40, and the refusal left the rows empty without a word (the triage round, 52 arms). Past
  the inspector's limit the round runs the same paired test on the scores directly; a refusal becomes a note.
• A project module edited within the second is what runs. Adapters and hooks were imported through
  Python's bytecode cache, which trusts a .pyc while the source's modification time (whole seconds) and size
  match: a scorer edited to a value of the same length (BONUS = 0.0 to 0.5) and re-scored at once ran the
  old code while the re-score recorded the new code's hash. They are compiled from the source every time.
• A system that ran in the harness's own process, with no hook, is in process. A deployment taken
  from the params was always in_process: false, so a local baseline (Lead-3, a keyword rule) kept its
  libraries on the runner instead of the deployment. evh fingerprint-migrate brings existing v5
  fingerprints to the rule when it has the run's params (11 runs here, one baseline's hash).
• A run stops when its server cannot be reached. An adapter that turns a failed model call into an
  answer (the triage replay records it as the fleet's "no terminal decision") wrote a complete-looking run
  when a tunnel dropped: 54 and 65 refused items scored as the model's own failures. When every call an
  item reports (meta["calls"]) failed reaching its server (connection refused, no route, a name that does
  not resolve), run stops there with STOPPED, that item unwritten, and --resume calls it
  again. A model's own failure is still a measurement.
• evh pair-test --round reads each arm as the round does: from its latest re-score when one stands in for
  its runs. It read data/runs alone, so a measure an arm's re-score added (production's triage run on
  triage_cost) found "only 0 shared item(s)" while the round's own evidence tested it. "(its leader)" is said
  only when the test chose the leader.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.6.0#subdirectory=harness"