evaluation-harness v0.3.0

Re-measure what you already have. Stored outputs can now be scored again by LLM judges, as an
optional pass that never touches a committed run: evh judge-score, for any text task you
declare (what an item is, what the outputs are, the dimensions, the scale). The harness refuses
a judge until the scorer names, for each judge, a committed run that measured it against human
ratings; it runs at least two judges from different vendors blind, measures their position bias
and noise, and stores every verdict so a rerun costs $0. The tutorial has an offline, $0
judging step (GUIDE step 10). Upgrading from 0.2.0 changes no committed run or score and adds
no dependency. Design: ADR-0001;
what it found here: the judging page.

Added

• evh judge-score (and make judge-score). Re-measure an experiment's frozen outputs
  with a validated LLM judge, billed, cost-capped and resumable. It runs from the installed
  package in a project that has only its own data directory, scorer, prompt and runs: the
  prompt builder, reply parser, tau-b and Spearman, and an OpenAI-compatible judge client
  (LiteLLM proxy, OpenRouter direct, per-judge base_url / api_key_env, billed cost read
  from the response, retries) are in evaluation_harness.judging, with no dependency on
  examples/.
• The judge scorer artifact (data/scorers/<id>.yaml): two or more judges from different
  vendor families, the task, the prompt (with its sha256), the protocol, a validation
  block and a cost cap.
• Any text task, declared in the scorer. An optional task block names what each item is
  (source_label), what the arms wrote (output_label), the dimensions the judge scores
  and the scale; a prompt writes {n} where the number of outputs per call goes. Without a
  task block a scorer judges SummEval's news-summary task (ARTICLE, SUMMARIES, four
  dimensions, 1-5), byte for byte as before. A declared task is hashed into the judging set.
• A judging step in the tutorial (docs/GUIDE.md step 10):
  harness/tutorial/scorers/tickets-judge-v1.yaml re-measures the tutorial's arms with two
  fake judges, offline and $0,
  through the refusal, the dry run, the pass and the leaderboard.
• A validation gate. A scorer runs only if its validation block names a committed run
  for each judge whose metric equals the stated value (and is at least min_metric, when the
  scorer sets one) and whose model, temperature, reasoning, provider, max tokens, prompt file
  and prompt sha256 are the judge's. The gate checks that a validation exists and is current,
  not that it transfers to your outputs. Otherwise --unvalidated runs it and labels every number
  "unvalidated judge". provider: fake judges (offline, free, for testing a scorer) can never
  pass the gate.
• Judging sets and stored verdicts. One item is the article plus every arm's output, arm
  identities hidden, order shuffled by a seed; the set is hashed into its id and frozen in
  data/judging-sets/. Every reply is stored per call in data/judge-verdicts/, with its
  scores, so a rerun resumes and re-reading costs $0; a reply that is not bare JSON scores, or
  repeats 8 consecutive words of the item or an output, is kept only as its hash, since a
  judge that reasons aloud can copy the source. A different arm set is a new set, never an
  append. A block retried with --retry-unparsed keeps the earlier calls' cost in its record.
• Controls. A position check (a second shuffle that re-forms the blocks) and a noise check
  (a repeat on a subset), plus per-judge agreement and a self-family flag, in summary.json.
• Adjudicated gold re-measurement (repository tooling, examples/_shared/adjudicate.py,
  not in the package): a committed adjudication file overwrites only the disputed items of a
  local gold reference set to make a new reference version, and rescore re-scores every run
  against it for $0, the old reference kept beside it.
• Judge validation on published human ratings (repository tooling, not in the package):
  validate_judge_goyal.py (Goyal, Li & Durrett 2022) and validate_judge_zhang.py (Zhang et
  al. 2024, crowd ratings of 19 LLM-era summarisers) check a scorer's judges against people.
  The ratings are downloaded when the check runs, pinned by sha256, and never committed; the
  committed records hold article ids, the judges' replies and scores, and hashes; a reply that
  is not bare JSON scores, or repeats 8 consecutive words of the source, is kept only as its hash.
• validate V11-V13. V11: every judging set's stored hash is the hash of its content.
  V12: every stored verdict names a real set, item, pass and block, and its n is that
  block's size, and every committed judge reply (verdicts and validation records) is bare
  JSON or kept only as its hash. V13: every judge scorer loads, is structurally valid (its task included),
  and its prompt file resolves and renders. They read the configured data directory.
• judge-score judges every arm on the dataset by default. A scorer's arms_match,
  --match and --arms narrow it, and it warns when the arm set mixes config versions (_v1
  and _v2 of one arm are judged as two arms).
• A scorer's prompt.file resolves against the project root, then the scorer file's own
  directory, then (in a checkout of this repository only) the repository root.
• Judged runs carry no reference_tier, since a judge reads no reference: the source
  run's tier is kept as judged_with.source_reference_tier, and leaderboard says the scores
  come from a judge, with its label.
• A refused judge call stops the pass cleanly. A call refused, or still failing after its
  retries (a provider's own budget guardrail answering 403, for example), stops the pass the
  way the cost cap does: every stored verdict is kept, the partial result is projected and
  labelled, and the same command resumes.
• Scorer checks. task.scale must be whole numbers, and a prompt with both {n} and
  SummEval's fixed count phrases is refused (the phrases would keep saying 16). V13 renders
  each prompt at 2 outputs, so a prompt that cannot be rendered fails there, not at run time.
• Experiment-level results this enables are reported in the repository's reports (the
  summarisation experiment re-measured with two pairs of SummEval-validated judges, not confirmed by people on those outputs; AG News and DBpedia
  against adjudicated gold); none of them is in the package.

Changed

• leaderboard's precision note no longer fires on values that are exact small-integer
  fractions (a mean of 1-5 ratings over two judges and four dimensions is in eighths); scores
  rounded on the way to disk still get it.
• Nothing that scores a run changed: committed runs and their recorded hashes are untouched,
  and the committed summarisation judging set rebuilds from its stored verdicts with the same
  id.

Install

pip install "evaluation-harness @ git+https://github.com/chipi/eval-harness@v0.3.0#subdirectory=harness"