v0.1.0

First tagged release. It contains no real provider results: artifacts/leaderboard.json
is still the placeholder ("placeholder": true), because the provider sweep hasn't run. What
0.1.0 marks is the harness itself: golden-set schema and rubric, deterministic graders, the judge
with its bias controls, the statistics module, calibration, regression watch, the triage model,
and the static leaderboard, all under CI. Everything below was the Unreleased section until
this release.

Also in this release: .github/workflows/release.yml (a version tag publishes a GitHub Release
from this file, only if the tagged commit has a green ci run), a README screenshot of the
placeholder leaderboard, and SECURITY.md now names GitHub private vulnerability reporting as
the only route (the placeholder email address is gone).

Added

• 2026-10-28: docs/methodology.md Scope now states the single-annotator limitation: the rubric has one
  author, judge-agreement human labels come from one annotator with no second labeller, and
  inter-rater and intra-rater agreement are unmeasured. Guarded by
  test_methodology_states_the_single_annotator_limitation.
• 2026-10-19: docs/run-schedule.md, a draft plan for the real sweep. Limits in it are desk-research
  figures, not measured; nothing in it is a result.
• 2026-10-19: plan_batches packs a provider's slots into shared daily pools and takes an optional
  per-provider daily token budget, so token-bound providers (e.g. Groq) are no longer over-planned.
  _load_items now loads each item's labelled fields and its corpus document instead of fields=[].
• 2026-10-18: python -m ground_truth.simulate [--scenario default|contested] runs the real sweep,
  judging, grading, agreement and publish code with a simulated provider into the gitignored
  data/simulated/ (never artifacts/). /simulated-example and /judge/simulated-example show it
  through the real views behind a SIMULATED banner; both 404 unless GT_E2E_FIXTURES=1 and are caught
  by the data-gt-fixture build scan. Nothing it produces is a measurement of any model.

Changed

• 2026-10-18: grading writes a new verdict with a plain INSERT after its existence lookup, not
  INSERT OR IGNORE. The latter cost 6-10 ms a row against ~1 ms and was 99% of a grading pass
  (30 simulated items: 65-197 s before, 13.6 s after; 60 items: 273-537 s before, 26.8 s after, and now
  linear). Same verdicts, same conflict rules; a re-grade of an unchanged store writes nothing.

• 2026-10-18: emit_all now writes distinguishability (the plain-language statement for every
  pair of overlapping intervals) and methodology_url into leaderboard.json. The leaderboard page
  reads both; only the hand-written placeholder had them, so the first real publish would have
  crashed the page build. Additive: no published number changes.

• 2026-10-16 (JP-9, JP-10): the completeness check in matrix/build.py now reports a field
  type as held back only when it is one of the judged types in play. Before, a field name shared
  by two items under different types could list the other item's type as "judging incomplete".
  Judge targets are now selected once per field name per headline selection instead of twice. No
  published number changes: neither affects a verdict the headline counts.

• 2026-10-16 -- the leaderboard says which judged field types the headline left out
  (JP-12). A judged field type dropped as a whole (judging incomplete, or not every ranked
  slot's judge accepted on it) used to vanish from the headline with no word to the reader.
  leaderboard.json now carries held_back_field_types, a list of {field_type, reason}
  (matrix.build.held_back_field_types), and the leaderboard page prints it. The entry is a type
  name and a fixed reason string: no item, output, hash or count, since it is derived from the
  private split. A rejected or unassessed judge is not listed; it is already published as its
  status in judge_agreement.json. No headline number changes.

• 2026-10-16 -- a re-grade that disagrees with a stored deterministic verdict now fails loudly
  (JP-2). Verdicts are append-only and keyed by (output, field, grader, grader version, rubric
  version); the key carries no outcome, so a re-grade at the same grader version that reached a
  different answer (an output or a grader changed without a version bump) was dropped silently
  and the old verdict kept counting. grade_stored_outputs now keeps the stored verdict, finishes
  the pass, and raises GradingConflict naming every disagreeing verdict; the new answer is
  recorded by bumping the grader version. Judge verdicts are exempt: a judge verdict is a majority
  over stored samples, so another sample at the same version can change it. No published number
  changes.

• 2026-10-15 -- a re-grade counts once across graders and across schema re-checks. A
  deterministic verdict is superseded by a later re-grade of the same output and field
  whichever deterministic grader wrote it, so a rubric that moves a field type to another
  grader no longer counts both graders' verdicts (superseded only by the same grader); a
  judge verdict is still superseded only by the same judge's. A deterministic verdict counts
  only while its output's latest schema verdict is correct, and the judged weight counts the
  same outputs (any correct schema verdict ever): an output re-graded as unparseable drops
  out of both, and counts again once re-graded as parseable. The judged weight then equals the
  number of verdicts each deterministic field has for the (item, slot) under any re-grade.
  Analysis plan v3 amended in place (no sweep has run). No live sweep exists, so no published
  figure changes.

• 2026-10-15 -- the judged weight counts schema-valid graded outputs only. The judge's one
  verdict per (item, slot) is weighted by the number of schema-valid graded outputs of that
  (item, slot), which is the number of verdicts a deterministic field has for it
  (the number of replicate outputs stored); an output that failed schema validation or has
  not been graded counts for neither, and the weight is at least 1. Analysis plan v3 amended
  in place (no sweep has run).

• 2026-10-15 -- a judged field counts only once every current target is judged. A judged
  field type counts only when every current judge target for it, across the ranked slots, has a
  verdict; while any is unjudged the field is out for every slot and the headline uses
  deterministic grading, like a rejected judge (a partial run would rank slots on different field
  sets). Inconclusive and not-extracted verdicts count as judged.

• 2026-10-15 -- a re-graded verdict supersedes the old one in the headline. A verdict
  superseded by a later re-grade of the same output, field and grader no longer counts
  (both the old and the re-graded verdict counted); counts_toward_headline is false for it.

• 2026-10-15 -- judge wins on a judged field type (JP-1). Where a judge's verdicts count on
  a field type, the headline scores the field from the judge alone; the field's older
  deterministic verdicts, left behind when a rubric moved the type to the judge, no longer count
  beside it (both counted, so the field was scored twice) and counts_toward_headline is
  false for them. Where the judge does not count (rejected, unassessed, unassigned, held back),
  deterministic grading still applies. This is analysis plan v4, not an edit of v3, which was
  already merged and pushed; v3 governed zero sweeps (SELECT count(*) FROM runs returned 0 on
  2026-10-15, on the local data/results.duckdb only).

• 2026-10-15 -- open items left by the judge-publish merge are logged in
  docs/process/reviews/2026-10-15-judge-publish-open-items.md (JP-1 to JP-12). None is
  fixed. JP-1 (a field moved to the judge still counts its old deterministic verdicts) can
  affect published numbers.

• 2026-10-14 -- the LLM judge is wired in: analysis plan v3, judge decisions per (field type,
  judge). docs/analysis-plan.md is now plan_version: v3 (threshold 0.667 unchanged; v2's
  judge paragraph kept below). v2 governed zero sweeps (SELECT count(*) FROM runs on
  data/results.duckdb returned 0 on 2026-10-14).

  • models.yaml declares each slot's judge and it is validated, including family separation.
    Decided lineup: Gemma judges slots A-E, gpt-oss-120b judges the Gemini slot, both on
    Cerebras; Gemini is judged by a different model from the other five (disclosed
    comparability limit). The real models.yaml is not yet filled.
  • run_judging samples each slot's judge three times on the modal output per (item, slot);
    grade scores LINE_ITEM_DESCRIPTION from the majority vote (grader_id judge:<slot>).
    No majority is INCONCLUSIVE, which scores as incorrect.
  • Decisions are recorded per (field type, judge). A judge with no decision is not scored in
    the headline (an unassessed judge was scored); its verdicts are still published. Today
    every judge is undecided: record_decision has no production caller and there are no human
    labels, so LINE_ITEM_DESCRIPTION is reported as not measured in the headline.
  • Headline counts a judge verdict only from the slot's currently assigned judge, with an
    accepted decision for that field type, on the current modal target. Every published verdict
    carries counts_toward_headline; judge_agreement.json names each judge and gains an
    inconclusive list (count and n per slot and judge, over both splits and all judge
    verdicts, not the headline's judged share). The web judge table shows per-judge agreement
    and inconclusive counts.
  • Final-review rulings, added to v3 before merge: a judged field type counts toward any
    slot's headline only when every slot has an assigned judge and every slot's currently
    assigned judge holds an accepted decision for it, otherwise it is ...