evalhub 0.2.0

Release 0.2.0: runs as first-class rows (#44)

0.2.0 makes an Eval's runs first-class. An Eval is now a versioned header plus a set of runs that belong to the record, not to a version: each run is a row written by run_id, with its own conditions (the header's facets as defaults), status, metrics and attachments, and it can be overwritten, archived or deleted without appending a header version. The verdict on a run moves to the Card (run_results), and GET /evals/{ns}/{name}/runs reads runs × metrics × the chosen Cards' judgements in one page. Four hashes keep it verifiable: the header's content_hash, each run's content_hash, the record's runs_hash, and a Card's used-set hash, from which changed_since_card is derived. This is the 9-stage change of #24, landed as one release.

Upgrading a hosted database

evalhub migrate for 0.2.0 applies the 0003 DDL and then a one-shot data migration, 0003_runs_split, recorded in data_migrations so it never runs twice. serve refuses to start until both have run.

• Runs are collected from every live 1.0 version by run_id (highest seq wins) and become rows. pass / fail → ok, error → error with error.kind: unrecorded, skipped → skipped with skip_reason: unrecorded; the dropped outcome is kept as meta.v0.2.0_migration.outcome; the source version's header facets are copied onto each run.
• Every live 1.0 version body becomes its 2.0 header (no runs, schema 2.0). Header content_hash values change; the old and new hash of every version are in the audit log (migration.runs_split). Versions that differed only in runs become identical header versions.
• Each Card's used runs are rebuilt from the Eval version it points at, so a run that changed since shows as changed at once. A Card's relations keep resolving because they point at version ids, not hashes. Used run ids with no run are skipped and recorded as migration.card_runs_unmatched (0.1.x never checked attrs.runs); these are expected, not errors.
• The step is one transaction: a failure (or the release command's 5-minute timeout) rolls it back and leaves it pending. A run_id it cannot accept is named in the error, and nothing is written.

Production deploy (docs/hosting.md, "Upgrading to 0.2.0"; not the plain fly deploy of other releases), from the checkout of the v0.2.0 tag:

1. Snapshot: fly volumes snapshots create <volume-id> --app evalhub-db (and a pg_dump).
2. Stop writes: fly machine update <machine-id> --autostart=false --skip-start --app evalhub, then fly machine stop <machine-id> --app evalhub; /api/v1/healthz must fail. Otherwise the old binary can store a 1.0 body after the migration and nothing would split it.
3. Deploy: fly deploy --ha=false. The release command logs applied data migration 0003_runs_split: …. If the Machine is still stopped afterwards, fly machine start <machine-id> --app evalhub.
4. Confirm: fly releases --app evalhub and fly ssh console --app evalhub -C "evalhub --version" (→ evalhub 0.2.0); /api/v1/healthz and /openapi.json (info.version stays the contract's 1.0.0) confirm liveness only.
5. Smoke: bash deploy/fly/smoke.sh (header POST, two run PUTs, a Card with run_results, the projection, anonymous reads after publishing).

Upgrading a client

0.1.x	0.2.0
POST /evals/{ns}/{name} with runs[] in the body	POST the header without runs, then PUT …/runs/{run_id} per run or POST …/runs:batch
runs[].outcome (pass / fail / error / skipped)	the run's status (ok / error with error.kind / skipped with skip_reason), and the pass / fail in the Card's run_results[]
"schema": "evalhub.eval/1.0"	"schema": "evalhub.eval/2.0" (GET /schemas/eval-2)
"schema": "evalhub.card/1.0"	"schema": "evalhub.card/1.1" (adds run_results); evalhub.card/1.0 is still accepted

• An evalhub.eval/1.0 body with runs[] is accepted and converted in 0.2.0 only: stored as a 2.0 header plus run rows, with a Deprecation: @1790294400 header (RFC 9745), converted_from: "evalhub.eval/1.0" and converted_runs in the response. 0.3.0 rejects it with 422 runs_moved, as 0.2.0 already does for a 2.0 header carrying runs. The issue for that removal is opened when 0.2.0 lands, with its date.
• Two things a 1.0 client notices already in 0.2.0: the returned content_hash is the converted 2.0 header's, not the hash of the posted body; and re-posting a body that omits a run no longer drops that run (runs are rows now; archive or delete it).
• A run PUT must carry run_id in the body, equal to the path's.

API

• Runs under /api/v1/evals/{ns}/{name}: PUT /runs/{run_id} (201 created / 200 updated / 200 unchanged), POST /runs:batch (all or nothing; a refusal lists every failing element), GET /runs (the projection: cards=, where, sort, limit 1–200, cursor, include=archived,deleted for members), GET / PATCH (archived) / DELETE (tombstone) /runs/{run_id}. Runs follow the Eval's visibility; every route is 404 for an Eval the caller may not see.
• Projection paths, bracket form: run_id, status, error.kind, started_at, ended_at, facet keys (model.id, …), fingerprint.{facet}, metrics[{ns}/{name}], results[{card}][{metric}].value / .label, ext.*. Per Card the page reports runs_used, used_set_hash, posted_used_set_hash and changed_since_card.
• GET /evals/{ns}/{name}[@…] carries runs: { count, by_status, archived, deleted, runs_hash } (archived / deleted for members only). The comparison view (GET /evals/…/cards) compares same_harness / same_model against every run a Card used and adds runs_used, used_set_hash, changed_since_card.
• Redaction: a public Card's run_results entries that name an Eval the reader may not see are removed and counted in withheld.run_results (GET and POST /cards/query); a writer naming a run of an Eval they may not see gets run_unknown, as for a missing run.
• Limits ([limits]): body_bytes 16 MiB (413 body_too_large, every endpoint; was axum's 2 MiB default), batch_runs 1,000 (413 batch_too_large), run_results 100,000 per Card version (422 too_many_run_results). Refused, never truncated.
• Schemas: GET /schemas/eval-2 (the header) and GET /schemas/run are new; GET /schemas/eval stays the 1.0 document until 0.3.0.
• Error codes (new): runs_moved, run_status_detail, run_id_invalid, run_id_mismatch, run_result_value_or_label, run_results_eval_unknown, run_unknown, run_not_in_used_set, too_many_run_results, batch_duplicate_run_id (422); run_deleted (409); batch_too_large, body_too_large (413).
• Audit: run.create / run.update / run.archive / run.unarchive / run.delete, migration.runs_split, migration.card_runs_unmatched.
• info.version of /openapi.json stays 1.0.0.

Crates

evalhub-schema, evalhub-core and evalhub-query are published at 0.2.0 (the store and the server ship as ghcr.io/ynishi/evalhub:0.2.0). just semver-check against 0.1.4 on crates.io, before the bump:

Crate	Verdict
evalhub-schema	requires a new major (0.2 on 0.x): 5 lints — Eval.runs removed, eval::Run and eval::Outcome removed, Card.run_results added to an exhaustively constructible struct, 13 ErrorCode variants added
evalhub-core	no semver update required (196 checks: 196 pass, 58 skip); bumped with the workspace
evalhub-query	requires a new major (0.2 on 0.x): 1 lint — Column::Run and SortKey::Run added to exhaustive enums

At 0.2.0 the same command reports "no semver update required" for all three (0 checks, 254 skipped: a major bump skips the lints).

Changes by stage

• #36 Schema: Run leaves the Eval body; per-run facets, status and metrics; Card.run_results; evalhub.eval/2.0 (#27)
• #37 Core: validate a Run, refuse runs in an Eval body, check run_results, fingerprint a run, publish the run hash formulas (#28)
• #38 Query: a run projection target — path table, run columns, sort keys and cursor for GET /evals/{ns}/{name}/runs (#29)
• #39 Store: run tables, run put/batch/archive/delete, runs_hash, audit actions, GC and download references for run attachments (#30)
• #40 Store: the Card's used set and run_results at ingest, the run projection query, hidden Evals for redaction, comparison over used runs (#31)
• #41 Migration: split runs out of stored Eval bodies (0.1.x → 0.2.0), record that it ran, refuse to serve until it has (#32)
• #42 Server: run endpoints, runs:batch, request limits, run_results redaction, the Eval runs summary, comparison counts, OpenAPI (#33)
• #43 Web UI and deploy scripts: the Eval page reads /runs and shows status, error kind, metrics and withheld counts; the Fly smoke test PUTs runs (#34)
• This PR: README (run paths, [limits], schema ids, Upgrading from 0.1.x), the crate docs' final pass, the 0.2.0 bump (#35)

Verified

Command	Outcome
cargo fmt --check	clean
cargo clippy --workspace --all-targets -- -D warnings	clean
cargo test -p evalhub-schema	20 passed
cargo test -p evalhub-core	62 passed
cargo test -p evalhub-query	27 passed
cargo test -p evalhub-store (Docker)	96 passed
cargo test -p evalhub-server (Docker)	82 passed, no .snap.new
just sqlx-check (throwaway postgres:16-alpine, migrations 0001–0003)	exit 0
cargo deny check	advisories ok, bans ok, licenses ok, sources ok
RUSTDOCFLAGS=-D warnings cargo doc --no-deps (five crates)	clean
just semver-check (before the bump)	schema: major, 5 lints; core: none; query: major, 1 lint
just semver-check (at 0.2.0, on the commit)	all three: "no semver update required" (0 checks, 254 skipped: a major bump skips the lints)
just package (on the commit)	evalhub-schema 66422, evalhub-core 75555, evalhub-query 68769 bytes, gate ok
just e2e (on the commit)	e2e: 18 passed, 0 failed; Playwright 6 passed
Cargo.lock	only the five evalhub-* versions changed

Not run: deploy/fly/smoke.sh (targets the hosted service; step 5 above, after the deploy).

Closes #24
Refs #35

---

SDK crates: evalhub-schema evalhub-core evalhub-query
Image: ghcr.io/ynishi/evalhub:0.2.0