feat: bind published benchmark figures to their upstream artifacts

[abrichr]

The gap

Every claim guard in this org checks that a file contains an attribution string. None of them checks the number. openadapt-web published success_count: 20 for the OpenEMR compiled arm for five weeks while the upstream artifact said 19, and nothing went red.

scripts/check_profile.py validates the org README's links, headings, and the phrase "more than 1.6k stars". It validates zero benchmark figures.

What this adds

benchmark-claims.json pins four openadapt-flow artifacts at commit aee0941, vendors their exact bytes under benchmark-claims/upstream/, and binds every figure printed on README.md and profile/README.md to a JSON pointer into one of them. Fifteen figures are bound today.

scripts/check_benchmark_claims.py does five things:

1. Re-hashes every vendored snapshot. Editing a snapshot to bless a wrong figure fails the run.
2. Renders each registered figure straight out of the artifact and requires the exact published text. 39.2s has to come from wall_s_p50, not from someone's memory.
3. Cross-checks the derived comparison.json against the two measured results.json files, the way paper/check_artifacts.py does in flow.
4. Sweeps both surfaces for figure-shaped tokens and fails on any token no entry claims. A new figure can't land unregistered, and an exemption that no longer matches anything can't sit there rotting.
5. With --online, re-fetches each pinned path from raw.githubusercontent.com and compares digests. An unreachable GitHub warns, because an outage isn't drift. A digest mismatch exits non-zero.

What it found on the way in

profile/README.md publishes 20/20 for the OpenEMR compiled arm. The pinned artifact records 19 of 20, because the saved-row oracle adjudication on 2026-07-28 rejected compiled run 20: the note stayed in the unsaved entry form and no saved row ever contained it.

This is the same defect openadapt-web had, on a page in the same family, and it is still live. This PR does not touch the prose, since front-page corrections are owned elsewhere. The mismatch is recorded as a dated drift that names an owner and expires on 2026-09-25. The checker prints it on every run, exits non-zero by default, and exits non-zero unconditionally once that date passes. Sixty days is the hard ceiling on any such record, enforced by the registry validator.

Someone still has to change that 20/20 to 19/20. The guard buys time and makes the clock visible. It does not fix the number.

Wiring

Offline half on every pull request, online half on the daily schedule. Standard library only, matching the rest of scripts/.

Proof

Each check was run failing, then passing, against the real repository. Verbatim failure output:

A vendored snapshot digit changed from 19 to 20

ERROR: source openemr: snapshot benchmark-claims/upstream/openadapt-flow/aee094193b232f472f991be6fa9b33c3c4b3f9be/benchmark/openemr/results.json has digest sha256:0169294be83640fa74865ff2c10ecf1d6978d7be59ebaa4c7c3e349f3de96893, but the registry pins sha256:30de6dbb43ad0db4fd8b55737d765d06747b896ffae04522b6957d567dcfd89c. The vendored bytes were edited, or the pin is wrong. Re-vendor from OpenAdaptAI/openadapt-flow@aee094193b232f472f991be6fa9b33c3c4b3f9be:benchmark/openemr/results.json.
ERROR: claim openemr-compiled-success-ratio: source openemr did not verify

An unregistered 19/20 added to README.md

ERROR: README.md: the figure '19/20' is not claimed by any benchmark-claims.json entry. Bind it to its upstream artifact, or record it in non_benchmark_figures with a reason. Context: ...e machine-readable values after merge. Compiled replay went 19/20 on the third-party EMR....

A published figure moved away from its artifact (70.4s to 7.4s, registry updated to match the wrong prose)

ERROR: claim openemr-agent-p50: profile/README.md publishes '7.4s', but openemr ['/arms/agent/wall_s_p50'] renders '70.4s'

A registered figure's expected text corrupted in the registry alone

ERROR: claim openemr-agent-p50 context does not contain its text

--online, pinned path repointed so upstream serves different bytes

ERROR: source openemr: https://raw.githubusercontent.com/OpenAdaptAI/openadapt-flow/aee094193b232f472f991be6fa9b33c3c4b3f9be/benchmark/BENCHMARK.md returned digest sha256:ceb6841165c2e18b43e511281737e59ae919824e4ee629a8fe6408afea33434f, but the vendored snapshot is sha256:30de6dbb43ad0db4fd8b55737d765d06747b896ffae04522b6957d567dcfd89c. The pin no longer describes the upstream bytes.

--online with GitHub unreachable warns and exits 0

WARNING: source openemr: https://raw.githubusercontent.invalid/OpenAdaptAI/openadapt-flow/aee094193b232f472f991be6fa9b33c3c4b3f9be/benchmark/openemr/results.json is unreachable: <urlopen error [Errno 8] nodename nor servname provided, or not known>
Bound 15 published figures across 2 surfaces to 4 pinned artifacts; re-fetched 0 of 4 upstream paths; 1 dated drift record(s) still open.
EXIT=0

Every file was restored after each proof. 43 new tests, 197 in the suite, all green locally.