Repository navigation
v0.2.0 — offline benchmark
v0.2 — offline benchmark (milestone issues #6, #7, #11).
amberfork-bench: the pre-registered chimera protocol (BENCHMARK.md) as a Rust harness —
four arms (random, pos-lexical, nw-structural/resync, nw-lexical/resync) scored on identical
fixtures with coverage accounting, Wilson 95% intervals, and a calibration curve; parameter
freeze viabench/params.toml(every table names the config sha256); committed results
documents + offlinereportmode that re-renders them with zero fetch.- Mode A′ cross-system pairs, end-to-end in one binary:
fetch(pinned upstream commits,
licensing notices, provenance record) → TapeAgents reference adapter →build-pairs→
scored disclosure. The honest result on the 4 constructible real pairs is a null
(engine 0.50 ±3 vs random 0.75 — short runs, murky cross-system gold; both pre-registered
threats) — shipped as a disclosed limit, not a headline (notebook 016). - CI-visible fork-localization gate: GAIA-sanitized dev fixtures for seeds 42/43/44 committed
underbench/fixtures/chimera_noise_seed*_dev/with a full regeneration recipe;
chimera_paritypins each seed's own baseline on everyamberfork-alignchange (issue #11,
notebooks 013–014). - Honesty correction: the seed-42 0.75 exact headline was a favorable draw — README and the
gate now lead with the cross-seed ±3 window and the 0.56 aggregate exact (notebook 014). - First sealed test-split reveal (protocol rule 2, at this tag): dev-tuned params
generalized — engine ±3 0.91 [0.78, 0.97] vs best baseline 0.49 [0.33, 0.64]
(non-overlapping), exact 0.49 [0.33, 0.64] vs 0.00, on n=35 unseen pairs across three seeds;
calibration monotone on unseen data; results committed withreportsnapshot tests
(notebook 017). amberfork-ingest: warn when a trace declares a non-native schema version.- Workspace version now single-sourced from
[workspace.package](0.2.0).