Repository navigation
No board number changes. The board is byte for byte the board of 0.1.2.
Install
python -m pip install "catchbench>=0.1.3"
catchbench --task pre # about a second, no key, no corpus download, no GRADE checkoutChanged
- The board computes its baselines through
auditable. 79 board entries now run through the method libraryauditable(0.3.0 or later), by way of the newcatchbench.via_auditable. Each board factory still builds its original entries in their original order, and replaces every migrated one with an adapter that asks a library method for the per-sample scores or findings and hands them to this repository's evaluation code. Data, labels, splits, protocols and metrics are unchanged. Theflag_all,flag_noneandoracle_privilege_diffPRE rows and the attribution random floor stay original. - The LLM-judge entries replay their caches through the library. The library builds each prompt, parses each recorded reply, and runs the single-step drivers on the decisions the cache logged. Every replay is checked against the cache as it is scored (the named step and ranking, the query count, the log headers), and a mismatch raises. The cache checks are the original ones, and
regenerate_cacheruns the library's drivers too. - The original implementations stay in their modules, because tools and the committed addendum sidecars refer to them by name; the board no longer calls their scoring code.
tools/statistical_tests.pyrecomputes its PRE predictions from them. - Dependency.
auditable>=0.3.0(was>=0.2.0); thegraph-adandfullextras addauditable[graph-ad]. CI pins the exact library version in.github/ci-constraints.txt.
Added
tools/board_consumption.pychecks that the board reads the library: with the original scoring cores replaced by functions that raise, the board completes; with the library outputs perturbed, every migrated entry changes in every table it appears in and nothing else changes.board.ymlruns it, andtests/test_via_auditable.pyandtests/test_via_auditable_graph_ad.pyrun the same two checks on synthetic tasks.- CI. The graph-ad job installs the pinned GRADE checkout and runs the library's PyGOD checks;
tools/check_test_report.py --onlytakes several files; the smoke check walks subpackages.