Skip to content

CatchBench 0.1.3

Latest

Choose a tag to compare

@yzhao062 yzhao062 released this 26 Oct 03:01
· 3 commits to main since this release

No board number changes. The board is byte for byte the board of 0.1.2.

Install

python -m pip install "catchbench>=0.1.3"
catchbench --task pre        # about a second, no key, no corpus download, no GRADE checkout

Changed

  • The board computes its baselines through auditable. 79 board entries now run through the method library auditable (0.3.0 or later), by way of the new catchbench.via_auditable. Each board factory still builds its original entries in their original order, and replaces every migrated one with an adapter that asks a library method for the per-sample scores or findings and hands them to this repository's evaluation code. Data, labels, splits, protocols and metrics are unchanged. The flag_all, flag_none and oracle_privilege_diff PRE rows and the attribution random floor stay original.
  • The LLM-judge entries replay their caches through the library. The library builds each prompt, parses each recorded reply, and runs the single-step drivers on the decisions the cache logged. Every replay is checked against the cache as it is scored (the named step and ranking, the query count, the log headers), and a mismatch raises. The cache checks are the original ones, and regenerate_cache runs the library's drivers too.
  • The original implementations stay in their modules, because tools and the committed addendum sidecars refer to them by name; the board no longer calls their scoring code. tools/statistical_tests.py recomputes its PRE predictions from them.
  • Dependency. auditable>=0.3.0 (was >=0.2.0); the graph-ad and full extras add auditable[graph-ad]. CI pins the exact library version in .github/ci-constraints.txt.

Added

  • tools/board_consumption.py checks that the board reads the library: with the original scoring cores replaced by functions that raise, the board completes; with the library outputs perturbed, every migrated entry changes in every table it appears in and nothing else changes. board.yml runs it, and tests/test_via_auditable.py and tests/test_via_auditable_graph_ad.py run the same two checks on synthetic tasks.
  • CI. The graph-ad job installs the pinned GRADE checkout and runs the library's PyGOD checks; tools/check_test_report.py --only takes several files; the smoke check walks subpackages.