Skip to content

Releases

CatchBench 0.1.3

Choose a tag to compare

@yzhao062 yzhao062 released this 26 Oct 03:01

No board number changes. The board is byte for byte the board of 0.1.2.

Install

python -m pip install "catchbench>=0.1.3"
catchbench --task pre        # about a second, no key, no corpus download, no GRADE checkout

Changed

  • The board computes its baselines through auditable. 79 board entries now run through the method library auditable (0.3.0 or later), by way of the new catchbench.via_auditable. Each board factory still builds its original entries in their original order, and replaces every migrated one with an adapter that asks a library method for the per-sample scores or findings and hands them to this repository's evaluation code. Data, labels, splits, protocols and metrics are unchanged. The flag_all, flag_none and oracle_privilege_diff PRE rows and the attribution random floor stay original.
  • The LLM-judge entries replay their caches through the library. The library builds each prompt, parses each recorded reply, and runs the single-step drivers on the decisions the cache logged. Every replay is checked against the cache as it is scored (the named step and ranking, the query count, the log headers), and a mismatch raises. The cache checks are the original ones, and regenerate_cache runs the library's drivers too.
  • The original implementations stay in their modules, because tools and the committed addendum sidecars refer to them by name; the board no longer calls their scoring code. tools/statistical_tests.py recomputes its PRE predictions from them.
  • Dependency. auditable>=0.3.0 (was >=0.2.0); the graph-ad and full extras add auditable[graph-ad]. CI pins the exact library version in .github/ci-constraints.txt.

Added

  • tools/board_consumption.py checks that the board reads the library: with the original scoring cores replaced by functions that raise, the board completes; with the library outputs perturbed, every migrated entry changes in every table it appears in and nothing else changes. board.yml runs it, and tests/test_via_auditable.py and tests/test_via_auditable_graph_ad.py run the same two checks on synthetic tasks.
  • CI. The graph-ad job installs the pinned GRADE checkout and runs the library's PyGOD checks; tools/check_test_report.py --only takes several files; the smoke check walks subpackages.

CatchBench 0.1.2

Choose a tag to compare

@yzhao062 yzhao062 released this 13 Sep 23:26

No board number changes. Every score reported in the paper and printed by catchbench --task pre is identical to 0.1.1.

Install

python -m pip install "catchbench>=0.1.2"
catchbench --task pre        # about a second, no key, no corpus download, no GRADE checkout

Why this release exists

The 0.1.1 wheel was not usable from a clean install in three ways, and each is fixed here.

  • A missing GRADE checkout is reported instead of raised. Running plain catchbench after installing the 0.1.1 wheel ended in an unhandled ImportError traceback and exit 1. It now prints one paragraph naming the setup that works, links the README's Full Board section, and exits 2. --task gold-v2 is new in this release rather than repaired: 0.1.1's parser accepted only all and pre, so that argument was an argparse invalid-choice error there. It reports a missing checkout the same way, at exit 2.
  • The GRADE checkout is found in the working directory. Resolution looked only at GRADE_DIR and the parent of the installed package. The old message asked the reader to clone GRADE to ../grade, beside a repository, which is a real location from a checkout and no location at all from a wheel install.
  • The wheel carries the licence documents its own NOTICE cites. THIRD_PARTY_LICENSES.md, ASSET_MANIFEST.json and third_party/licenses/ were absent from the 0.1.0 and 0.1.1 wheels while the bundled NOTICE referred readers to them for the terms on the bundled records.

Added

  • Judge addendum records. Twelve prediction caches and twelve sidecars under data/llm_judge_addendum/, covering later releases scored on the same 126 Who&When runs and the same all-at-once protocol as the published board. These sit outside the frozen arena, add no entrant to it, and declare no contrast. The 0.1.1 wheel carried none of them.
  • examples/add_a_method.py. A runnable end-to-end method, scored on the offline PRE board in under a second with no key, no corpus download, no GRADE checkout and no torch. It lands below the flag_all floor, which is the honest result for a keyword rule and the point the example is making.
  • A named-value Gold board, gold_v2_namedvalue on tau-bench-gold-v2, with its five-seed fixed-margin diagnostic.
  • Project URLs in the package metadata, and a CHANGELOG.md.

Verification

  • Full suite: 946 passed, with the deliberate canary failure and nothing else.
  • The wheel is reproducible: an independent build by a reviewing agent produced the identical SHA-256.
  • A clean install from PyPI reproduces the PRE board over 1,187 configurations from six corpora, exits 2 without a traceback on both bridge-dependent paths, and resolves every document its NOTICE cites.

Artifacts

File SHA-256
catchbench-0.1.2-py3-none-any.whl 086a854eb5c61006c06db5bb491e50963c7b2dff2e660e8ffda9c6667d931227
catchbench-0.1.2.tar.gz 82e3fe904846f4cf0938401c4e2d961072614edbba5e608a5e5b23782bf1df8f

Known issue in this tag

The Verify workflow fails at this commit. Its offline PRE step read the new documentation link in the command's footer as a board row that had gone missing. The package is unaffected, and the Board workflow, which re-derives the board itself, passed. Repaired on main in 167ba6b.