Repository navigation
Releases
Releases · yzhao062/catchbench
CatchBench 0.1.3
No board number changes. The board is byte for byte the board of 0.1.2.
Install
python -m pip install "catchbench>=0.1.3"
catchbench --task pre # about a second, no key, no corpus download, no GRADE checkoutChanged
- The board computes its baselines through
auditable. 79 board entries now run through the method libraryauditable(0.3.0 or later), by way of the newcatchbench.via_auditable. Each board factory still builds its original entries in their original order, and replaces every migrated one with an adapter that asks a library method for the per-sample scores or findings and hands them to this repository's evaluation code. Data, labels, splits, protocols and metrics are unchanged. Theflag_all,flag_noneandoracle_privilege_diffPRE rows and the attribution random floor stay original. - The LLM-judge entries replay their caches through the library. The library builds each prompt, parses each recorded reply, and runs the single-step drivers on the decisions the cache logged. Every replay is checked against the cache as it is scored (the named step and ranking, the query count, the log headers), and a mismatch raises. The cache checks are the original ones, and
regenerate_cacheruns the library's drivers too. - The original implementations stay in their modules, because tools and the committed addendum sidecars refer to them by name; the board no longer calls their scoring code.
tools/statistical_tests.pyrecomputes its PRE predictions from them. - Dependency.
auditable>=0.3.0(was>=0.2.0); thegraph-adandfullextras addauditable[graph-ad]. CI pins the exact library version in.github/ci-constraints.txt.
Added
tools/board_consumption.pychecks that the board reads the library: with the original scoring cores replaced by functions that raise, the board completes; with the library outputs perturbed, every migrated entry changes in every table it appears in and nothing else changes.board.ymlruns it, andtests/test_via_auditable.pyandtests/test_via_auditable_graph_ad.pyrun the same two checks on synthetic tasks.- CI. The graph-ad job installs the pinned GRADE checkout and runs the library's PyGOD checks;
tools/check_test_report.py --onlytakes several files; the smoke check walks subpackages.
CatchBench 0.1.2
No board number changes. Every score reported in the paper and printed by catchbench --task pre is identical to 0.1.1.
Install
python -m pip install "catchbench>=0.1.2"
catchbench --task pre # about a second, no key, no corpus download, no GRADE checkoutWhy this release exists
The 0.1.1 wheel was not usable from a clean install in three ways, and each is fixed here.
- A missing GRADE checkout is reported instead of raised. Running plain
catchbenchafter installing the 0.1.1 wheel ended in an unhandledImportErrortraceback and exit 1. It now prints one paragraph naming the setup that works, links the README's Full Board section, and exits 2.--task gold-v2is new in this release rather than repaired: 0.1.1's parser accepted onlyallandpre, so that argument was an argparse invalid-choice error there. It reports a missing checkout the same way, at exit 2. - The GRADE checkout is found in the working directory. Resolution looked only at
GRADE_DIRand the parent of the installed package. The old message asked the reader to clone GRADE to../grade, beside a repository, which is a real location from a checkout and no location at all from a wheel install. - The wheel carries the licence documents its own
NOTICEcites.THIRD_PARTY_LICENSES.md,ASSET_MANIFEST.jsonandthird_party/licenses/were absent from the 0.1.0 and 0.1.1 wheels while the bundledNOTICEreferred readers to them for the terms on the bundled records.
Added
- Judge addendum records. Twelve prediction caches and twelve sidecars under
data/llm_judge_addendum/, covering later releases scored on the same 126 Who&When runs and the same all-at-once protocol as the published board. These sit outside the frozen arena, add no entrant to it, and declare no contrast. The 0.1.1 wheel carried none of them. examples/add_a_method.py. A runnable end-to-end method, scored on the offline PRE board in under a second with no key, no corpus download, no GRADE checkout and no torch. It lands below theflag_allfloor, which is the honest result for a keyword rule and the point the example is making.- A named-value Gold board,
gold_v2_namedvalueontau-bench-gold-v2, with its five-seed fixed-margin diagnostic. - Project URLs in the package metadata, and a
CHANGELOG.md.
Verification
- Full suite: 946 passed, with the deliberate canary failure and nothing else.
- The wheel is reproducible: an independent build by a reviewing agent produced the identical SHA-256.
- A clean install from PyPI reproduces the PRE board over 1,187 configurations from six corpora, exits 2 without a traceback on both bridge-dependent paths, and resolves every document its
NOTICEcites.
Artifacts
| File | SHA-256 |
|---|---|
catchbench-0.1.2-py3-none-any.whl |
086a854eb5c61006c06db5bb491e50963c7b2dff2e660e8ffda9c6667d931227 |
catchbench-0.1.2.tar.gz |
82e3fe904846f4cf0938401c4e2d961072614edbba5e608a5e5b23782bf1df8f |
Known issue in this tag
The Verify workflow fails at this commit. Its offline PRE step read the new documentation link in the command's footer as a board row that had gone missing. The package is unaffected, and the Board workflow, which re-derives the board itself, passed. Repaired on main in 167ba6b.