Repository navigation
auditable 0.4.0
0.4.0 adds leaderboards: auditable.bench reruns the tables of the CatchBench board on the same populations and protocols, and places any new method on them.
Added
auditable.bench:leaderboard(task_id, methods=None, extra=())scores methods on a task and returns a rankedBoard(print,to_markdown(),to_dict(),compare_golden(),rank_of());extra=[MyMethod]adds a method that is not in the registry. A method whose extra is missing is listed as skipped with the extra named, and a method that raises is listed as an error while the rest of the board is still scored. Oracle rows, which read the labels, are shown as references and never ranked; control rows are ranked and marked.- Six tasks, each reproducing a table of the CatchBench 0.1.3 board row by row:
detect/swegym(376 runs) anddetect/tau(660 runs), run-level failure detection by ROC-AUC;live/swegymandlive/tau, early warning on 25, 50, 75 and 100 percent prefixes;localize/whoandwhen(126 failed runs), decisive-step localization by top-1, top-3 and MRR, including 31 LLM-judge rows replayed from CatchBench's committed prediction caches without an API call; andpre/over_privilege(1,187 configurations), configuration audit by F1, precision and recall, overall and per source. The populations are read through agentpile at the dataset revisions CatchBench pins, with the selection rules of GRADE commit 3839a57; the protocols are ported from CatchBench's adapters. run.lr_spline_flat, the flexible size control: a cubic spline of the four GRADEflatcounts, then the board's logistic regression, ported from thesize (spline)arm of CatchBench's size audits. It joins thedetectandliveboards as a control. Ondetect/swegymit ranks second, above both GRADE dependency regressions; ondetect/tauit ranks belowrun.lr_flatdep. The library now holds 44 methods.- Golden files with the board values of every row and the SHA-256 of the board they come from; the spline control's value comes from CatchBench's committed size audits instead, with a 0.001 tolerance.
python -m auditable.bench <task> --goldencompares a run with them and fails on a mismatch, on an error row, on a golden row left unchecked (skipped, missing a metric, absent from the task's default rows, or in a golden file that is missing), and on a run that compared no golden value;--allow-skippedaccepts rows skipped for a missing extra. Three rows differ on macOS arm64 by a known platform effect and are reported as platform values;--exactcounts them as mismatches, and the weekly Linux run uses it. python -m auditable.bench, the command-line entry: lists the tasks, prints a board, or emits it as Markdown or JSON.- The
benchextra (pyod,scikit-learn,numpy,networkx,agentpile[hf]); the graph rows also needgraph-ad. - CI: a
bench-corejob runs the bench unit tests, which download nothing, a weeklybench goldenworkflow reruns every task on Linux against its golden file, and a weeklybench platformsworkflow installs the package from a clean checkout on Linux, macOS, and Windows (Python 3.10 and 3.13) with the README's graph-ad steps and checks three tasks against their golden files.