Skip to content

auditable 0.4.0

Choose a tag to compare

@yzhao062 yzhao062 released this 15 Nov 06:22
· 2 commits to main since this release

0.4.0 adds leaderboards: auditable.bench reruns the tables of the CatchBench board on the same populations and protocols, and places any new method on them.

Added

  • auditable.bench: leaderboard(task_id, methods=None, extra=()) scores methods on a task and returns a ranked Board (print, to_markdown(), to_dict(), compare_golden(), rank_of()); extra=[MyMethod] adds a method that is not in the registry. A method whose extra is missing is listed as skipped with the extra named, and a method that raises is listed as an error while the rest of the board is still scored. Oracle rows, which read the labels, are shown as references and never ranked; control rows are ranked and marked.
  • Six tasks, each reproducing a table of the CatchBench 0.1.3 board row by row: detect/swegym (376 runs) and detect/tau (660 runs), run-level failure detection by ROC-AUC; live/swegym and live/tau, early warning on 25, 50, 75 and 100 percent prefixes; localize/whoandwhen (126 failed runs), decisive-step localization by top-1, top-3 and MRR, including 31 LLM-judge rows replayed from CatchBench's committed prediction caches without an API call; and pre/over_privilege (1,187 configurations), configuration audit by F1, precision and recall, overall and per source. The populations are read through agentpile at the dataset revisions CatchBench pins, with the selection rules of GRADE commit 3839a57; the protocols are ported from CatchBench's adapters.
  • run.lr_spline_flat, the flexible size control: a cubic spline of the four GRADE flat counts, then the board's logistic regression, ported from the size (spline) arm of CatchBench's size audits. It joins the detect and live boards as a control. On detect/swegym it ranks second, above both GRADE dependency regressions; on detect/tau it ranks below run.lr_flatdep. The library now holds 44 methods.
  • Golden files with the board values of every row and the SHA-256 of the board they come from; the spline control's value comes from CatchBench's committed size audits instead, with a 0.001 tolerance. python -m auditable.bench <task> --golden compares a run with them and fails on a mismatch, on an error row, on a golden row left unchecked (skipped, missing a metric, absent from the task's default rows, or in a golden file that is missing), and on a run that compared no golden value; --allow-skipped accepts rows skipped for a missing extra. Three rows differ on macOS arm64 by a known platform effect and are reported as platform values; --exact counts them as mismatches, and the weekly Linux run uses it.
  • python -m auditable.bench, the command-line entry: lists the tasks, prints a board, or emits it as Markdown or JSON.
  • The bench extra (pyod, scikit-learn, numpy, networkx, agentpile[hf]); the graph rows also need graph-ad.
  • CI: a bench-core job runs the bench unit tests, which download nothing, a weekly bench golden workflow reruns every task on Linux against its golden file, and a weekly bench platforms workflow installs the package from a clean checkout on Linux, macOS, and Windows (Python 3.10 and 3.13) with the README's graph-ad steps and checks three tasks against their golden files.