Repository navigation
Releases
auditable 0.5.0
The README now groups the methods by the stage at which they audit (PRE, LIVE, POST), following PyOD's catalogue layout, and presents the six leaderboards as part of that catalogue. Every leaderboard row now names the kind of method it runs.
Added
Rowgainstype(Rule, Supervised, PyOD detector, Graph detector, LLM judge, Baseline, Oracle, or Custom) andalgorithm(the name on the method card), filled for scored, skipped, and error rows and exported byBoard.to_dict(), whose existing keys are unchanged. A class defined outside the library is Custom whatever its role; only the benchmark-only controls and oracle ofauditable.benchtake Baseline or Oracle from their role.- A hero figure (
assets/hero.png, editable source infigure-src/hero/) and adetect/swegymchart (assets/leaderboard-swegym.png, generated from the golden file byfigure-src/leaderboard_swegym.py, colored by method type). docs/install.md(the extras and their dependencies, the graph methods on Colab and on torch patch releases, running the examples from a clone, the platform checks) anddocs/agent-traces.md(dependency rules, the GRADE equality results per corpus, the text and subagent conventions), both moved out of the README and added to the documentation navigation.tests/test_bench_readme.pybuilds the six board blocks of the README offline from the golden files and checks that the README shows them, that the Stage column appears exactly when the shown rows span more than one stage, and that the method counts of the category table match the registry.python tests/test_bench_readme.pyrewrites the board blocks and the four method tables.
Changed
- Table layouts differ from 0.4.0, so code that parses them by column position must update.
Board.to_markdown()prints Rank, Type, Method, Algorithm, and the metrics, with a Stage column (PRE, LIVE, or POST) after Rank when the shown rows audit at more than one stage, as onlive/swegymandlive/tau;print(board)adds a type column.method_table()adds a Year column taken from the key of a method's first citation, which the Reference text then omits (ECOD (Li et al.)). It lists the families in stage order (config,online,run,step) in both the default and thedetailed=Truetable. A method registered from outsideauditable.methodshas the typeCustom. - README: a stage-and-family table, why-bullets, and the hero figure open the page; then Install (graph steps folded), a five-line example, Implemented Algorithms (one generated table per family and a folded utility table), Leaderboards (the
detect/swegymchart and all six boards folded), Quick Start with a PRE example, a short API Cheatsheet, and shorter sections for real traces, reproducing CatchBench, adding a method, and records. - The method tables link the sources of the PyOD KNN and HBOS rows and of the PyGOD CONAD, AnomalyDAE, and GAAN rows, which had no link before.
- The CatchBench BibTeX entry in the README lists all eight authors of arXiv:2608.22808.
docs/index.md,docs/leaderboard.md,docs/methods-contract.md, anddocs/api.mddescribe the stages and the Type, Algorithm, Stage, and Year columns.
auditable 0.4.0
0.4.0 adds leaderboards: auditable.bench reruns the tables of the CatchBench board on the same populations and protocols, and places any new method on them.
Added
auditable.bench:leaderboard(task_id, methods=None, extra=())scores methods on a task and returns a rankedBoard(print,to_markdown(),to_dict(),compare_golden(),rank_of());extra=[MyMethod]adds a method that is not in the registry. A method whose extra is missing is listed as skipped with the extra named, and a method that raises is listed as an error while the rest of the board is still scored. Oracle rows, which read the labels, are shown as references and never ranked; control rows are ranked and marked.- Six tasks, each reproducing a table of the CatchBench 0.1.3 board row by row:
detect/swegym(376 runs) anddetect/tau(660 runs), run-level failure detection by ROC-AUC;live/swegymandlive/tau, early warning on 25, 50, 75 and 100 percent prefixes;localize/whoandwhen(126 failed runs), decisive-step localization by top-1, top-3 and MRR, including 31 LLM-judge rows replayed from CatchBench's committed prediction caches without an API call; andpre/over_privilege(1,187 configurations), configuration audit by F1, precision and recall, overall and per source. The populations are read through agentpile at the dataset revisions CatchBench pins, with the selection rules of GRADE commit 3839a57; the protocols are ported from CatchBench's adapters. run.lr_spline_flat, the flexible size control: a cubic spline of the four GRADEflatcounts, then the board's logistic regression, ported from thesize (spline)arm of CatchBench's size audits. It joins thedetectandliveboards as a control. Ondetect/swegymit ranks second, above both GRADE dependency regressions; ondetect/tauit ranks belowrun.lr_flatdep. The library now holds 44 methods.- Golden files with the board values of every row and the SHA-256 of the board they come from; the spline control's value comes from CatchBench's committed size audits instead, with a 0.001 tolerance.
python -m auditable.bench <task> --goldencompares a run with them and fails on a mismatch, on an error row, on a golden row left unchecked (skipped, missing a metric, absent from the task's default rows, or in a golden file that is missing), and on a run that compared no golden value;--allow-skippedaccepts rows skipped for a missing extra. Three rows differ on macOS arm64 by a known platform effect and are reported as platform values;--exactcounts them as mismatches, and the weekly Linux run uses it. python -m auditable.bench, the command-line entry: lists the tasks, prints a board, or emits it as Markdown or JSON.- The
benchextra (pyod,scikit-learn,numpy,networkx,agentpile[hf]); the graph rows also needgraph-ad. - CI: a
bench-corejob runs the bench unit tests, which download nothing, a weeklybench goldenworkflow reruns every task on Linux against its golden file, and a weeklybench platformsworkflow installs the package from a clean checkout on Linux, macOS, and Windows (Python 3.10 and 3.13) with the README's graph-ad steps and checks three tasks against their golden files.
auditable 0.3.0
0.3.0 turns auditable into a library of auditing methods for AI agents, with one interface per family and the ability to read public agent traces through agentpile.
Added
auditable.methods: 43 methods in four families, each with one interface. Theconfigfamily (8 methods) audits an agent configuration before a run,online(5) raises causal alarms while a run streams, andrun(16) andstep(14) detect a failed run and localize its decisive step afterwards. A registry (list_methods,get_method) returns each method'sMethodCard, which records its fit scope, fidelity to its source, required fields, extras and citations. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench; every method that reproduces a CatchBench 0.1.2 board row matches it sample by sample.auditable.featurize: trace normalization for step lists, run dicts, typed graphs,SessionGraphandTraceViewinputs, and the features the methods compute (the GRADE structural layers, the Who&When step matrix, the step dependency graph, PyG encodings).auditable.llm: aprompt -> replybackend protocol,FixedResponseBackendfor tests and replays,OpenAICompatibleBackend(standard library only), and a prediction cache.auditable.from_agentpile,configs_from_agentpile,step_targets,run_targetsandauditable.datasets.load_tracesread agentpile records (public datasets, trace files, captured runs) into the methods' inputs and labels.convention="grade"withdeps="grade"reproduces the steps, dependencies and graphs of GRADE's converters (commit 3839a57) on seven corpora;convention="generic"(the default) makes one step per message and one per tool call.- Extras
tabular-ad,graph-adandagentpile; the core install still has no third-party dependency. docs/methods-contract.md, the contract a method follows and the behaviours a pinned consumer such as CatchBench relies on.
Behaviour of the Conversion
- In
convention="generic", content given as a list of parts (dicts with a stringtype, possibly mixed with bare strings) keeps its text and writes every other part (an image, a document) as[<type>], dropping that part's other fields;meta["report"]counts those parts and says how each step of the record maps into the trace. convention="grade"raisesValueErrorfor a run from a dataset revision other than the one its projection was checked on.- A run with subagent trajectories is refused by default; the error names the two ways to read it,
from_agentpile(run.subagents, ...)andsubagents="skip".
Deprecated
ActionGate,ReferenceLedgerandDataAuditoremit aDeprecationWarningwhen constructed and will be removed in a later release.
auditable 0.2.0
What's New
Data module completed (internal v0.2). The fitted dependency-state detector now has its full story: fit from captured history (DataAuditor.fit_records over decision records, fit_from_log over a FileSink JSONL through the fail-closed loader), a value-level fault injector for detector evaluation (auditable.data.inject: stale / drift / OOD over snapshot copies, deterministic, non-circular injection-site labels), the previously missing anomaly extra (pip install "auditable[anomaly]"), a documentation page (Data Module), and an offline composed example (examples/example_state_anomaly.py). The learned path remains an unpromoted module API: concept-demo evidence only, benchmark-grade evidence pending.
Two new capture integrations.
- LangChain (
auditable.integrations.langchain,pip install "auditable[langchain]"): a callback handler that lowers a real runnable/tool/LLM run into the typed session graph with honest dependency grading: OBSERVED only on digest-matched value handoffs, DECLARED for sequence topology, INFERRED for payload equality. Scored support is scoped to flat sequences. - MCP (
auditable.integrations.mcp,pip install "auditable[mcp]", Python 3.10+): instrument a FastMCP server in place so every tool call is captured at the boundary with digest-only arguments, results, and errors, and an honest zero-coverage report (no fabricated dependency or handoff edges).
LiveSession data hook. LiveSession(data_auditor=...) scores per-step dependency snapshots while a run streams, with an owned evidence boundary: observe(step, snapshot=...) deep-copies at ingestion, duplicate step ids are rejected before commit, and scorer outages surface as availability notes instead of vanishing.
Hardening. The block went through two external review rounds; fixes include version bounds for both integration extras, error fingerprints instead of raw exception text (privacy), generation-aware reset semantics with cancellation capture, canonical hashing for direct CallToolResult returns, and a pinned integration-contracts CI job with a langchain-core floor canary.
Full test suite: 311 passed. See the docs and examples/README.md.
auditable v0.1.1
Syncs PyPI with the real-run capture work added since 0.1.0. Additive only: no breaking changes to the existing API.
Plug in your real agent
- LangGraph capture. Wrap a
StateGraphwithinstrument(...)and every node's state-channel reads and writes become observed dependency edges, matched across LangGraph's superstep barrier and reducer-aware, with no change to your node logic. Optional extra:pip install "auditable[langgraph]". - Framework-agnostic capture.
TouchRecorderrecords the same observed edges from any tool loop (a raw OpenAI or Anthropic agent, your own scheduler) by declaring each step'sreads()andwrites(). - Real-LLM example.
examples/example_langgraph_llm_agent.pyruns the capture with live LLM nodes against any OpenAI-compatible endpoint (a plain OpenAI key, a gateway, or a local server), showing the capture is model-agnostic. Optional extra:pip install "auditable[llm]".
Both capture paths feed the existing analyze_run (POST): one typed two-layer decision graph, with the dependency layer recorded as OBSERVED edges over the resources each step read and wrote.
Install: pip install -U auditable
Full changelog: v0.1.0...v0.1.1
auditable v0.1.0
Capture, replay, and audit AI agent decisions on one typed two-layer decision graph, read at three points in a run: PRE (lint a declared plan before deploy), LIVE (replay a decision under the state that is live now and recover), POST (rank a finished run and name the keystone).
New in 0.1.0
- Audit report generation.
AuditReportover all three pillars, with an agent-facing Markdown report (to_markdown, dependency-free) and a human-facing PDF (to_pdf, via the[report]extra). - Docs. PRE / LIVE / POST guides, a PRE coverage map (OWASP / CWE to graph-structural predicates), and an audit-report reference.
- Two end-to-end examples that walk one payment through all three pillars.
The typed two-layer graph model is introduced in GRADE (arXiv:2606.22741).
Install
pip install auditable # core: capture, replay, recovery
pip install "auditable[graph]" # PRE lints + POST analyze_run
pip install "auditable[report]" # the human-facing PDF