Skip to content

Releases

auditable 0.5.0

Choose a tag to compare

@yzhao062 yzhao062 released this 16 Nov 01:28

The README now groups the methods by the stage at which they audit (PRE, LIVE, POST), following PyOD's catalogue layout, and presents the six leaderboards as part of that catalogue. Every leaderboard row now names the kind of method it runs.

Added

  • Row gains type (Rule, Supervised, PyOD detector, Graph detector, LLM judge, Baseline, Oracle, or Custom) and algorithm (the name on the method card), filled for scored, skipped, and error rows and exported by Board.to_dict(), whose existing keys are unchanged. A class defined outside the library is Custom whatever its role; only the benchmark-only controls and oracle of auditable.bench take Baseline or Oracle from their role.
  • A hero figure (assets/hero.png, editable source in figure-src/hero/) and a detect/swegym chart (assets/leaderboard-swegym.png, generated from the golden file by figure-src/leaderboard_swegym.py, colored by method type).
  • docs/install.md (the extras and their dependencies, the graph methods on Colab and on torch patch releases, running the examples from a clone, the platform checks) and docs/agent-traces.md (dependency rules, the GRADE equality results per corpus, the text and subagent conventions), both moved out of the README and added to the documentation navigation.
  • tests/test_bench_readme.py builds the six board blocks of the README offline from the golden files and checks that the README shows them, that the Stage column appears exactly when the shown rows span more than one stage, and that the method counts of the category table match the registry. python tests/test_bench_readme.py rewrites the board blocks and the four method tables.

Changed

  • Table layouts differ from 0.4.0, so code that parses them by column position must update. Board.to_markdown() prints Rank, Type, Method, Algorithm, and the metrics, with a Stage column (PRE, LIVE, or POST) after Rank when the shown rows audit at more than one stage, as on live/swegym and live/tau; print(board) adds a type column. method_table() adds a Year column taken from the key of a method's first citation, which the Reference text then omits (ECOD (Li et al.)). It lists the families in stage order (config, online, run, step) in both the default and the detailed=True table. A method registered from outside auditable.methods has the type Custom.
  • README: a stage-and-family table, why-bullets, and the hero figure open the page; then Install (graph steps folded), a five-line example, Implemented Algorithms (one generated table per family and a folded utility table), Leaderboards (the detect/swegym chart and all six boards folded), Quick Start with a PRE example, a short API Cheatsheet, and shorter sections for real traces, reproducing CatchBench, adding a method, and records.
  • The method tables link the sources of the PyOD KNN and HBOS rows and of the PyGOD CONAD, AnomalyDAE, and GAAN rows, which had no link before.
  • The CatchBench BibTeX entry in the README lists all eight authors of arXiv:2608.22808.
  • docs/index.md, docs/leaderboard.md, docs/methods-contract.md, and docs/api.md describe the stages and the Type, Algorithm, Stage, and Year columns.

auditable 0.4.0

Choose a tag to compare

@yzhao062 yzhao062 released this 15 Nov 06:22

0.4.0 adds leaderboards: auditable.bench reruns the tables of the CatchBench board on the same populations and protocols, and places any new method on them.

Added

  • auditable.bench: leaderboard(task_id, methods=None, extra=()) scores methods on a task and returns a ranked Board (print, to_markdown(), to_dict(), compare_golden(), rank_of()); extra=[MyMethod] adds a method that is not in the registry. A method whose extra is missing is listed as skipped with the extra named, and a method that raises is listed as an error while the rest of the board is still scored. Oracle rows, which read the labels, are shown as references and never ranked; control rows are ranked and marked.
  • Six tasks, each reproducing a table of the CatchBench 0.1.3 board row by row: detect/swegym (376 runs) and detect/tau (660 runs), run-level failure detection by ROC-AUC; live/swegym and live/tau, early warning on 25, 50, 75 and 100 percent prefixes; localize/whoandwhen (126 failed runs), decisive-step localization by top-1, top-3 and MRR, including 31 LLM-judge rows replayed from CatchBench's committed prediction caches without an API call; and pre/over_privilege (1,187 configurations), configuration audit by F1, precision and recall, overall and per source. The populations are read through agentpile at the dataset revisions CatchBench pins, with the selection rules of GRADE commit 3839a57; the protocols are ported from CatchBench's adapters.
  • run.lr_spline_flat, the flexible size control: a cubic spline of the four GRADE flat counts, then the board's logistic regression, ported from the size (spline) arm of CatchBench's size audits. It joins the detect and live boards as a control. On detect/swegym it ranks second, above both GRADE dependency regressions; on detect/tau it ranks below run.lr_flatdep. The library now holds 44 methods.
  • Golden files with the board values of every row and the SHA-256 of the board they come from; the spline control's value comes from CatchBench's committed size audits instead, with a 0.001 tolerance. python -m auditable.bench <task> --golden compares a run with them and fails on a mismatch, on an error row, on a golden row left unchecked (skipped, missing a metric, absent from the task's default rows, or in a golden file that is missing), and on a run that compared no golden value; --allow-skipped accepts rows skipped for a missing extra. Three rows differ on macOS arm64 by a known platform effect and are reported as platform values; --exact counts them as mismatches, and the weekly Linux run uses it.
  • python -m auditable.bench, the command-line entry: lists the tasks, prints a board, or emits it as Markdown or JSON.
  • The bench extra (pyod, scikit-learn, numpy, networkx, agentpile[hf]); the graph rows also need graph-ad.
  • CI: a bench-core job runs the bench unit tests, which download nothing, a weekly bench golden workflow reruns every task on Linux against its golden file, and a weekly bench platforms workflow installs the package from a clean checkout on Linux, macOS, and Windows (Python 3.10 and 3.13) with the README's graph-ad steps and checks three tasks against their golden files.

auditable 0.3.0

Choose a tag to compare

@yzhao062 yzhao062 released this 25 Oct 17:14

0.3.0 turns auditable into a library of auditing methods for AI agents, with one interface per family and the ability to read public agent traces through agentpile.

Added

  • auditable.methods: 43 methods in four families, each with one interface. The config family (8 methods) audits an agent configuration before a run, online (5) raises causal alarms while a run streams, and run (16) and step (14) detect a failed run and localize its decisive step afterwards. A registry (list_methods, get_method) returns each method's MethodCard, which records its fit scope, fidelity to its source, required fields, extras and citations. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench; every method that reproduces a CatchBench 0.1.2 board row matches it sample by sample.
  • auditable.featurize: trace normalization for step lists, run dicts, typed graphs, SessionGraph and TraceView inputs, and the features the methods compute (the GRADE structural layers, the Who&When step matrix, the step dependency graph, PyG encodings).
  • auditable.llm: a prompt -> reply backend protocol, FixedResponseBackend for tests and replays, OpenAICompatibleBackend (standard library only), and a prediction cache.
  • auditable.from_agentpile, configs_from_agentpile, step_targets, run_targets and auditable.datasets.load_traces read agentpile records (public datasets, trace files, captured runs) into the methods' inputs and labels. convention="grade" with deps="grade" reproduces the steps, dependencies and graphs of GRADE's converters (commit 3839a57) on seven corpora; convention="generic" (the default) makes one step per message and one per tool call.
  • Extras tabular-ad, graph-ad and agentpile; the core install still has no third-party dependency.
  • docs/methods-contract.md, the contract a method follows and the behaviours a pinned consumer such as CatchBench relies on.

Behaviour of the Conversion

  • In convention="generic", content given as a list of parts (dicts with a string type, possibly mixed with bare strings) keeps its text and writes every other part (an image, a document) as [<type>], dropping that part's other fields; meta["report"] counts those parts and says how each step of the record maps into the trace.
  • convention="grade" raises ValueError for a run from a dataset revision other than the one its projection was checked on.
  • A run with subagent trajectories is refused by default; the error names the two ways to read it, from_agentpile(run.subagents, ...) and subagents="skip".

Deprecated

  • ActionGate, ReferenceLedger and DataAuditor emit a DeprecationWarning when constructed and will be removed in a later release.

auditable 0.2.0

Choose a tag to compare

@yzhao062 yzhao062 released this 20 Jul 11:24

What's New

Data module completed (internal v0.2). The fitted dependency-state detector now has its full story: fit from captured history (DataAuditor.fit_records over decision records, fit_from_log over a FileSink JSONL through the fail-closed loader), a value-level fault injector for detector evaluation (auditable.data.inject: stale / drift / OOD over snapshot copies, deterministic, non-circular injection-site labels), the previously missing anomaly extra (pip install "auditable[anomaly]"), a documentation page (Data Module), and an offline composed example (examples/example_state_anomaly.py). The learned path remains an unpromoted module API: concept-demo evidence only, benchmark-grade evidence pending.

Two new capture integrations.

  • LangChain (auditable.integrations.langchain, pip install "auditable[langchain]"): a callback handler that lowers a real runnable/tool/LLM run into the typed session graph with honest dependency grading: OBSERVED only on digest-matched value handoffs, DECLARED for sequence topology, INFERRED for payload equality. Scored support is scoped to flat sequences.
  • MCP (auditable.integrations.mcp, pip install "auditable[mcp]", Python 3.10+): instrument a FastMCP server in place so every tool call is captured at the boundary with digest-only arguments, results, and errors, and an honest zero-coverage report (no fabricated dependency or handoff edges).

LiveSession data hook. LiveSession(data_auditor=...) scores per-step dependency snapshots while a run streams, with an owned evidence boundary: observe(step, snapshot=...) deep-copies at ingestion, duplicate step ids are rejected before commit, and scorer outages surface as availability notes instead of vanishing.

Hardening. The block went through two external review rounds; fixes include version bounds for both integration extras, error fingerprints instead of raw exception text (privacy), generation-aware reset semantics with cancellation capture, canonical hashing for direct CallToolResult returns, and a pinned integration-contracts CI job with a langchain-core floor canary.

Full test suite: 311 passed. See the docs and examples/README.md.

auditable v0.1.1

Choose a tag to compare

@yzhao062 yzhao062 released this 24 Jun 04:58

Syncs PyPI with the real-run capture work added since 0.1.0. Additive only: no breaking changes to the existing API.

Plug in your real agent

  • LangGraph capture. Wrap a StateGraph with instrument(...) and every node's state-channel reads and writes become observed dependency edges, matched across LangGraph's superstep barrier and reducer-aware, with no change to your node logic. Optional extra: pip install "auditable[langgraph]".
  • Framework-agnostic capture. TouchRecorder records the same observed edges from any tool loop (a raw OpenAI or Anthropic agent, your own scheduler) by declaring each step's reads() and writes().
  • Real-LLM example. examples/example_langgraph_llm_agent.py runs the capture with live LLM nodes against any OpenAI-compatible endpoint (a plain OpenAI key, a gateway, or a local server), showing the capture is model-agnostic. Optional extra: pip install "auditable[llm]".

Both capture paths feed the existing analyze_run (POST): one typed two-layer decision graph, with the dependency layer recorded as OBSERVED edges over the resources each step read and wrote.

Install: pip install -U auditable

Full changelog: v0.1.0...v0.1.1

auditable v0.1.0

Choose a tag to compare

@yzhao062 yzhao062 released this 23 Jun 03:42

Capture, replay, and audit AI agent decisions on one typed two-layer decision graph, read at three points in a run: PRE (lint a declared plan before deploy), LIVE (replay a decision under the state that is live now and recover), POST (rank a finished run and name the keystone).

New in 0.1.0

  • Audit report generation. AuditReport over all three pillars, with an agent-facing Markdown report (to_markdown, dependency-free) and a human-facing PDF (to_pdf, via the [report] extra).
  • Docs. PRE / LIVE / POST guides, a PRE coverage map (OWASP / CWE to graph-structural predicates), and an audit-report reference.
  • Two end-to-end examples that walk one payment through all three pillars.

The typed two-layer graph model is introduced in GRADE (arXiv:2606.22741).

Install

pip install auditable            # core: capture, replay, recovery
pip install "auditable[graph]"   # PRE lints + POST analyze_run
pip install "auditable[report]"  # the human-facing PDF

Docs: https://auditable-ai.readthedocs.io/