Repository navigation
auditable 0.3.0
0.3.0 turns auditable into a library of auditing methods for AI agents, with one interface per family and the ability to read public agent traces through agentpile.
Added
auditable.methods: 43 methods in four families, each with one interface. Theconfigfamily (8 methods) audits an agent configuration before a run,online(5) raises causal alarms while a run streams, andrun(16) andstep(14) detect a failed run and localize its decisive step afterwards. A registry (list_methods,get_method) returns each method'sMethodCard, which records its fit scope, fidelity to its source, required fields, extras and citations. The library holds baselines from the literature (PyOD and PyGOD detectors, GUARDIAN-Lite, G-Safeguard-Lite, the Who&When LLM-judge protocols, OWASP-style configuration rules) beside the methods introduced with GRADE and CatchBench; every method that reproduces a CatchBench 0.1.2 board row matches it sample by sample.auditable.featurize: trace normalization for step lists, run dicts, typed graphs,SessionGraphandTraceViewinputs, and the features the methods compute (the GRADE structural layers, the Who&When step matrix, the step dependency graph, PyG encodings).auditable.llm: aprompt -> replybackend protocol,FixedResponseBackendfor tests and replays,OpenAICompatibleBackend(standard library only), and a prediction cache.auditable.from_agentpile,configs_from_agentpile,step_targets,run_targetsandauditable.datasets.load_tracesread agentpile records (public datasets, trace files, captured runs) into the methods' inputs and labels.convention="grade"withdeps="grade"reproduces the steps, dependencies and graphs of GRADE's converters (commit 3839a57) on seven corpora;convention="generic"(the default) makes one step per message and one per tool call.- Extras
tabular-ad,graph-adandagentpile; the core install still has no third-party dependency. docs/methods-contract.md, the contract a method follows and the behaviours a pinned consumer such as CatchBench relies on.
Behaviour of the Conversion
- In
convention="generic", content given as a list of parts (dicts with a stringtype, possibly mixed with bare strings) keeps its text and writes every other part (an image, a document) as[<type>], dropping that part's other fields;meta["report"]counts those parts and says how each step of the record maps into the trace. convention="grade"raisesValueErrorfor a run from a dataset revision other than the one its projection was checked on.- A run with subagent trajectories is refused by default; the error names the two ways to read it,
from_agentpile(run.subagents, ...)andsubagents="skip".
Deprecated
ActionGate,ReferenceLedgerandDataAuditoremit aDeprecationWarningwhen constructed and will be removed in a later release.