Running the Benchmark

Running the Benchmark

a320-bench is the command-line harness for Phase 5. Three subcommands: run (recorded episodes against a real model), serve (stdio surface for an external client), and score (offline scoring). See Benchmark-Overview for the pipeline and Scoring for the metric.

Install

pip install -e bindings/   # a320_sim (not on PyPI)
pip install -e mcp/        # a320_mcp (START_STATES, create_server)
pip install -e bench/
pip install -e "bench/[providers]"   # only for `run` (pinned litellm)

score needs none of the binding, network, or providers — pip install -e bench/ alone scores a reviewer's results.

run — recorded episodes against a model

a320-bench run --scenario scenarios/elec/apu_gen_fault.yaml `
               --model anthropic/claude-opus-4-8 `
               --runs 3 --out runs/

Flag	Meaning
--scenario <yaml>	the scenario to run (required)
--model <litellm-id>	e.g. anthropic/claude-opus-4-8, gpt-... (required)
--runs N	episodes to run (default 1)
--out DIR	output directory (default runs/)
--sampling JSON	JSON dict passed to litellm.completion verbatim, e.g. '{"temperature": 0}'

Each run gets a fresh Sim, a fresh benchmark-profile MCP server (no inject_failure/clear_failure, plus report_done) over the SDK's in-memory transport, and its own JSONL trajectory under <out>/<scenario_id>/. The runner is privileged: it owns setup, world controls, failure injection, the ECAM validity gate, and ground truth (active_failures()), while the agent sees only the tool surface.

An episode ends on: report_done, budget exhaustion (max_tool_calls / max_sim_time_s), two empty turns (after one nudge), a provider error (partial trajectory kept), or a failed validity gate (invalid_scenario). Exit code is red on infrastructure problems — an invalid scenario or a provider error — so a paid batch of broken runs doesn't end green; an agent that merely failed the procedure is a result, not an error.

serve — stdio surface for an external client

a320-bench serve --scenario scenarios/elec/apu_gen_fault.yaml --result out.json

Same privileged setup + injection + validity gate, then the benchmark tool surface on stdio for an external MCP client that owns its own agent loop — above all claude -p on a Claude subscription. The harness's verdict lands in --result on shutdown. Here the trajectory is the client's transcript (e.g. --output-format stream-json), and the client's system prompt is a confound for model-vs-model baselines — those go through run.

score — offline compliance scoring

a320-bench score runs/                       # human table + per-(scenario,model) aggregates
a320-bench score runs/ --json > scores.json  # ScoreCards + aggregates for plots
a320-bench score runs/elec-apu-gen-fault/one.jsonl --json --detail

Flag	Meaning
--json	emit ScoreCards + aggregates as JSON
--detail	include per-command classification (with --json)
--include-errors	fold provider_error runs into the aggregate means (excluded by default)

Accepts trajectory .jsonl files or run directories. Pure and offline — no Sim, no network. See Scoring for what the numbers mean.

Providers

Adapter	Use
scripted	no LLM — the CI path; flies a scenario's scripted procedure
litellm_adapter	the canonical real-model path ([providers] extra, litellm pinned exact)

litellm is pinned to an exact version — it normalizes everything to the OpenAI format, so the version is part of a run's identity and is recorded in each trajectory's meta. CI installs without the extra; no test imports litellm (importorskip). See Design-Decisions (D-024, D-025).

Tests

python -m pytest bench/tests -q     # no LLM, no network