Benchmark Overview

Benchmark Overview

Phase 5 is the research payload. The contribution is not the aircraft model (that is FlyByWire's) — it is the evaluable environment, the failure-scenario suite, and the procedure-compliance scoring. This page frames how the pieces fit; the detail pages are Scenarios, Scoring, and Running-the-Benchmark.

The pipeline: run → record → score

flowchart LR
    S["Scenario YAML<br/>+ QRH ground truth"] --> R["Episode runner<br/>(a320-bench run)"]
    M["LLM via litellm"] --> R
    R --> T["JSONL trajectory<br/>(self-contained)"]
    T --> SC["Scorer<br/>(a320-bench score)"]
    SC --> V["ScoreCard<br/>vector + scalar"]
    V --> A["Aggregates<br/>per (scenario, model)"]

Loading

1. A scenario (Scenarios) declares an initial state, a failure to inject, the ECAM that must appear (a validity gate), the task prompt, and the QRH-sourced ground truth procedure.
2. The episode runner (a320-bench run) owns a privileged Sim: it sets up the world, injects the failure, checks the validity gate, and sits the agent in front of a benchmark-profile MCP server (no failure tools; report_done added). It records every turn and tool call.
3. The trajectory is a self-contained JSONL file — the scenario, ground truth, vendor pin, and versions are embedded, so scoring never re-simulates.
4. The scorer (a320-bench score) turns trajectories into procedure-compliance ScoreCards and per-(scenario, model) aggregates. It is pure and offline — no Sim, no network, no binding — so a reviewer scores results with pip install -e bench/ alone.

Two frontends into the runner

• a320-bench run — the runner owns the agent loop and talks to the model through litellm (pinned exact), recording the trajectory itself. This is the path for model-vs-model baselines.
• a320-bench serve — serves the benchmark tool surface over stdio for an external MCP client that owns its own agent loop (above all claude -p on a subscription). The trajectory is then the client's transcript.

See Running-the-Benchmark.

Determinism, honestly

The issue asked for "deterministic replay — same inputs, same trajectory." FBW's RNG is not seedable (the native path uses rand::rng(), OS-seeded, and ~24 files consume randomness — the APU intake flap alone rolls 6–12 s). Rather than patch the vendor (which would break the "zero patches" invariant and buy nothing the scorer needs), determinism is defined at the level of checkpoints with tolerance windows:

• the vendor pin is part of the benchmark's identity;
• scenarios are declarative and assert only ECAM messages that are stable run-to-run;
• an ECAM validity gate aborts a run as invalid_scenario if the promised failure never manifests;
• success predicates carry tolerance and never compare snapshots;
• N runs per (scenario, model) report variance;
• a seed field exists in every trajectory's meta from day one (currently always null) so a future Sim::set_seed wouldn't change the schema.

The LLM is, in any case, the dominant source of non-determinism. See Design-Decisions (D-022).

What the Phase-3 demo was, and wasn't

An LLM already resolved the APU GEN failure from the ECAM alone over the real MCP protocol — proof the environment can pose a resolvable, observable problem and close the observe→reason→act→advance loop. It is not a baseline: whoever drove it had already seen the answer. Blind evaluation with ≥ 2 models is exactly what separates "demo" from "benchmark," and that is Phase 5.

Status

The code half (bench/) and the data half (scenarios/) are implemented: scenario loading + validation, the in-process episode runner, trajectory recording, the scoring metric, and 4 shipped scenarios. Remaining: broader coverage and the multi-model baselines + ablations. See Roadmap and Design-Decisions (D-022 – D-026).