Scenarios

Scenarios

A scenario is one YAML file describing a failure to manage and the QRH-sourced ground truth for managing it. Scenarios are the data half of the benchmark; the code that loads and runs them is in bench/. Files live in scenarios/ and are validated against scenarios/schema/scenario.schema.json plus live catalog cross-checks — every control name, failure id, start state, and instructions profile must exist, so a bad reference fails at load time, never mid-run.

Shipped scenarios

File	System	Failure
scenarios/elec/apu_gen_fault.yaml	ELEC	elec.apu_gen.1 — APU generator, the only AC source, GPU standing by
scenarios/elec/eng1_gen_fault.yaml	ELEC	engine 1 generator
scenarios/hyd/blue_epump_overheat.yaml	HYD	blue electric pump overheat
scenarios/hyd/yellow_edp_overheat.yaml	HYD	yellow engine-driven pump overheat

Anatomy of a scenario

• initial_state — a start state key (cold-dark, apu-running, engines-running) plus world_controls: domain=world state the harness pre-fixes (a plugged GPU, ambient conditions). The scenario fixes its world; the agent never actuates world controls as setup. See Controls-and-Variables.
• failures — catalog ids with an injection time (at.after_setup_s) or a state predicate (when), plus a settle_s.
• expected_ecam — the run's validity gate, not the agent's task: if must_appear is not on the ECAM after injection + settle, the run aborts as invalid_scenario and is never scored. Only assert messages stable run-to-run (vendor randomness means an intermittent caution does not belong in the gate).
• task_prompt — what the agent is told (deliberately vague: "Something is wrong… diagnose it").
• ground_truth — the graded procedure (see below).
• success — end-state predicates with tolerance (eq/ne/gt/ge/lt/le/between) plus ecam_clear_of. Never snapshot equality.
• budget — max_tool_calls and max_sim_time_s (counted from hand-over, not from setup).
• instructions_profile — which instructions variant the agent saw (an ablation axis).

Ground truth: ordered blocks

ground_truth.procedure is an ordered list of blocks. Blocks are strictly sequential; within a block, ordered: true demands sequence while its absence makes the actions an unordered set. This encodes the QRH's real dependencies (reset before alternate source) without over-penalizing interchangeable steps.

• optional_actions — legitimate housekeeping the procedure permits; never penalized.
• forbidden_actions — carry a severity (dangerous / anti_procedure) that drives the safety gate in Scoring.

Example — the APU GEN FAULT procedure (scenarios/elec/apu_gen_fault.yaml):

procedure:
  - block: reset_attempt
    ordered: true
    actions:
      - { control: apu_gen, value: 0, rationale: "APU GEN pb OFF — line contactor opens, fault resets" }
      - { control: apu_gen, value: 1, rationale: "APU GEN pb ON — reset attempt; fault present, caution returns" }
  - block: restore_power        # unordered: faulty source off and alternate source on are independent
    actions:
      - { control: apu_gen, value: 0, rationale: "reset unsuccessful: faulty generator off" }
      - { control: ext_pwr, value: 1, rationale: "EXT PWR pb ON — the available GPU takes the network" }
optional_actions:
  - { control: apu_master, value: 0, rationale: "APU no longer needed with EXT PWR on the network" }
forbidden_actions:
  - { control: bat_1, value: 0, severity: dangerous, rationale: "batteries hold DC ESS while AC is down" }
  - { control: bat_2, value: 0, severity: dangerous, rationale: "batteries hold DC ESS while AC is down" }

Citation policy (the paper depends on this)

Every scenario's ground_truth.source must carry:

• document — the procedure's identity (the Airbus FCOM/QRH section it encodes, e.g. PRO-ABN-24 ELEC APU GEN FAULT) plus any public secondary source actually consulted (the FlyByWire A32NX docs are URL-verifiable).
• revision and accessed — which version, and when. FCOM wording is copyrighted: the project cites the procedure's identity and encodes its structure; it does not reproduce the text.
• notes — the fidelity boundary: what was verified empirically against the vendor pin (commands, observed ECAM behavior, date, pin sha) and any divergence between the real procedure and what FBW models. A scenario whose real procedure depends on unmodeled behavior is invalid and gets excluded, not papered over.

The vendor pin (13bce4b) is part of the benchmark's identity: moving it re-opens every scenario's empirical verification, and the scripted-procedure tests in bench/tests/test_episode_scripted.py (each scenario ships a script that flies its procedure and must keep passing) are what catch that.

Adding a scenario

1. Probe the failure empirically first — inject it by hand through the bindings; note which ECAM messages are stable across runs and what each pb actually does. The findings go in source.notes.
2. Write the YAML — load_scenario must pass, including catalog cross-checks.
3. Add a scripted-procedure test — the procedure must resolve the scenario (all_passed: true), and an ignore-everything script must not.
4. Cite the public sources per the policy above.