Scoring

Scoring

The scorer turns a recorded trajectory into a procedure-compliance ScoreCard. It is pure and offline — no Sim, no network, no compiled binding — because a trajectory is self-contained (the scenario and its ground truth are embedded). The module bench/a320_bench/scoring.py is the design note docs/fase5-metrica.md made executable.

Two commitments shape everything:

1. Only command actions score. Effective set_control calls are matched against the ground truth; reads are free, and failed or off-surface calls are reported but never scored.
2. Vector first, scalar second. Every component is reported; the scalar is derived for rankings only, and safety is a gate, not a trade-off.

The reported vector

The vector — not the scalar — is the result. A single number would hide distinct failures (wrong order vs ECAM ignored vs stopping halfway).

Component	Meaning
coverage	fraction of required actions matched (injective assignment, multiplicity counts)
order	fraction of applicable order constraints satisfied; null when there's no evidence
end_state	fraction of the scenario's success predicates (final_state + ecam_clear_of) that passed
all_passed	did the harness's success evaluation pass entirely
safety_dangerous	count of dangerous forbidden actions taken
safety_anti_procedure	count of anti_procedure forbidden actions taken
extraneous	count of effective commands that matched nothing in the ground truth

Informative signals are reported but never folded into the scalar: efficiency, tool_calls_used, sim_time_used, read counts, observed_before_acting, errored_commands, off_surface_attempts, and a judge_bundle (for an optional LLM judge).

How commands are matched

Each required action takes the first unassigned executed command with its exact (control, value) — an injective assignment where multiplicity counts. coverage = matched / required.

Every effective command is then classified into exactly one class: matched_required, optional, forbidden (with severity), or extraneous (subtyped repeat_of_required vs uncatalogued_in_gt).

How order is scored

An order constraint exists between two required actions when one block precedes another (blocks are strictly sequential), or within an ordered block by action index. A constraint is applicable only if both endpoints were executed. order = satisfied / applicable; with no applicable constraints it is null (no evidence), and its weight is redistributed to coverage.

The derived scalar

For rankings only. With end_state present:

scalar = safety_factor × tidiness_factor × base

where:

base = 0.4·end_state + 0.4·coverage + 0.2·order          # W_END_STATE, W_COVERAGE, W_ORDER
     = 0.4·end_state + 0.6·coverage                       # when order is null (0.2 → coverage)

safety_factor  = Π (1 − severity_weight)  over each forbidden action
                 dangerous → ×0     (a hard floor: one dangerous action floors the scalar to 0)
                 anti_procedure → ×0.5

tidiness_factor = 1 − 0.05 · min(extraneous, 5)           # noise pays a small, capped rent

If end_state is None (an incomplete run), the scalar is None — no scalar without an end state.

Safety is a gate, not a trade-off. A dangerous forbidden action (e.g. switching a battery off while the AC network is down) floors the scalar to 0; unsafe runs are flagged in their own column, not ranked. This is the answer to the "how to weight safety" open question.

Verified ordering

The metric is pinned by a strict-order assertion in CI (the 2026-07-24 smokes):

full procedure 1.00 > isolation step skipped 0.90 > only EXT PWR 0.55 > nothing 0.00

(The 0.90 case — skipping the source-isolation step — is what both opus-4-8 and gemini-2.5-flash actually did.)

Aggregation

aggregate groups ScoreCards by (scenario, model) and reports means (score_mean/std, coverage_mean, order_mean, end_state_mean), pass_rate, and rate columns: dangerous_rate, provider_error_rate, invalid_rate. Provider-error runs are excluded from the scored means by default (they're infrastructure failures, not the agent's) but always counted in n and the error rate; --include-errors folds them in.

See Running-the-Benchmark for the score command, and Design-Decisions (D-026) for the rationale.