v0.6.0: procedure-compliance metric, scorer, HYD coverage, experiment runbook - #95
Merged
Conversation
scoring.py turns a recorded trajectory into a ScoreCard with no Sim, no network, no compiled binding: the vector (coverage, order, end_state, safety, extraneous) is always reported and the derived scalar (safety x tidiness x (0.4 end_state + 0.4 coverage + 0.2 order), order weight redistributed when null) is for rankings only. Safety is a gate, not a trade-off: one dangerous forbidden action floors the scalar to 0. Only command actions score; reads, failed commands and off-surface attempts are reported but never scored. Matching is a deterministic canonical assignment (injective, multiplicity counts) that classifies every effective command as matched_required / optional / forbidden / extraneous, kept in the ScoreCard detail. docs/fase5-metrica.md is the full spec (matching, every component, the edge-case table, the contamination section) and D-026 records the shape decision. Refactor: parse_ground_truth/parse_success extracted from load_scenario so the scorer parses the embedded ground truth through the same path that validated it. Lazy exports keep the scorer binding-free. 21 synthetic tests, one per edge case, plus the import-without-binding guard and the acceptance-case strict ordering (full 1.00 > omitted 0.90 > ext-pwr-only 0.55 > nothing 0.00 -- the real smoke finding). 77 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…eat) (#88, #89) Cross-system coverage for #19: two hydraulic pump-overheat scenarios on engines-running, both probed empirically on pin 13bce4b (3 runs each). yellow_edp_overheat: isolate the overheating yellow EDP and let the PTU carry yellow from green; killing the PTU (anti_procedure) or shutting an engine (dangerous) is the wrong move. First scenario to exercise the scorer's mild severity. blue_epump_overheat: pure containment — isolate the pump and accept the loss of blue (inconsequential parked). Both overheat cautions LATCH (they do not retire on pb-off, as on the real aircraft), so success is state-based (pump commanded off + pressures where achievable + engines running), no ecam_clear_of. The green reservoir leak was probed as a sibling and DISCARDED (documented in the yellow scenario's source.notes): its PTU cross-compensation cascades into yellow, leaving no bounded stable state — a fidelity boundary, not a scenario. Scripted procedure + ignore tests for both; 59 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…ror (#85) score_trajectory takes raw dicts from a trajectory file, not the jsonschema-validated objects load_scenario yields, so a structurally corrupt record must not crash deep in a comprehension. Three spots that raised a bare KeyError/ValueError now raise a named ScoringError that says what is malformed: - a successful set_control (is_error=false) whose args lack a 'control' or carry a non-numeric 'value'; - a run whose meta.scenario has no ground_truth (names the run_id). _end_state also tolerates a null final_state/ecam_clear_of list, and a null meta.scenario no longer trips the id lookup. ScoringError is exported for parity with ScenarioError so slice G can skip corrupt files. Corruption stays as loud as read_trajectory: never a silent wrong number. New tests cover each malformed shape plus the previously untested defensible edges (empty procedure, forbidden/optional overlap, ecam-only success_eval, asdict round-trip of the nested CommandClass). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
…just <=1 executed) Review follow-up on #85: two required actions in one unordered block also yield zero applicable constraints; the parenthetical now says so. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
Procedure-compliance metric + pure scorer module (#85)
score PATH... [--json] [--detail] [--include-errors] turns recorded trajectories (files or run directories, recursive) into ScoreCards: a human table (per-run rows + per scenario x model aggregates with mean/std, pass/dangerous/error/invalid rates) or JSON for plots. Kept in cli_score.py, binding-free: a reviewer scores with pip install -e bench/ alone. Exit 2 on an unreadable/corrupt trajectory or no files found; a badly-scored run is a result, not an error. Tests generate the four acceptance fixtures in-test with the ScriptedAdapter over the real apu_gen_fault scenario and assert the strict score ordering end to end through score_file, plus --json/--detail/exit codes. Verified on the real Gemini smoke trajectory: scored 0.90 (coverage 0.75 for the omitted isolation step, order 1.00, end_state 1.00) -- the metric assigns the predicted value to the actual real-model run. 94 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
… paths (#86) cli.py imported a320_bench.episode (which imports the compiled a320_sim) and a320_bench.scenario at module top, so `a320-bench score` — the entry point a320_bench.cli:main — failed to even start on a reviewer's binding-free machine (pip install -e bench/ alone), defeating the slice's whole point. The narrow `import a320_bench.cli_score` check passed and hid it. Both imports are now deferred into the run/serve paths that need them, mirroring the existing lazy-litellm pattern; the score dispatch touches neither. _collect now collapses overlapping arguments (a file passed twice, or a directory plus a file inside it) by resolved path, so a run is never double-counted into the aggregate means. Added type hints to the CLI's public helpers. Tests: a binding-free score-dispatch regression (a320_sim/litellm blocked in sys.modules), path dedupe, good-file + empty-dir -> exit 0, and a truncated/incomplete run serializing as JSON null. 98 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
… not the leaf module) The #86 review caught a320-bench score pulling a320_sim via a module-top import in cli.py while the test only checked the clean leaf module. Rule: test the no-binding property through cli:main with the dependency blocked in sys.modules, and keep binding/optional imports lazy inside their branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
a320-bench score CLI + aggregation (#86)
Slice I of epic #20, docs only (blocked on provider keys for the real matrix): the metric doc gains an Experimental protocol section (>=2 models, N=10/cell power justification, ablation axes as existing levers — instructions profile, sampling, tool surface, QRH access — and what the vector reports), and bench/README.md gains the exact run+score runbook the matrix executes. No new code: run records, score measures, the matrix is a loop over the two. Keys stay in the environment, never in a trajectory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm
Experimental protocol + matrix runbook (#90)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration for the v0.6.0 milestone: the measurement half of the benchmark (#20) plus the HYD coverage that closes #19's cross-system gap.
What lands on main
bench/a320_bench/scoring.py) — vector always reported, derived scalar for rankings, safety as a gate. Full spec indocs/fase5-metrica.md, decision D-026.a320-bench scoreCLI + aggregation, binding-free (a reviewer scores withpip install -e bench/alone).13bce4b; the green reservoir leak was probed and discarded (PTU cascade, a fidelity boundary). Coverage now spans ELEC×2 + HYD×2.Validation
The scorer assigns 0.90 to the actual Gemini smoke trajectory (coverage 0.75 for the omitted isolation step, order 1.00, end_state 1.00) — the metric measures the trajectory-vs-end-state gap on a real run, not just in synthetic tests. 98 tests green, no LLM, no network.
Also: L-006 recorded (verify binding-free contracts by the entry point, not the leaf module).
After merge,
maingets the annotated tag v0.6.0: the metric, scorer and HYD coverage become the citable reference for the benchmark's measurement layer.🤖 Generated with Claude Code
https://claude.ai/code/session_014JCrwRbtN7UA5ijn13vPmm