Skip to content

wasm-backend AC3.2 boundary: inline-fixture VM tests bypass the wasm parity hook #617

@bpowers

Description

@bpowers

Summary

The wasm-backend parity gate (wasm-backend.AC3.2 -- "every core-simulation corpus model runs through both the VM and the wasm backend") is enforced by an inline wasm_parity_hook that lives inside the shared corpus helpers simulate_path_with_excluding / simulate_mdl_path / simulate_mdl_path_with_data (src/simlin-engine/tests/simulate.rs) and the systems-format helper (src/simlin-engine/tests/simulate_systems.rs). Any VM-simulated model that flows through one of those helpers also runs through the wasm backend, and an Unsupported outcome is now a hard failure.

However, a number of tests in tests/simulate.rs build inline string-fixture models and run them through the VM with bespoke assertions without calling those helpers, so they never reach the wasm hook. A literal reading of AC3.2's success text ("every XMILE, MDL, and systems-format model in the corpus", phase_08.md line 22) would include these inline fixtures, so they sit outside the wasm gate by construction.

This is filed as a deliberate-coverage / AC-boundary note identified during the Phase 8 final review of the wasm-backend. It is not a defect -- the reviewer approved the phase for merge. It documents the boundary so a future session can decide whether to tighten it.

Concrete examples (verified)

Inline-fixture VM tests that bypass the hook (each calls open_vensim/open_vensim_with_data -> compile_vm -> Vm::new -> run_to_end with hand-written assertions, never simulate_*_path / wasm_parity_hook):

  • simulates_except_basic_mdl (tests/simulate.rs:1306)
  • na_existence_test_and_arithmetic_finite (tests/simulate.rs:1384)
  • the GET DIRECT CSV trio simulates_get_direct_data_scalar_csv / simulates_get_direct_constants_scalar_csv / simulates_get_direct_lookups_scalar_csv (tests/simulate.rs:3286 / 3349 / 3389)
  • several inline-fixture simulates_macro_* tests (e.g. simulates_macro_multi_expression_mdl, tests/simulate.rs:3566, which adds a namespace-isolation assertion the helper can't express)

Why the practical impact is small (verified)

Every wasm-supported feature these inline tests exercise is independently routed through the hook by a real corpus model, so no wasm-supported feature rests only on a non-hooked inline test:

  • EXCEPT: simulates_except (:1288) and simulates_except2 (:1293) both call simulate_mdl_path(...) (-> wasm_parity_hook).
  • GET DIRECT (data/constants/lookups/subs): directdata.mdl / directconst.mdl / directlookups.mdl / directsubs.mdl all go through simulate_mdl_path_with_data(...) (:3265-:3280), which ends in the hook.
  • Macros: many simulates_macro_* tests route through simulate_mdl_path (e.g. simulates_macro_expression_mdl, simulates_macro_stock_mdl, simulates_macro_cross_reference_mdl).
  • :NA: finite-sentinel arithmetic and basic vector ops / allocation likewise have corpus-routed simulate_*_path tests.

Why it matters

This is internal test-coverage texture, not user-facing. It matters only for the strength of the AC3.2 invariant: the gate's literal scope ("every model in the corpus") is broader than its implemented scope ("every model the VM simulates through the simulate_*_path / systems helpers in the default suite"). The implementer's notes already scope it this way deliberately (phase_08.md line 29: "The end-state gate applies to models the VM actually simulates in the default suite"), so the gap between literal AC text and implementation is intentional and reasonable -- but undocumented outside the plan.

Component(s) affected

  • src/simlin-engine/tests/simulate.rs (inline-fixture tests + the wasm_parity_hook / simulate_*_path helpers)
  • src/simlin-engine/tests/simulate_systems.rs (systems-format helper)
  • src/simlin-engine/tests/test_helpers.rs (wasm_results_for / ensure_wasm_matches, the alternative hook entry points)

Possible approaches (optional future enhancement)

  • Route the inline-fixture VM tests through the wasm hook too -- either by adding a wasm_parity_hook(&datamodel_project, &expected, &[]) call after their VM assertions, or by asserting parity directly via the wasm_results_for helper in test_helpers.rs. That would make AC3.2 hold in its strongest, most literal form.
  • Alternatively, leave the boundary as-is and simply document (in the engine CLAUDE.md test notes and/or a comment near the inline fixtures) that the wasm parity gate covers only the simulate_*_path/systems-helper-routed corpus, so the boundary is discoverable without reading the implementation plan.

How it was discovered

Identified during the Phase 8 final review of the wasm-backend (branch wasm-backend-poc). The reviewer flagged it as a deliberate coverage-boundary worth tracking and approved the phase for merge. The "small impact" claim above was independently verified by spot-checking simulates_except_basic_mdl, the GET DIRECT CSV tests, and the macro fixtures, and confirming each feature is covered by a hooked corpus model elsewhere.

Metadata

Metadata

Assignees

No one assigned

    Labels

    engineIssues with the rust-based simulation enginehygieneToil, but its useful to get get too behind on itrustPull requests that update Rust code

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions