test(nasim): reproduce the selected public baseline through RAES

[Brad-Edwards]

Objective

Reconstruct and rerun a public NASim baseline through both the source-native path and the RAES adapter.

Source protocol

Use the tiny named benchmark as the executable example plus at least one standard scaled or generated benchmark and an upstream-supplied baseline agent. Freeze the exact scenario, agents, run counts, seeds/stochastic policy, reset rules, budgets, stopping/exclusion rules, evaluator, metric definitions, aggregation, and tolerances before the RAES-mediated collection.

Work

• Execute a source-native smoke/baseline to establish the predeclared oracle.
• Execute the same supported conditions through the adapter and versioned environment pack.
• Retain immutable inputs, attempt inventories, per-run provenance/evidence, failures/exclusions, aggregate measures, and validation reports.
• Compare authored-source, contract, execution-control, state/observation, outcome/evaluation, and disclosure tiers separately.
• Recompute aggregate results from retained evidence in a clean environment.
• Report missing source artifacts, uncontrolled stochasticity, compatibility patches, and unsupported RAES facts without tuning the oracle after observation.

Acceptance criteria

• The protocol and oracle are machine-readable and frozen before mediated runs.
• Every scheduled attempt has a retained terminal state and never-reused ID.
• Source-native and RAES-mediated results use the same definitions and identities.
• Aggregate measures are recomputable from retained evidence.
• Every equivalence tier has a pass, fail, or weakened result with cited evidence.
• The report makes no deterministic replay, cross-simulator equality, agent-ranking, or general scientific reproducibility claim beyond the data.
• The resulting artifact is usable by RAESystem/research#14 and the frozen study.
• Manual integration test — honest NASim execution: actually execute the source-native smoke and at least one RAES-mediated run against a live installed NASim (qualified runtime per build(nasim): qualify and pin the public source and experiment profile #32), so the reproduction pipeline is demonstrably run end-to-end and not merely validated against its frozen protocol.

---

Data capture layout

Retain this issue's reproduction evidence as a frozen, machine-readable bundle under packages/nasim_adapter/reproduction/<frozen-revision>/, so the research reproduction study can consume it without reformatting:

• protocol.json — the frozen matrix/oracle: conditions, agents, seeds, run counts, tolerances, and the exact/bounded/failed reproduction criteria, frozen before mediated runs.
• source-ledger.json — pinned upstream sources with versions and content digests, plus loss and exclusion disclosures.
• runs/ — per-run inventory with never-reused IDs, terminal state (valid/invalid/failed/excluded), and per-run provenance/evidence.
• aggregates.json — aggregate measures with 95% confidence intervals, recomputable from runs/.
• tiers.json — a pass/fail/weakened result for each equivalence tier (authored-source, contract, execution-control, state/observation, outcome/evaluation, disclosure) with cited evidence.
• validation-report.md — clean-environment recomputation and the one-command reproduction path.
• environment.json — exact software/source/pack/study identities and stated limitations.

Research consumer

This bundle is the study input: it qualifies the adapter in OpenRAE/research#14 and is a source for the reproduction artifact and paper in OpenRAE/research#20 (part of the frozen "Public Autonomous Cyber-Simulator Reproduction Study"). Export only reviewed, non-sensitive material; keep private, restricted, credential, provider-log, and native hidden-truth data out of the public bundle.