build(nasim): qualify and pin the public source and experiment profile

[Brad-Edwards]

Objective

Establish one immutable, legally usable, runnable NASim source and experiment profile for the RAES study.

Scope

• Select and pin the published NASim package/source, named benchmark scenarios, scenario generator, Gymnasium interface, benchmark scripts, and supplied planning/RL baseline agents.
• Record repository/release, package, dependency, Python, scenario, agent, evaluator, seed, and configuration identities and digests.
• Verify licenses, redistribution/attribution requirements, external downloads, model weights, and retained-output permissions.
• Reconstruct installation in a clean environment and execute reset, representative action/step, observation, reward/result, and termination behavior.
• Identify undocumented defaults, incompatible dependencies, known defects, and stochastic entry points.
• Choose the exact example scenario and public baseline protocol that downstream issues must implement.
• Isolate dependencies behind the nasim extra; use uv conflicts declarations where simulator stacks are mutually incompatible.
• Do not patch semantics silently. Any compatibility patch is pinned, explained, and tested against the unmodified source behavior.

Qualification semantics

The maintainer-selected simulator and experiment profile are admitted inputs to the RAES study. Qualification records the exact source and installation route, what RAES can attest or reproduce, stochastic-control coverage, known upstream defects, and limits on claim strength. Packaging gaps, incomplete determinism, compatibility limits, and source defects are disclosed limitations; they do not by themselves veto adapter implementation or require RAES to create behavior the simulator lacks.

Acceptance criteria

• One immutable runtime/source profile and one public experiment protocol are selected.
• Clean installation and a source-native smoke run pass through a documented supported route.
• Exact scenario, agent, evaluator, reset, seed, metric, and termination identities are known.
• License, attribution, redistribution, archive/maintenance, known-defect, and patch decisions are explicit.
• Base-only and unrelated simulator extras remain independent.
• The qualification record grades attainable RAES claims across source identity, configuration/control attestation, run evidence, and outcome reproduction, with every weakened or unsupported claim disclosed without changing the maintainer admission decision.

References

• https://github.com/Jjschwartz/NetworkAttackSimulator
• RAESystem/research#12

[Brad-Edwards]

🛠️ Picked up by /implement - driver claude, branch 32-qualify-nasim-source, 2026-08-02T14:59:14.253Z.

[Brad-Edwards]

gc workflow phase recorded: preflight (issue #32). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

[Brad-Edwards]

gc workflow phase recorded: plan (issue #32). Posted by the MCP server to enforce ordering between workflow steps (issue #794 MVP-2). Do not edit or delete — used by downstream tools to gate phase prerequisites.

Plan — qualify and pin NASim public source + experiment profile (#32)

Requirement-free maintainer-qualification run, mirroring the CyberBattleSim
profile-only pattern (#25): a backend-local evidence module under
raes_adapters.nasim (loader + qualification.json + public-protocol.md) plus
one nasim extra and a structural test. No adapter, backend manifest,
conformance profile, RAES SDL, or raes_adapters.base change is delivered
(guardrails: docs/decisions/nasim-qualification-guardrails.md). All evidence
below was produced by actually acquiring, installing (clean venv, PYTHONPATH
cleared, safe-path), and running NASim — no value is fabricated.

Selected immutable source

• Upstream Jjschwartz/NetworkAttackSimulator, tag v0.12.0, commit
  7c732bc4620d20a25b221a782adee29c2a89d800, tree 4eedd4cc3ed9c31b17281baf447e67fcaa826298.
• Publishable distribution nasim==0.12.0 (PyPI wheel sha256:4c059e64…,
  sdist sha256:c5da8be9…, both verified). MIT (LICENSE.md sha256:1d8084ef…).

Selected qualified runtime

• CPython 3.12.3, Linux glibc2.39. gymnasium 0.26.3 + numpy 1.26.4
  (both within NASim's declared ranges). This pin is deliberate — see defect 1.

Selected public experiment protocol

• Scenario: tiny static benchmark (nasim/scenarios/benchmark/tiny.yaml),
  3 hosts / 3 subnets, one exploit e_ssh (prob 0.8), one privesc pe_tomcat
  (prob 1.0), step_limit: 1000, sensitive hosts (2,0),(3,0).
• Construction: nasim.make_benchmark("tiny", fully_obs=True, flat_actions=True, flat_obs=True)
  (the documented bruteforce_agent.py tiny route). Flat action space n=18,
  Box obs shape (56,) float32.
• Baseline/evaluator: nasim.agents.bruteforce_agent.run_bruteforce_agent
  (exhaustive round-robin; no torch, no model weights).
• Gymnasium mapping: reset()->(obs,info); step()->(obs, reward, terminated=goal_reached, truncated=step_limit_reached, info).
• Seed: np.random.seed(20260802) — the only binding seed (see defect 2).

Observed smoke evidence (real)

• Representative step (action 0 = ServiceScan target=(1,0)): reward −1.0,
  terminated False, truncated False, step 1.
• Termination probe (supplied bruteforce, seed 20260802): steps 47, total
  reward 153.0, goal True.
• Truncation probe (unproductive action to step limit): steps 1000,
  terminated False, truncated True — proving truncated is the step-limit
  boundary, distinct from goal termination.

Findings recorded as known defects / limitations

1. Supplied agents break on gymnasium ≥ 0.27 — FlatActionSpace.get_action
   asserts isinstance(action_idx, int) (nasim/envs/action.py:698); gymnasium
   ≥ 0.27 makes Discrete.n/.sample() return np.int64, so bruteforce/random
   and the greedy paths of ql/dqn raise AssertionError. Qualified runtime pins
   gymnasium 0.26.3, the last release where Discrete.n is a Python int.
2. Action-success RNG is unbound by the env seed — success is drawn from the
   global np.random (network.py:87), never the gym-seeded self.np_random;
   make_benchmark(seed)/reset(seed) do not make runs reproducible. Only an
   explicit np.random.seed() does.
3. Static-benchmark seed ignored — make_benchmark_scenario calls
   load_scenario for static YAML benchmarks and ignores seed entirely.
4. Headless import requires tkinter — nasim/envs/render.py unconditionally
   import tkinter + matplotlib.use('TkAgg') at import (pulled in by
   environment.py), so import nasim needs the Tk system libraries even for
   headless use.
5. Model weight nasim/agents/policies/dqn_tiny.pt ships in the git repo
   (for the dqn baseline) but is excluded from the PyPI wheel; the selected
   tiny+bruteforce profile uses no weights.

Packaging (single distribution, one lock — ADR-003)

• Add nasim = ["nasim==0.12.0"] to [project.optional-dependencies]; refresh
  the single uv.lock. uv.lock currently holds no gymnasium/numpy/etc. and no
  other extra constrains them, so no conflicts declaration is needed. Base,
  cyberbattlesim, and cyborg extras remain independent (_verification_envs
  verifies each alone). raes_adapters.nasim imports none of the simulator
  stack, so CI never imports NASim (no tkinter needed in CI).

Files

• src/raes_adapters/nasim/__init__.py — loader (load_qualification,
  read_public_protocol), mirroring raes_adapters.cyberbattlesim.
• src/raes_adapters/nasim/qualification.json — record_version: "nasim-qualification/1"; source/runtime identities + digests, source_files,
  import-root trees (git 42-file + wheel 36-file, __pycache__-excluded),
  runtime_artifacts (nasim/gymnasium/numpy wheels + trees), protocol selection,
  dependencies (19, all permissive), stochastic_sources, legal, known_defects,
  patches [], maintenance, admission + claim_strength.
• src/raes_adapters/nasim/public-protocol.md — the experiment profile; its
  sha256 pinned in the record.
• pyproject.toml — nasim extra; uv.lock — refreshed.
• tests/test_nasim_qualification.py — structural assertions (source identity,
  bounded smoke, import-root attestation, protocol identities + stochastic
  streams, legal/patch dispositions, admission + extras["nasim"] == []… i.e.
  extras["nasim"] == ["nasim==0.12.0"]).
• README.md — nasim section; docs/decisions/nasim-qualification-guardrails.md
  • mkdocs.yml nav (already present from preflight) committed.

Claim strength (admission = admitted, maintainer-selection, target research#12)

source_identity attested; protocol_configuration attested;
execution_controls partial (unbound action RNG); run_evidence attestable;
outcome_reproduction stochastic-bounded (exploit prob 0.8 + agents require
gymnasium 0.26.3). Limitations disclosed; they bound claims, not the admission.

Verification

make verify (nox graph: hygiene, policy, lint, tool-tests, typecheck, tests,
distributions, docs). Branch carries no REQ/REP token → policy runs
--skip-requirement (genuine requirement-free maintenance).

[Brad-Edwards]

gc_codex_review — cycle 1 of 1 (pre-push) on issue #32 (branch 32-qualify-nasim-source)
Diff mode: inline — the complete diff was supplied in one prompt

Core review

Verdict: ship-with-fixes

The change is shaped as backend-local NASim qualification evidence: it keeps NASim out of the shared base layer, records a bounded source-native protocol, and avoids presenting qualification as adapter or RAES-conformance semantics. The decisive seam is the optional extra: the documented qualified runtime must be what the distributable extra actually resolves, otherwise the evidence and the install contract diverge.

Blocking findings (1):

1. [one-off] Make the NASim extra enforce its qualified runtime pins — pyproject.toml:56
   The extra declares only nasim==0.12.0, while the updated lock resolves its transitive dependencies to gymnasium==1.3.0 and numpy==2.5.1. The qualification record and protocol explicitly require gymnasium 0.26.3 / numpy 1.26.4 because later Gymnasium versions make NASim's supplied bruteforce agent fail. Consequently, raes-adapters[nasim] installs a known-incompatible runtime rather than the admitted protocol. Declare the qualified pins in the extra (and regenerate the lock/evidence hashes) so installation honors the published qualification contract.

Security review

Verdict: ship

This change adds backend-local NASim qualification evidence, a narrowly scoped optional dependency extra, and documentation without introducing an HTTP surface, user-controlled input path, credential handling, or execution of untrusted content. The resource loader reads only package-owned static files, while the supply-chain additions are registry-resolved and hash-locked. The design seam is appropriately confined to the NASim package and leaves the existing shared security boundaries unchanged.

No blocking findings.

[Brad-Edwards]

gc_codex_review pre-push cycle 1 of 1 complete for issue #32 on branch '32-qualify-nasim-source'. Posted by the MCP server to enforce the pre-push hard-cap-1 contract (issues #796, #804, #906). Do not edit or delete — used by the next gc_codex_review (uncommitted) invocation to count cycles.

[Brad-Edwards]

Review decision record — codex cycle 1 (issue #32)

Reviewer: codex
Cycle: 1

Architectural read:

Core reviewer: The change is shaped as backend-local NASim qualification evidence: it keeps NASim out of the shared base layer, records a bounded source-native protocol, and avoids presenting qualification as adapter or RAES-conformance semantics. The decisive seam is the optional extra: the documented qualified runtime must be what the distributable extra actually resolves, otherwise the evidence and the install contract diverge.

Security reviewer: This change adds backend-local NASim qualification evidence, a narrowly scoped optional dependency extra, and documentation without introducing an HTTP surface, user-controlled input path, credential handling, or execution of untrusted content. The resource loader reads only package-owned static files, while the supply-chain additions are registry-resolved and hash-locked. The design seam is appropriately confined to the NASim package and leaves the existing shared security boundaries unchanged.

Blocking findings: 1

Finding 1 — one-off

• ID: F1
• Title: [core] Make the NASim extra enforce its qualified runtime pins
• Location: pyproject.toml:56
• Decision: fix
• Rationale: The extra declares only nasim==0.12.0, while the updated lock resolves its transitive dependencies to gymnasium==1.3.0 and numpy==2.5.1. The qualification record and protocol explicitly require gymnasium 0.26.3 / numpy 1.26.4 because…

[Brad-Edwards]

Test-quality review — cycle 1 (pre-push, manual)

Run manually because gc_test_quality_review_cycle was blocked mid-run by the MCP
identity guard (implement_repo_identity_changed) in this linked-worktree /
concurrent-run setup. Filed upstream: autarchy-ai/Ground-Control#1502.

Scope: tests/test_nasim_qualification.py (new, 6 tests) and the digest re-pin
in tests/test_cyborg_qualification.py.

Assessment: The suite follows the accepted CyberBattleSim qualification-test
pattern — it locks the immutable evidence record and adds three genuine
cross-consistency checks that are not tautological:

• sha256(public-protocol.md) recomputed == record.protocol.sha256 (guards
  doc↔record drift);
• record.packaging.extra_declaration == pyproject optional-dependencies.nasim
  (guards record↔packaging drift);
• record.dependency_resolution.qualified_pins ⊆ the nasim extra (guards that
  raes-adapters[nasim] installs exactly the admitted runtime).

No over-mocking, no assertion that merely restates production constants, and the
smoke/termination/truncation facts are the checked-in evidence (re-running NASim in
CI is out of scope per the qualification guardrails). Bounded-output invariants are
asserted (raw_observation/info absent from the record).

Finding fixed this cycle: added the qualified_pins ⊆ extra cross-check,
directly reinforcing the codex finding that the extra must install the qualified
runtime.

Decision: fix (applied). No open test-quality findings.

[Brad-Edwards]

Pre-merge readiness — ready for review & merge (#32)

Plain-English outcome: RAES now has one pinned, legally-usable, runnable NASim
source and one public experiment profile to build a NASim adapter against.
pip install raes-adapters[nasim] installs the exact qualified runtime
(NASim 0.12.0 + gymnasium 0.26.3 + numpy 1.26.4); the checked-in record fixes the
source, wheel, and protocol identities and grades what RAES can attest. Four
upstream limitations are disclosed (supplied agents break on gymnasium >= 0.27;
action-success RNG unbound by the env seed; static-benchmark seed ignored; tkinter
required to import). No adapter, manifest, or outcome-equivalence claim is made
here — that is downstream.

PR: #68 (base dev), head 44d988e.

Automated gates — all green:

• CI: Fast checks, Policy, Tool tests, Typecheck, Tests, Distributions, Docs,
  CodeQL, Lint PR title, GitGuardian, PR Gate — pass.
• SonarCloud strict gate: OK (new_violations 0, new_coverage 100%, duplication
  0.0%, reliability/security/maintainability A).
• make verify + make policy completion gate green locally on the final tree.

Pre-push reviews: Codex — 1 finding (the nasim extra must pin the qualified
runtime) fixed; test-quality — run manually, consistency checks added. A follow-up
refactor commit extracts the shared qualification-evidence loader
(raes_adapters._qualification) to remove the SonarCloud new-code duplication the
nasim loader introduced.

Tooling note: the GC MCP identity guard refused this linked worktree mid-run
(implement_repo_identity_changed) because the concurrent issue-#39 run in the
sibling worktree rewrites the shared .git/config origin (RAESystem <-> OpenRAE).
Filed upstream autarchy-ai/Ground-Control#1502; the test-quality review, publish,
PR, CI/Sonar watch, and this readiness record were therefore driven manually.

Next: ready for your review and merge into dev. After merge, re-run
/implement 32 for Phase E (requirement transition — N/A, this is requirement-free;
traceability reconciliation; final report; issue close). Because issue #39
concurrently edits pyproject.toml, uv.lock, and
src/raes_adapters/cyborg/qualification.json, whichever PR merges second will need
a base-sync conflict resolution (re-run uv lock, re-pin the CybORG
uv_lock_sha256).