feat(sandbox): network-identity-invariant snapshots

[AprilNEA]

Stacked on #561 (← #560 ← #559 ← #558 ← master); merge in order. Part of CORE-81, project Sandbox Cold Start.

What (4 commits)

Every sandbox guest now boots the identical link-local identity (ip=169.254.100.2::169.254.100.1:255.255.255.252::eth0:off); the pool IP the rest of the system sees is applied host-side, per TAP. Snapshots become network-agnostic: a network_override restore of an invariant snapshot needs zero guest-side work — the eth0-reconfig vsock RPC and its resolv rewrite are skipped entirely (the reconfig wiring stays for legacy snapshots, gated by a net_invariant flag in snapshot metadata, serde(default) so old metas read false; chained checkpoints record the guest's actual addressing).

Mechanism (chosen after surveying what the System VM actually ships — iptables-legacy 1.8.13 + busybox, no nftables, no NETMAP/CONNMARK in the kernel): per-TAP mark-from-pre-NAT-destination in mangle + DNAT to the fixed guest IP; SNAT back to the pool IP in nat INPUT (so DNS/relays keep seeing pool IPs) and POSTROUTING (mark-scoped, so real client IPs are never rewritten); post-DNAT routing via per-sandbox fwmark fib rule + table (mark = table = pool-ip u32) programmed over hand-rolled rtnetlink (busybox ip fwmark support is build-dependent). Expose DNAT targets the guest IP with a mangle MARK companion; boot rules gain an explicit vmtap→pool isolation drop pair, making the "sandboxes are deliberately isolated" claim enforced rather than incidental. vm-agent is untouched — it derives DNS from ip=, so resolv.conf is invariant automatically.

Also updates the sandbox smoke's fresh-net assertions to the new contract (guest carries the invariant IP; pool IPs never appear in-guest).

Hardware-validated (full 5-layer stack, probe + smoke)

Pool-hit restores with the net RPC deleted: best-case total 83–88 ms (spawn 17–22 + stage 0 + load 4 + guest_cfg ≈58 — guest_cfg is now the clock RPC alone); probe p50 128 ms with back-to-back-restore variance attributable to pool refill I/O racing the next resume (worst-case pattern; noted for CORE-78 follow-up). Full sandbox smoke green: both restore modes, expose, file I/O, docker template.

Review notes / risks for reviewers

• rp_filter interplay handled via src_valid_mark; mangle LOCAL_OUT reroute-on-mark and LOCAL_IN SNAT are standard kernel semantics but called out for scrutiny.
• A sandbox reaching its own pool IP now drops (previously looped in-guest); sandbox↔sandbox via pool IPs is explicitly dead — both aligned with the isolation contract but observable changes.
• kube-proxy mark bits collide only beyond ~16k concurrent sandboxes (documented at the fwmark helper).

Validation

151 arcbox-vm tests green (new: exact NAT rule argv pinning, rtnetlink encodings, snapshot serde compat, expose both-halves removal, sweep matching); clippy clean host + musl (zero new warnings on arcbox-agent's changed lines); fmt clean.

[linear-code]

CORE-81

[pullfrog]

This run croaked 😵

The workflow encountered an error before any progress could be reported. Please check the link below for details.

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ 𝕏}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Greptile Summary

The PR makes sandbox snapshots independent of external pool addresses by giving every guest a fixed link-local identity and translating each sandbox’s pool identity on the host side.

• Adds per-TAP marking, NAT, policy routing, isolation, and cleanup for invariant networking.
• Records network invariance in snapshot metadata while retaining legacy snapshot reconfiguration.
• Updates expose-port handling, lifecycle paths, tests, and API documentation for the new identity contract.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains within the scope of this follow-up review.

No blocking failure remains.

Important Files Changed

Filename	Overview
virt/arcbox-vm/src/network/invariant.rs	Implements the per-TAP NAT, marking, policy-routing, sysctl, and teardown machinery for fixed guest identities.
guest/arcbox-agent/src/agent/linux/port_forward.rs	Extends exposed-port mappings with paired MARK and DNAT rules for invariant-addressed sandboxes.
virt/arcbox-vm/src/snapshot.rs	Adds backward-compatible snapshot metadata identifying network-invariant guests.
virt/arcbox-vm/src/sandbox/checkpoint.rs	Records the guest’s effective network identity when producing checkpoints.
virt/arcbox-vm/src/sandbox/lifecycle.rs	Skips legacy guest-side network reconfiguration when restoring invariant snapshots.
guest/arcbox-agent/src/init.rs	Updates guest boot forwarding and isolation rules for the fixed link-local identity.
tests/e2e/src/sandbox.rs	Updates sandbox network assertions to require the fixed guest address and exclude pool addresses.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Client[Client or host relay] --> PoolIP[Sandbox pool IP]
    PoolIP --> Mark[Mark packet for sandbox TAP]
    Mark --> DNAT[DNAT to fixed guest IP]
    DNAT --> Route[Per-sandbox fwmark route]
    Route --> TAP[Sandbox TAP]
    TAP --> Guest[Guest 169.254.100.2]
    Guest --> SNAT[Mark-scoped SNAT]
    SNAT --> Client

Loading

_{Reviews (2): Last reviewed commit: "test(e2e): assert the invariant guest id..." | Re-trigger Greptile}

[pullfrog]

No repo CI failure to fix on this PR

Diagnosis: the failing check suite is not a repository CI job, so nothing in the branch needed fixing.

• get_check_suite_logs for suite 84460393250 returns no failed workflow runs found on both attempts — the suite contains no GitHub Actions run.
• No repo workflow can run on this PR. Every pull_request trigger in .github/workflows/ is filtered to branches: [master, develop] (ci.yml, test-vm-linux.yml, codeql.yml); the rest are tag/schedule/workflow_dispatch only. This PR targets perf/sandbox-pool, so ci.yml and test-vm-linux.yml never fire.
• The only recorded failure on the PR is an earlier Pullfrog run that croaked before reporting progress (run 31127967280) — an infrastructure error, not a code failure.

Verified the branch is green anyway

Ran the exact commands CI will execute once this stack reaches master, at 172e3577:

Command	Result
cargo fmt --check	clean
cargo fmt -p arcbox-vm --check	clean
cargo clippy -p arcbox-vm -- -D warnings	clean
cargo test --lib -p arcbox-vm	150 passed, 0 failed
cargo test --test integration -p arcbox-vm --no-run	builds
cargo clippy -p arcbox-agent --all-targets	exit 0 (pre-existing pedantic warnings only)
cargo xtask --help	exit 0

Working tree is clean; nothing was committed or pushed.

Unrelated pre-existing issues found (left alone)

• virt/arcbox-hypervisor/src/linux/vcpu.rs fails to compile on Linux (E0432), which blocks cargo check -p arcbox-e2e there. Untouched by this PR and invisible to CI, which builds on macos-26.
• virt/arcbox-vm/tests/common/mod.rs and the is_root() helper in network.rs trip clippy::map_unwrap_or, but only under --all-targets, which neither CI job passes.

One note on Greptile's flag: arcbox-vm.workspace = true in guest/arcbox-agent/Cargo.toml [dev-dependencies] is unconditional, while the real dependency sits under [target.'cfg(target_os = "linux")'.dependencies]. Every module that references arcbox_vm (error, config, sandbox, init::platform, agent/linux/*) is #[cfg(target_os = "linux")]-gated, so the dev-dep looks redundant — but I could not verify a macOS cargo test -p arcbox-agent here, so I did not remove it.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Important

The new invariant::remove teardown step is ?-propagated ahead of TAP destruction, so a transient iptables/netlink failure now strands the TAP device and permanently leaks the pool address. Details inline.

Reviewed changes — no prior Pullfrog review existed on this PR, so this covers all 4 commits. The 172e357 → 0360e72 range-diff is rebase-only (11 added lines, all context shifts around chroot_owner/pool_slot_id now that #561 has landed), so scope is the full diff.

• Verified the 7-rule per-TAP NAT contract in virt/arcbox-vm/src/network/invariant.rs against the netfilter hook order it depends on. mangle runs before nat in the same hook, so marking to-sandbox packets from their pre-NAT destination is sound, and the mangle OUTPUT + nat OUTPUT pair correctly re-routes locally-originated replies after the destination rewrite forces a route re-lookup. Gating the POSTROUTING SNAT on -s 169.254.100.2/32 rather than the mark alone is the right call — the mark is shared with to-sandbox packets that carry a real client source — and both properties are pinned by exact-match tests.
• Traced the DNS chain end to end, since the design leans on "the nameserver is the fixed gateway everywhere". It holds: the sandbox's vm-agent writes nameserver {ip_param.gateway} from the ip= cmdline, boot.rs now bakes GUEST_GATEWAY there, and the System VM's resolver binds 0.0.0.0:53. Legacy restores still resolve because the pool gateway is the one address exempted from the new mangle isolation DROP.
• Confirmed crash recovery needs no persisted net_invariant. invariant::remove derives every rule spec from (tap_name, pool_ip) alone, so sweep_orphans replays teardown correctly for an invariant TAP it holds no record of. That state-free property is load-bearing and worth preserving.
• Checked the fwmark == table id == u32::from(pool_ip) choice. Unique per active sandbox, and the documented kube-proxy caveat is accurate: the 0x4000/0x8000 masks land in the third octet, so the first ~16k addresses of a /16 pool are conflict-free.
• Reviewed the hand-rolled rtnetlink encoder in network/rtnetlink.rs. Attribute padding, the NLM_F_EXCL-plus-tolerated-EEXIST idempotency pattern for rule adds, NLM_F_REPLACE for the table route, and NLMSG_ERROR ack decoding all look correct and are unit-tested at the byte level.
• Confirmed the compatibility flag flows through the whole stack — #[serde(default)] on SnapshotMeta.net_invariant means pre-PR snapshots deserialize to false and take the LegacySnapshot TAP shape with the guest reconfig RPC intact.

ℹ️ TapMode::Invariant is selected on a different condition than net_invariant is recorded

sandbox/lifecycle.rs:126 activates the TAP as TapMode::Invariant unconditionally whenever a network allocation exists, but line 195 records net_invariant = network.is_some() && !spec.boot_args.contains("ip="). When a caller supplies their own ip=, the TAP gets the invariant NAT (DNAT'ing to 169.254.100.2) while the guest boots a different address, and any snapshot taken from it would restore as LegacySnapshot — changing the TAP's local address out from under it.

This is unreachable today because boot_args is never populated from a request, which is why it's informational rather than a bug. The question for you is whether the contains("ip=") guard is meant to be a supported escape hatch (in which case activate needs the same condition) or is vestigial (in which case dropping it makes the two sites agree by construction).

Technical details

# `net_invariant` and `TapMode` disagree on the caller-supplied `ip=` case

## Affected sites
- `virt/arcbox-vm/src/sandbox/lifecycle.rs:126` — `activate(net, TapMode::Invariant)` with no `boot_args` condition
- `virt/arcbox-vm/src/sandbox/lifecycle.rs:195` — `net_invariant` gated on `!spec.boot_args.contains("ip=")`
- `virt/arcbox-vm/src/sandbox/boot.rs` — appends the fixed `ip=` param only when `spec.boot_args` lacks one

## Required outcome
- The condition that decides the TAP's addressing mode and the condition that records it in instance/snapshot metadata must be the same expression, so no reachable input can produce an invariant TAP paired with `net_invariant: false`.

## Open questions for the human
- Is a caller-supplied `ip=` a supported configuration, or is the guard vestigial now that the sandbox API always passes an empty `boot_args`? If vestigial, deleting it (and the `boot.rs` branch) removes the divergence outright.

ℹ️ Nitpicks

• virt/arcbox-vm/src/network/rtnetlink.rs:175 — libc::recv isn't retried on EINTR. A signal arriving during install() would fail the whole TAP activation (and, via activate's rollback, the sandbox creation) for a spurious reason. A loop around the recv that continues on EINTR would close it.
• guest/arcbox-agent/Cargo.toml:65 — arcbox-vm is added to the unconditional [dev-dependencies], while the real dependency sits under [target.'cfg(target_os = "linux")'.dependencies] (line 54). Since the tests that consume it are Linux-gated anyway, [target.'cfg(target_os = "linux")'.dev-dependencies] would match the existing shape and keep the Firecracker VMM crate out of macOS dev-host test builds.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}