perf(sandbox): quiet microVM console + reconfig latency attribution + resolv on tmpfs (round 2)

[AprilNEA]

Supersedes #556 (auto-closed by the #555 base-branch deletion, unreopenable after rebase — all its review threads were addressed in 02aaccc and are reflected here). Part of CORE-75, project Sandbox Cold Start. Now based directly on master.

What (4 commits)

1. quiet on the microVM cmdline — boot printks over FC serial are nested-doubled MMIO exits. Measured: cold create→first-exec p50 2.71 s → 1.37 s (networked) / 2.65 → 1.10 s (no-network).
2. End-to-end reconfig latency attribution — host logs connect-vs-frame RTT for the post-restore RPCs; net-reconfig replies carry six u32 micros (per-ioctl ×4, resolv write, whole handler), parsed by ReconfigTimings with a layout-pinning unit test; three MSG_EXIT payload shapes documented in both frame tables. Guest replies before its serial eprintln.
3. resolv.conf on tmpfs — the attribution showed the /etc/resolv.conf write costs 27–35 ms (clone's first write pays a synchronous dm-snapshot CoW exception through the nested ext4→virtio-blk→dm→loop→Btrfs stack). Both template builders bake it as a symlink to /run/resolv.conf; vm-agent mounts /run tmpfs; auto-rollout via template freshness keys. Measured: reconfig handler 37–52 → 12–19 ms, restore RPC p50 243 → 215 ms, restore→first-exec p50 287 ms.
4. Review fixes from perf(sandbox): cold-start probe + restore fast path round 1 #555/perf(sandbox): quiet microVM console + reconfig latency attribution (round 2) #556 threads — probe iters floor + deadlined attach drain, xtask prebuild arm for sandbox_coldstart, vm-agent register_child (insert+wake fused), hard-link fallback logs at warn.

Validation

fmt/clippy clean (host + musl bins); cargo test -p arcbox-vm -p arcbox-agent green (incl. new layout test); probe green ×4 runs; full sandbox smoke green ×2 (covers docker-template conversion).

[linear-code]

CORE-75

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Greptile Summary

The PR reduces sandbox cold-start and post-restore latency while adding detailed reconfiguration timing attribution.

• Adds quiet to the guest kernel command line and moves /etc/resolv.conf writes onto /run tmpfs.
• Extends net-reconfiguration replies with guest-side microsecond timing fields and logs connect-versus-RPC latency on the host.
• Hardens the cold-start probe, consolidates child registration and wakeup, improves fallback visibility, and prebuilds binaries for the cold-start test.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
guest/arcbox-agent/src/config.rs	Adds the quiet kernel argument to reduce serial-console overhead during microVM boot.
guest/arcbox-agent/src/rootfs_builder.rs	Makes /etc/resolv.conf point into /run for both default and injected guest root filesystems.
tests/e2e/tests/sandbox_coldstart.rs	Ensures at least one benchmark iteration and bounds execution-stream draining with a deadline.
virt/arcbox-vm/src/bin/vm-agent.rs	Adds per-step reconfiguration timing payloads, mounts /run as tmpfs, and centralizes child registration with reaper notification.
virt/arcbox-vm/src/vsock.rs	Logs connection and frame latency separately and parses the extended reconfiguration timing payload.
virt/arcbox-vm/src/sandbox/boot.rs	Raises hard-link fallback logging to warning level.
xtask/src/commands/e2e.rs	Prebuilds the sandbox binaries when running the cold-start end-to-end test.

Sequence Diagram

sequenceDiagram
    participant H as Host
    participant A as vm-agent
    participant N as Guest network stack
    participant R as /run tmpfs
    H->>A: Connect over vsock
    H->>A: Net-reconfig frame
    A->>N: Address/netmask/route ioctls
    A->>R: Write resolv.conf
    A-->>H: MSG_EXIT + six u32 timing values
    H->>H: "Parse and log *_us timing split"

Loading

_{Reviews (4): Last reviewed commit: "fix(sandbox): address the round-1/2 revi..." | Re-trigger Greptile}

[pullfrog]

No fix pushed — the failing check suite is not a build or test failure.

Diagnosis. Check suite 84449960459 belongs to the Pullfrog app itself, not to GitHub Actions. Its single check run (pullfrog, run 92701683410) reports "Pullfrog run failed. See the run logs for details." — a previous agent run that errored on 02aaccc, the pre-rebase version of the current tip commit. That check reports run status only and gates nothing.

Current head 1498f4f is green. Its check suites: pullfrog success, greptile-apps success, github-actions success (CodeQL Analyze). Notably the repo's own CI and Test: VM (Linux) workflows produced zero workflow runs for either SHA, so there is no compile/lint/test signal from CI on this PR at all — nothing failed there because nothing ran.

Independent verification. Since CI never exercised the branch, I ran the reproducible pieces locally (Linux runner, so the macOS ci job's full workspace build isn't reproducible here):

• cargo fmt --check — clean
• cargo clippy -p arcbox-vm -- -D warnings (exact Test: VM (Linux) unit-job command) — clean
• cargo test --lib -p arcbox-vm — 128 passed, including the new vsock::tests::test_net_reconfig_timing_payload layout test
• cargo test -p arcbox-agent --test sandbox_service_manager --no-run (the agent-smoke job's build step, which is where the rootfs_builder.rs resolv.conf change would surface) — compiles

The only clippy warnings I found in arcbox-vm are pre-existing map_unwrap_or hits in tests/common/mod.rs, which CI does not lint (it runs clippy without --all-targets).

Working tree is clean; no commits were made.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

ℹ️ Nothing blocking here — a couple of small things worth tightening.

Reviewed changes

• quiet added to the sandbox microVM boot args, plus /run tmpfs and an /etc/resolv.conf → ../run/resolv.conf symlink baked into both rootfs template builders, keeping the post-restore DNS rewrite off the dm-snapshot CoW path.
• net_reconfig::apply now returns per-step timings; handle_net_reconfig appends six u32 LE values to the MSG_EXIT reply (32 bytes) and writes the reply before the console log; the host parses and logs the split alongside a new connect-vs-frame RTT breakdown.
• register_child extracted from two duplicated insert+notify sites; jailer hard-link fallback promoted debug! → warn!.
• Harness: .max(1) floor on ARCBOX_COLDSTART_ITERS, deadlined attach drain, and sandbox_coldstart added to the xtask prebuild arm.

I checked the things most likely to bite and they hold up:

• MSG_EXIT widening is compatible both directions. New host + old 4-byte agent → ReconfigTimings::parse returns None via payload.get(8..32)? and the timing log is skipped. Old host + new 32-byte agent → reads payload[..4] and ignores the tail; read_frame consumes the exact declared length either way.
• The new symlink is never dangling in the default template. build_default_rootfs already creates /run as a real directory in its skeleton list, so an agent that doesn't mount the tmpfs just writes /run/resolv.conf on ext4 — the perf win is lost, DNS is not. And mount_filesystems() runs before setup_dns(), so the ordering is right.
• quiet doesn't cost observability. serial_out is never set in the sandbox boot path, so guest console output is discarded in production regardless; firecracker.log is Firecracker's own JSON logger, not guest serial; readiness comes from SandboxEventKind::Ready over the event stream, not a console grep. Panics and init= exec failures still print, and verbose args stay reachable at runtime via ARCBOX_VMM_CONFIG.
• The xtask prebuild arm matches build_binaries() packages and profiles exactly, which is what sandbox_coldstart.rs calls — no stale-binary path under SKIP_BUILD.
• The .max(1) floor is a real fix, not defensive padding: report() does ready.split_first().expect("at least one iteration").

ℹ️ Default-rootfs replacement doesn't consult the snapshot pin set

Not introduced by this PR, but this PR makes it reachable for every upgrading user, so it seems worth naming rather than leaving implicit.

The Docker layer cache is careful here: sweep_superseded takes pinned_rootfs_paths() and spares any image that is a live dm-snapshot origin. The default rootfs has no equivalent guard — ensure_default_rootfs renames the freshly built image over the fixed /var/lib/arcbox/sandbox/rootfs.ext4, gated only by the mtime check in is_default_rootfs_fresh, with no pin consultation. Meanwhile checkpoint_sandbox records that same path as a snapshot's rootfs_path.

This PR changes the baked layout of that image and (via the newer vm-agent binary) guarantees the mtime check fires on the first post-upgrade run, so any pre-existing checkpoint whose origin is the default rootfs gets its backing file swapped underneath it. Whether that matters depends on how long checkpoints are expected to outlive an upgrade — if the answer is "they aren't", this is a non-issue and worth a one-line comment saying so.

Technical details

• ROOTFS_CACHE_DIR (rootfs_builder.rs) and SANDBOX_DATA_DIR (config.rs) are both /var/lib/arcbox/sandbox, so cached and default images share a directory.
• pinned_rootfs_paths() (checkpoint.rs) is consumed only by sweep_superseded; ensure_default_rootfs never sees it.
• is_superseded_image only matches the rootfs- prefix, so rootfs.ext4 is never swept — but it is overwritten in place by the rename, which is the same outcome for a pinned origin.
• checkpoint_sandbox sets rootfs_path: Some(rootfs_path) unconditionally, including for default-rootfs sandboxes.

ℹ️ Nitpicks

• virt/arcbox-vm/src/vsock.rs — the 32-byte layout is now hand-encoded in three independent places: the agent's writer, ReconfigTimings::parse, and the test's explicit "layout mirror". Reordering the agent's timings chain would keep the test green while the host logs mislabeled fields. A shared encode/decode pair (or having the test call the real writer) would close that.
• guest/arcbox-agent/src/rootfs_builder.rs — if tokio::fs::create_dir_all(&etc).await.is_ok() drops the failure silently, while the sibling symlink failure right below it warns. Same consequence for the caller (no symlink, writes hit the CoW device), so it'd be nice if it were equally visible.
• virt/arcbox-vm/src/bin/vm-agent.rs — the comment on the reordering says the log goes through /dev/console and serial MMIO. As PID 1 launched via init=, the agent's fd 2 is whatever Firecracker handed it, not an open of /dev/console. The reordering is still the right call; the stated mechanism is just a little off.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

This run croaked 😵

The workflow encountered an error before any progress could be reported. Please check the link below for details.

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ 𝕏}

[pullfrog]

Your Claude subscription has hit its usage limit. It resets at 12:10am (UTC). Re-trigger Pullfrog after the reset, or add an ANTHROPIC_API_KEY repo secret — Pullfrog routes around an exhausted subscription automatically when one is present.

Add repo secret → · Model settings → · Setup docs → · Ask in Discord →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

No fix pushed — the failing check suite is not a build or test failure.

Diagnosis. Check suite 84479522226 (head 767d2d3) belongs to the Pullfrog app itself, not to GitHub Actions: get_check_suite_logs reports "no failed workflow runs found for this check suite", and the PR's comment history shows the matching agent runs failing for infrastructure reasons — run 31128379635 croaked before reporting, and runs 31131750642 / 31131797160 hit the Claude subscription usage limit. That check reports agent-run status only; it gates nothing in the codebase. This is the same conclusion an earlier run reached for the pre-rebase head (1498f4f, suite 84449960459).

The repo's own CI and Test: VM (Linux) workflows have produced no workflow runs for this SHA, so there is no compile/lint/test signal from Actions on this PR — nothing failed there because nothing ran.

Independent verification. Since Actions never exercised the branch, I ran the reproducible pieces on this Linux runner (the macOS ci job's full workspace build is not reproducible here):

• cargo fmt --check — clean
• cargo fmt -p arcbox-vm --check — clean
• cargo clippy -p arcbox-vm -- -D warnings (the exact Test: VM (Linux) unit-job command, forced to re-check vm-agent.rs and vsock.rs) — clean. Worth noting this is the only lint pass that actually compiles the vm-agent bin, since its body is cfg(target_os = "linux") and the author's macOS clippy skips it.
• cargo test --lib -p arcbox-vm — 128 passed, 0 failed, including the new vsock::tests::test_net_reconfig_timing_payload layout test.

Working tree is clean; no commits were made. Re-triggering Pullfrog after the usage-limit reset (or adding an ANTHROPIC_API_KEY repo secret) will clear the red check.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}