perf(sandbox): make boot readiness an event via guest vsock dial-out

[AprilNEA]

Stacked on #559 (← #558 ← master); merge those first, then retarget/rebase. Part of CORE-80, project Sandbox Cold Start (the ptp_kvm clock-deletion half of CORE-80 is deliberately not in this PR).

What

Replaces the readiness gate's detection-by-polling with a real event, using Firecracker hybrid-vsock guest-initiated connections (convention verified against FC docs/vsock.md: a guest connect to port P arrives at the host Unix socket {uds_path}_P):

• Host: ReadyListener binds {vsock_uds}_51 before InstanceStart (stale file removed; chown'd to the jailer uid/gid — FC needs write permission to connect; Drop deletes the per-boot socket on every path including boot failures).
• Guest: vm-agent, after both its listeners are bound, dials CID 2 port 51, writes one byte, closes. Failures are logged and tolerated (an old host without the listener must not break a new agent).
• Gate: accept() + one byte under AGENT_GATE_TIMEOUT; timeout/error takes the same fail_started_boot path. Clock sync stays on the boot path but strictly best-effort (10 s cap, warn on failure), pending ptp_kvm.

Hardware-validated (full stack: #558 + #559 + this)

Cold create→READY p50 1219 → 1118 ms (networked) / 1056 → 1065 ms (no-network), still converged with first-exec (1167/1121 ms) — the event fires the moment the agent is accepting, ~100 ms earlier than a completed clock round trip. Restore path untouched (229 ms p50). Full sandbox smoke green.

Validation

132 arcbox-vm unit tests + 1 integration green (new: ready-socket path pinning, ReadyListener accept/cleanup over a real UDS); clippy clean host + musl bins; fmt clean.

[linear-code]

CORE-80

[pullfrog]

This run croaked 😵

The workflow encountered an error before any progress could be reported. Please check the link below for details.

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ 𝕏}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Greptile Summary

The PR replaces cold-boot readiness polling with a guest-initiated hybrid-vsock event.

• Adds a guest readiness dial-out after the exec and file listeners are bound.
• Pre-binds and cleans up a per-boot host Unix listener for the event.
• Retains cold-boot clock synchronization as a best-effort operation.

Confidence Score: 4/5

The PR does not appear safe to merge at this HEAD because clock synchronization can still postpone Ready publication after the agent has signaled readiness.

Although the prior reply states that AprilNEA fixed the issue in stacked PR #566, this HEAD still awaits the capped clock-sync operation before persisting and publishing Ready, leaving the previously reported delay intact.

Files Needing Attention: virt/arcbox-vm/src/sandbox/boot.rs

Important Files Changed

Filename	Overview
virt/arcbox-vm/src/bin/vm-agent.rs	Adds the best-effort guest-to-host readiness notification after both service listeners are bound.
virt/arcbox-vm/src/sandbox/boot.rs	Replaces polling with the readiness listener and performs capped, non-fatal cold-boot clock synchronization.
virt/arcbox-vm/src/vsock.rs	Adds readiness-socket path derivation, listener lifecycle management, event consumption, and focused tests.

Sequence Diagram

sequenceDiagram
    participant Host as Sandbox host
    participant FC as Firecracker
    participant Agent as vm-agent
    Host->>Host: "Bind {vsock_uds}_51"
    Host->>FC: InstanceStart
    Agent->>Agent: Bind exec and file listeners
    Agent->>FC: Connect CID 2, port 51
    FC->>Host: Forward connection to UDS listener
    Agent->>Host: Write readiness byte
    Host->>Agent: Best-effort clock sync
    Host->>Host: Persist and publish Ready

Loading

_{Reviews (2): Last reviewed commit: "perf(sandbox): make boot readiness an ev..." | Re-trigger Greptile}

[pullfrog]

No CI failure to fix — the failed suite is Pullfrog's own

I could not find a CI failure introduced by this PR, because no CI ever ran on this branch.

Diagnosis

The check suite named in the trigger (84454504520) belongs to the pullfrog app, not GitHub Actions — get_check_suite_logs returned "no failed workflow runs found for this check suite". What concluded failure was a previous Pullfrog agent run on 1db7503, so this task is self-referential rather than a signal about the code.

Confirmed against the API for 1db7503:

Check suite	App	Conclusion
84454504246	greptile-apps	success
84454504520	pullfrog	failure (the trigger)
84454502104 / 84454503985 / 84454504766	cloudflare-pages / railway / coderabbitai	queued

GET /actions/runs?branch=perf/sandbox-ready-event returns total_count: 0. Every workflow in .github/workflows/ gates on pull_request: branches: [master, develop], and this PR targets perf/sandbox-ready-honesty — so ci.yml and test-vm-linux.yml are correctly inert until the stack is merged down and this PR is retargeted to master.

Verification instead of a blind fix

Since CI has never exercised this branch, I ran the gates it will face once retargeted. All green on 1db7503, no changes needed:

• cargo fmt -p arcbox-vm --check — clean
• cargo clippy -p arcbox-vm -- -D warnings — clean (this is the exact test-vm-linux unit-job command)
• cargo clippy -p arcbox-vm --bin vm-agent -- -D warnings — clean, covering the new signal_ready() dial-out
• cargo test --lib -p arcbox-vm — 131 passed, 0 failed, including both new tests: ready_socket_path_appends_the_port_suffix and ready_listener_accepts_the_dial_out_and_cleans_up
• cargo test -p arcbox-vm --test integration --test e2e --no-run — both targets compile

One pre-existing wart (not a blocker, not from this PR)

Under the stricter --all-targets, clippy flags four map(..).unwrap_or(false) lints and two dead-code items, all in virt/arcbox-vm/tests/common/mod.rs — a file this PR does not touch. CI does not pass --all-targets, so it stays green either way; worth a separate cleanup if you want that file lint-clean.

No commit was pushed: the working tree is clean and there is nothing here that a code change would address.

  ^{｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Reviewed 3 files across 1 commit (866ce18c), covering the new ReadyListener vsock abstraction, the boot-gate rewrite in sandbox/boot.rs, and the guest agent's signal_ready() dial-out.

✅ No new issues found.

I chased the parts of this that could plausibly be wrong and they all hold up:

• Firecracker hybrid-vsock contract. Guest-initiated connections land on {uds_path}_{port} with no handshake bytes (unlike the host-initiated CONNECT/OK direction), so reading a single byte in wait() is the right shape. Binding before InstanceStart is load-bearing — FC replies VIRTIO_VSOCK_OP_RST when no host listener exists — and the ordering is correct here.
• The guest can't wedge on a missing listener. That same immediate RST means signal_ready()'s blocking connect fails fast rather than hanging; combined with the eprintln!-and-continue handling, a lost dial-out degrades to a 35s gate timeout instead of a stuck agent. signal_ready() also runs after both create_vsock_listener calls, which already listen(fd, 128), so there's no accept race on the reverse direction.
• Both path derivations agree. Jailed (boot.rs:827-828, chroot-relative) and direct (boot.rs:862-868) both produce {uds}_51, and 51 doesn't collide with the existing 52/53.
• BootFailure { process: None, cow_handle: None } is right on the two new early returns — the FC process moves into the instance at boot.rs:713 and the cow handle at 880-883, both after these returns, matching the neighbouring returns at 908-916 and 1030-1035.
• Drop-unlink can't clobber a later generation's socket, since cleanup.rs:47 remove_dir_all(&vm_dir) runs before the slot is released at cleanup.rs:66.
• The host/guest compat comment checks out. The rootfs cache key embeds a content hash of the injected vm-agent (rootfs-{layer_key}-{agent_key}.ext4), and every resolve_template branch injects the current binary — so a host expecting the dial-out can't pair with an agent that doesn't send it.

Both new tests are genuinely falsifiable (the suffix test asserts the exact path, and the listener test covers stale-file cleanup, dial, and Drop unlink).

One behavioral note, not a defect: READY no longer proves the host→guest vsock direction works before it's published. The old sync_clock gate exercised a host-initiated connect, so a broken forward path surfaced at the gate; now the first host-initiated call happens after READY. Given the guest dial-out and the host listener share the same device, a one-directional failure seems unlikely enough to not be worth guarding — flagging it only because the gate's coverage narrowed.

I did not re-raise the clock-sync-gating point from the earlier greptile thread; it's resolved and deferred to stacked #566.

ℹ️ Nitpicks

• virt/arcbox-vm/src/vsock.rs:65-66 — the MSG_CLOCK_SYNC doc comment still reads "after snapshot restore, and as the cold-boot agent-readiness gate". This PR removes that second role, so the line is now stale. (Outside the diff, hence not inline.)

  ^{｜ View workflow run ｜ Using Claude Opus ｜ 𝕏}