fix(vmm): keep stopped VMs' CIDs reserved across a reload

[kvinwang]

Problem

reload_vms_sync() (the ReloadVms RPC) clears the CID pool and rebuilds it from the supervisor's list of running processes. A stopped VM does not appear in that list, but it still records its CID in vm.config.cid and keeps it when restarted. load_or_update_vm() takes the Some(existing_vm) branch for such a VM, which preserves the CID but never occupies it.

Once the reload finishes, that CID is reserved by nobody. allocate() hands it to the next VM created, and the two collide on the same vsock CID the moment the stopped one starts:

1. deploy VM A → CID 1000, stop it (A stays in memory and in the UI)
2. call the ReloadVms RPC → 1000 is not re-occupied
3. deploy VM B → allocate() returns 1000
4. start A → A and B both claim CID 1000

Root cause and fix

The rebuild treated "running according to the supervisor" as equivalent to "holds a CID", but VM state in memory outlives the process.

Reserve the union of the running CIDs and the in-memory CIDs. The set is keyed by CID, so a VM present in both is reserved once and the running process wins as the recorded owner — the supervisor is authoritative about who actually holds a CID right now.

The orphan sweep at the end of reload_vms_sync() is unchanged and still frees CIDs whose VM disappeared from the filesystem, so this does not leak reservations for removed VMs.

Implementation

• fix(vmm): keep stopped VMs' CIDs reserved across a reload

The reservation set is computed by a small pure helper, cids_to_reserve(), so the rule is testable without standing up a supervisor and a VM directory.

Changed paths:

• dstack/vmm/src/app.rs

Scope

One logical vmm finding. Found while reviewing #907; it is independent of that change and of the IdPool bounds work — this is about which CIDs get put back into the pool, not about the pool's own semantics.

Verification

• cargo test -p dstack-vmm --bins: 83 passed.

• The new stopped_vms_keep_their_cid_reserved_across_a_reload test was confirmed to fail against the previous logic before the fix was applied:

  assertion `left == right` failed
    left: [(1000, "running-vm")]
   right: [(1000, "running-vm"), (1001, "stopped-vm")]
  

• cargo fmt --check --all: passed.

• cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables: passed.

Not covered: no end-to-end run against a real supervisor with a stopped CVM. The failure mode in step 4 is reasoned from the code path, not observed on hardware.

[Copilot]

Pull request overview

This PR fixes a vsock CID collision hazard in dstack-vmm when reload_vms_sync() rebuilds the CID pool: stopped VMs that still retain their CID in memory are now kept reserved across reloads, preventing allocate() from reusing a CID that a stopped VM will later reclaim on restart.

Changes:

• Rebuild the CID pool from the union of (a) CIDs held by currently running supervisor processes and (b) CIDs recorded in in-memory VM state.
• Add a small pure helper (cids_to_reserve) to compute the reservation set with “running wins” semantics for ownership.
• Add unit tests covering stopped-VM reservation, de-duplication, and precedence rules.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.