vmm: sanitize GPUs with SBR before QEMU attach

[kvinwang]

Summary

• reset each assigned GPU's dedicated upstream PCIe bus before QEMU attaches it
• default cvm.gpu.sanitize_on_attach to enabled while allowing operators to disable it
• reject shared-bridge topologies instead of risking disruption to unrelated PCI devices
• apply the same sanitization to server-managed and one-shot VM launches

Motivation

An H200 CVM incident and a subsequent controlled reproduction established that terminating QEMU while GSP/SPDM initialization is in progress can leave stale FSP/SPDM session state in the GPU. VFIO's attach-time FLR did not clear that state, while a Secondary Bus Reset did. The next guest otherwise fails with an SPDM timeout followed by GSP/RmInitAdapter errors.

This implements sanitize-on-attach at the VMM ownership boundary. A reset or topology validation failure aborts the launch rather than handing a potentially contaminated GPU to the guest.

Safety

• only manifest GPU devices are sanitized; NVSwitch bridge entries are not reset
• every GPU must be directly behind a PCI bridge
• the bridge must not contain another downstream PCI function
• duplicate bridges are reset only once
• the original PCI Bridge Control value is preserved apart from the SBR bit

Tests

• cargo check -p dstack-vmm
• cargo test -p dstack-vmm (116 passed)
• topology checked read-only against all eight H200 devices on tdx-usc3: every GPU has a dedicated direct upstream PCIe bridge

[Copilot]

Pull request overview

This PR adds a GPU “sanitize-on-attach” step in the VMM by issuing a PCIe Secondary Bus Reset (SBR) on each selected GPU’s dedicated upstream bridge immediately before QEMU attaches the device, with topology validation to avoid disrupting unrelated PCIe devices. It also enables this behavior by default via configuration, and applies it to both server-managed VM launches and one-shot launches.

Changes:

• Add sanitize_on_attach (default: enabled) to the VMM GPU config and document it in vmm.toml.
• Introduce gpu_reset module that resolves each GPU’s upstream bridge, rejects shared-bridge topologies, and performs SBR while preserving bridge control bits (except the SBR bit).
• Invoke sanitization prior to QEMU attach for both the server flow (App) and one-shot flow.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
dstack/vmm/vmm.toml	Documents and enables cvm.gpu.sanitize_on_attach by default.
dstack/vmm/src/config.rs	Adds sanitize_on_attach field with a default-true serde default + a config default test.
dstack/vmm/src/gpu_reset.rs	New implementation for validating GPU PCIe topology and performing SBR resets + unit tests.
dstack/vmm/src/main.rs	Registers the new gpu_reset module.
dstack/vmm/src/app.rs	Calls GPU sanitization before building QEMU processes in the server-managed VM start path.
dstack/vmm/src/one_shot.rs	Calls GPU sanitization before executing QEMU in one-shot mode.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.