Skip to content

firecracker: opt-in virtio-pci transport - #217

Merged
CMGS merged 1 commit into
masterfrom
fc/pci-transport
Sep 5, 2026
Merged

firecracker: opt-in virtio-pci transport#217
CMGS merged 1 commit into
masterfrom
fc/pci-transport

Conversation

@CMGS

@CMGS CMGS commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Why

Firecracker device hot-plug (Developer Preview since 1.16.0) only works on the virtio-pci transport, which Firecracker enables per process with --enable-pci. cocoon always ran Firecracker on MMIO. This is the first of two steps: make the transport a per-VM choice, keep MMIO the default, and leave hot-plug itself to a follow-up.

What

  • --pci on vm run / vm create (Firecracker only; validateBackendFlags rejects it on Cloud Hypervisor, which is always virtio-pci). Stored as Config.PCI, so it travels with snapshots: CloneVMConfigFromFlags inherits it and RestoreVMConfigFromFlags already copies the whole Config, so clones and restores relaunch on the transport their snapshot was taken with.
  • Firecracker launch args gain --enable-pci when the record says so (launchArgs); vm debug prints the same line.
  • cocoon's own pci=off leaves the Firecracker command line: Firecracker inserts it itself whenever PCI is disabled (builder.rs, in every release from 1.13 on, below the 1.16.1 floor), and with --enable-pci it would hide every device (the boot hangs, as seen in the hot-plug round). The MMIO guest still boots with exactly one pci=off.
  • Docs: --pci row in cli.md, a "PCI Transport (Firecracker)" section in vm.md, the clone-constraint sentence in snapshots.md.

Tests: TestValidateBackendFlagsRejectsPCIOnCloudHypervisor, TestBuildCmdlineLeavesPCIToFirecracker, TestLaunchArgsEnablePCI, and the clone-config test now asserts PCI inheritance.

Hot-path cost: none (one bool read at launch).

Evidence

Gates on the branch: make fmt-check clean; make lint 0 issues on GOOS=linux and darwin; asl ./... and GOOS=linux asl ./... 0 findings; GOWORK=off go test -race -count=1 ./... 35 ok packages, no FAIL.

Hardware (cocoon-test2, isolated root, Firecracker fork dev build v1.18.0-dev, ghcr.io/cocoonstack/cocoon/ubuntu:24.04):

# boot latency, vm run -> first vm exec, 5 rounds interleaved with the order swapped (ms)
ubuntu mmio: 1899 1779 1794 1928 1928
ubuntu pci:  1791 1814 1803 1797 1806

# final branch build
-- m1 (default):  0 enable-pci; pci=off x1 pcidev 0 virtio 7
-- p1 (--pci):    1 enable-pci; pci=off x0 pcidev 8 virtio 7
-- p2 (clone of p1's snapshot): 1 enable-pci; pci=off x0 pcidev 8 virtio 7
-- p1 (restored from that snapshot): 1 enable-pci; pci=off x0 pcidev 8 virtio 7

Full record: cocoon-specs tests/2026-09-05-fc-hotplug-mtu-parity.md (2026-09-06 addendum). Not covered: the sandbox guest image, whose kernel has no CONFIG_VIRTIO_MMIO and did not come up on Firecracker on either transport within the round; that is a separate investigation and vm.md says so.

--pci on vm run/create starts Firecracker with --enable-pci so the guest
enumerates virtio-pci devices, the prerequisite for Firecracker device
hot-plug. The transport is recorded on the VM config, so clones and
restores relaunch on the transport their snapshot was taken with, and
Cloud Hypervisor rejects the flag since it is always virtio-pci. cocoon
no longer puts pci=off on the Firecracker command line: Firecracker adds
it itself whenever PCI is disabled (since the PCI work in 1.13, below the
1.16.1 floor), and on --enable-pci the flag would hide every device.
@CMGS
CMGS merged commit 545d63e into master Sep 5, 2026
4 checks passed
@CMGS
CMGS deleted the fc/pci-transport branch September 5, 2026 15:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant