firecracker: hot-plug parity for --pci VMs - #218
Merged
Merged
Conversation
RunningVMClient, AppendNetworkConfig, TruncateNetworkConfigs and NICPersisted move from the Cloud Hypervisor backend to Backend, and resolveExternalVolume becomes BaseConfig.ResolveExternalVolume, so the Firecracker backend can share them instead of copying. No behaviour change; the tests move with the code.
The reconcile / add-with-rollback / remove-with-truncate sequence is VMM-neutral; only listing live NICs, adding one and removing one differ. Backend.NetResizeWith owns the sequence and the record writes, and the Cloud Hypervisor backend supplies those three operations, so the Firecracker backend can reuse the driver instead of a second copy.
A Firecracker VM created with --pci now resizes NICs, attaches and detaches raw disks, and takes --nics and --data-disk on clone the way a Cloud Hypervisor VM does. The backend drives the virtio-pci devices through GET /, GET /vm/config and PUT/DELETE on /drives and /network-interfaces under the VM ops lock; MMIO VMs keep the explicit rejection because only the PCI transport hot-plugs. Hot-attached disks block snapshot and hibernate like on Cloud Hypervisor, use cocoon_disk_<name> ids because Firecracker ids allow [A-Za-z0-9_] only, and hot-added TAPs are single-queue, the only shape Firecracker opens. A --pci clone restores every snapshot NIC through network_overrides and resizes to --nics afterwards, once the clone's reservation lock is released, and hot-plugs --data-disk files as the next drive slots so later snapshots carry them as ordinary sidecar entries. Firecracker delivers no hot-plug notification to the guest, so the CLI prints the rescan and stale-node removal steps and returns them as hints in JSON.
The --pci hot-plug and NIC MTU paths are validated on the fork's dev release build, so --upgrade now installs it the way it installs the Cloud Hypervisor and firmware fork builds: from the release tag FC_REF, verified against the published SHA256SUMS, with the build commit recorded. Upstream v1.16.1 stays the documented floor.
… first A --pci clone's --output json now embeds the VM record plus hints, so an orchestrator learns the rescan/remove steps the way vm net and vm disk already report them; the text output prints the PCI rescan before the post-clone setup that names the new interfaces. The docs named the JSON flag --format; the commands register --output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#217 gave Firecracker VMs an opt-in virtio-pci transport. This PR makes a
--pciFirecracker VM behave like a Cloud Hypervisor VM on every dimension Firecracker can physically support: NIC resize, raw-disk attach/detach, clone--nicsoverride and clone--data-disk. What Firecracker cannot do by construction (vhost-user-fs, VFIO, the memory knobs, Windows,--restore-mode) keeps its explicit rejection, and MMIO Firecracker VMs keep theirs.What
Five commits, squash-friendly:
RunningVMClient,BaseConfig.ResolveExternalVolume,Backend.PrepareCloneDataDisksmove out of the Cloud Hypervisor backend unchanged.Backend.NetResizeWithowns the reconcile / add-with-rollback / remove-with-truncate sequence and the record writes; a backend suppliesNICDeviceOps(LiveNICs,AddNIC,RemoveNIC,TAPQueues). Devices are matched to record slots by host index (TAP name on CH,eth%don FC), never by MAC: Firecracker clones keep the snapshot's guest MAC while the record carries fresh ones.ErrEjectPendingseparates "guest never released it" (host slot reclaimed, error surfaced) from "VMM refused" (nothing touched).NetResize,DiskAttach/DiskDetach/DiskListon the Firecracker backend throughGET /,GET /vm/config,PUT/DELETE /drives/{id}and/network-interfaces/{id}, all under the VM ops lock with the record reloaded under it; MMIO VMs return the extend package'sErrUnsupportedBackendwith the transport named. Hot disks usecocoon_disk_<name>ids (Firecracker ids allow[A-Za-z0-9_]only; hyphenated names are rejected with that reason). TAPs for hot-added NICs are created single-queue, the only shape Firecracker opens.--pciVMs only), mirroring Cloud Hypervisor.--nics Non a--pcisnapshot restores every snapshot NIC throughnetwork_overrides, then resizes to N with the same driver after the clone's reservation lock is released;--data-diskfiles are created under the clone's runDir and hot-plugged as the nextdrive_<n>slots so later snapshots carry them as ordinary sidecar entries. A failed post-clone resize still reports the VM (JSON included) and then the error.vm net,vm disk attach/detachand clone print the rescan / stale-node removal commands and return them ashintsin--output json(clone output is the VM record plushints).--upgradeinstalls Firecracker from the fork release tagFC_REF(defaultdev, upstream main plus release CI) verified against itsSHA256SUMSand records the build commit, the same way doctor: install the cocoonstack fork builds of cloud-hypervisor and the firmware #215 did for Cloud Hypervisor and the firmware; it is the build the hot-plug and MTU paths are validated on. Upstream v1.16.1 stays the documented floor (its API parser already routesDELETEfor drives and network interfaces). install.md follows.--output jsonon clone now embeds the VM record plushints(additive, vk-cocoon's plain unmarshal is unaffected); the text output prints the PCI rescan before the post-clone setup that names the new interfaces; three doc pages said--format jsonwhere the commands register--output. Two Codex items were checked and not taken: Firecracker's guide removes the guest node before the host unplugs, but cocoon runs nothing inside the guest, so it unplugs first (after unmount/down) and prints the stale-node removal — the E2E matrix exercises exactly that order on NICs and disks and devices.md now states it; and a hot-plug PUT/DELETE whose reply is lost behaves as on Cloud Hypervisor (single-shot on a local unix socket,vm inspectshows the live device set for the retry decision).Hot-path cost: none (no change on VM create, boot or claim;
vm netfetches the live device set once per call).Evidence
Gates on b3e0af0:
make fmt-checkclean;make lint0 issues on GOOS=linux and darwin;asl ./...andGOOS=linux asl ./...0 findings;GOWORK=off go test -race -count=1 ./...35 ok packages, no FAIL.Doctor
--upgradeindebian:bookworm-slimcontainers on both architectures: Firecrackerdev (commit aa9ca227)installed with sha256e16bbd38…on arm64 and5b03ba7e…on x86_64, both equal to the releaseSHA256SUMSentries; Cloud Hypervisor, ch-remote, firmware and CNI unchanged from #215.Review round: three independent readers (adversarial correctness, /simplify four lenses, /code style ledger over every touched file). Findings applied: clone re-acquiring the VM ops lock (deadlock on
clone --nics), boot-time NIC shrink on Cloud Hypervisor losing the eject (MAC keying), Firecracker clones deleting restored NICs (MAC divergence), multi-queue TAPs Firecracker cannot open, host slot reclaimed after a refused eject,GET /vm/configon MMIO snapshots,Result.Hintsleaking into the contract type, deadnewTestCH, duplicatedrunningVMClient, stale disk-package comments. Rejected: constructor placement between type and methods (the repo's documented layout), pushing the clone resize into the backend (no second caller exists).Hardware E2E (cocoon-test2, isolated root, Firecracker fork
devbuildv1.18.0-dev, Cloud Hypervisor forkdevbuildv54.0.0,ghcr.io/cocoonstack/cocoon/ubuntu:24.04, CNI bridge MTU 9000). The same matrix on three arms — Cloud Hypervisor, Firecracker--pci, Firecracker MMIO — each step verified inside the guest:Two earlier runs on intermediate builds caught the multi-queue TAP Firecracker cannot open and the
cocoon-disk-id Firecracker rejects; both are fixed in the branch. On Cloud Hypervisor the guest shows five or sixeth*names right after a clone because the hot-swapped snapshot NICs disappear only once the guest processes the ACPI ejects (docs/known-issues.md: "cannot wait for guest B0EJ"); the check accepts that on Cloud Hypervisor and verifies the record exactly. That path is untouched by this PR. Full log and the round's write-up: cocoon-specstests/2026-09-05-fc-hotplug-mtu-parity.md(2026-09-06 addendum).