Skip to content

virtio: restore the PCI config space before reinstalling the doorbells - #4146

Merged
Steven Malis (smalis-msft) merged 1 commit into
microsoft:mainfrom
bitranox:virtio-restore-config-space-first
Aug 4, 2026
Merged

virtio: restore the PCI config space before reinstalling the doorbells#4146
Steven Malis (smalis-msft) merged 1 commit into
microsoft:mainfrom
bitranox:virtio-restore-config-space-first

Conversation

@bitranox

Copy link
Copy Markdown
Contributor

VirtioPciDevice::restore restored the PCI config space last, after
restore_common. restore_common is where install_doorbells runs, and it
asks doorbell_region for the notify address, which reads bar_address(0).
With the config space not yet restored there is no active BAR, so it answers
None and no queue doorbell is installed.

Nothing recovers from that later. The only other install site is the guest
writing DRIVER_OK, and a guest resumed mid-flight never writes it again, so a
restored device runs without doorbells for the life of the VM.

The effect is performance, not correctness: the notify write still reaches
notify_queue through mmio_write. On a restored virtio-net NIC every one of
the guest's kicks at BAR0's notify register left the kernel as an MMIO exit,
across three boots, where the same VM before its snapshot took none. With the
config space restored first, both sides take none.

The fix moves the existing config_space.restore call ahead of
restore_common. Nothing in restore_common reads the interrupt state that is
restored after it, so the order of the rest is unchanged.

On the test

pci_restore_reinstalls_doorbells passed throughout, because it programmed the
target device's BARs by hand before restoring. A real restore does not do that,
which is exactly the condition that hides the bug.

It now restores into a device with unprogrammed BARs, and asserts the set of
installed doorbells rather than a registration count. The count could not tell a
reinstall from an install that happened once before the save, and the address
alone is not enough either: every queue of one device registers at the same
notify address and is distinguished by its datamatch value.

Against the unfixed code the test now fails with no doorbells installed where
two are expected.

Copilot AI review requested due to automatic review settings August 2, 2026 10:37
@bitranox
Robert Nowotny (bitranox) requested a review from a team as a code owner August 2, 2026 10:37
@github-actions github-actions Bot added the unsafe Related to unsafe code label Aug 2, 2026
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a virtio PCI restore-order bug where queue doorbells weren’t reinstalled after restoring from saved state because BAR0 wasn’t active yet when restore_common() attempted to compute the notify address. Restoring the PCI config space earlier ensures BARs are available before doorbell reinstallation, avoiding persistent MMIO exits after resume.

Changes:

  • Restore PCI config space before restore_common() so doorbell_region() can see an active BAR0 and reinstall queue doorbells during restore.
  • Strengthen pci_restore_reinstalls_doorbells to restore into a “fresh” device (unprogrammed BARs) and assert the exact installed doorbell set (address + datamatch + length).
  • Enhance the test memory backend to track currently-live doorbell registrations (not just a cumulative registration count).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
vm/devices/virtio/virtio/src/transport/pci.rs Reorders restore steps so PCI config space (incl. BARs) is restored before doorbell installation occurs in restore_common().
vm/devices/virtio/virtio/src/tests.rs Improves PCI restore test to reflect real restore conditions and adds live doorbell tracking to validate reinstalls precisely.

Comment thread vm/devices/virtio/virtio/src/tests.rs Outdated
Comment thread vm/devices/virtio/virtio/src/transport/pci.rs
Comment thread vm/devices/virtio/virtio/src/tests.rs Outdated
`VirtioPciDevice::restore` called `restore_common` first, and that is where
`install_doorbells` runs. At that point the config space had not been restored,
so `doorbell_region` read `bar_address(0)` with no active BAR, answered `None`,
and no queue doorbell was installed. The only other install site is the guest
writing DRIVER_OK, which a guest resumed mid-flight never does again, so the
omission lasted for the life of the VM.

Correctness was unaffected, since the notify write falls through to
`mmio_write` and `notify_queue`. That is why it took counting exits to notice:
on a restored virtio-net NIC every one of the guest's kicks at BAR0's notify
register left the kernel as an MMIO exit, measured across three boots, where
the same VM before its snapshot took none. With the config space restored
first, both sides take none.

The existing test could not see this. It programmed the target device's BARs by
hand before restoring, which is not what a restore does. It now restores into a
device with unprogrammed BARs, the way production does, and asserts the
installed doorbell set (address plus per-queue datamatch) rather than a
registration count, which cannot tell a reinstall from an install that happened
once before the save.
Copilot AI review requested due to automatic review settings August 2, 2026 10:46
@bitranox
Robert Nowotny (bitranox) force-pushed the virtio-restore-config-space-first branch from 912d27e to dc6abbc Compare August 2, 2026 10:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@jstarks

Copy link
Copy Markdown
Member

Thanks for the fix!

@bitranox

Robert Nowotny (bitranox) commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Uh, You are motivated, working on sundays ...

@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown

@smalis-msft
Steven Malis (smalis-msft) merged commit d31d664 into microsoft:main Aug 4, 2026
68 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

unsafe Related to unsafe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants