virtio-devices: save pci configuration capability state in snapshot

[alex-matei]

When restoring a VM, the VirtioPciCfgCapInfo struct is not properly initialized. All fields are 0, including the offset where the capabibility should start. Hence, when you read a PCI configuration register in the range [0..length(VirtioPciCfgCap)] you get the value 0 instead of the actual register contents.

Linux rescans the whole PCI bus when adding a new device. It reads the values vendor_id and device_id for every device. Because these are stored at offset 0 in pci configuration space, their value is 0 for existing devices. As such, Linux considers that the devices have been unplugged and it removes them from the system.

Fixes: #6265

[rbradford]

Fab - good work. I tested this with live migration which is how I bisected the first bad commit in #6265

[likebreath]

I want to point out that this is a breaking change for live upgrade, and I don't think we can (easily) workaround it (like what we did for xsave_state, etc).

Also, I can see this is an very important bug fix, given any migrated/restored VM won't be functional if there is a device hotplug without this fix. We will need to see how we want to backport it.

[liuw]

I want to point out that this is a breaking change for live upgrade, and I don't think we can (easily) workaround it (like what we did for xsave_state, etc).

Also, I can see this is an very important bug fix, given any migrated/restored VM won't be functional if there is a device hotplug without this fix. We will need to see how we want to backport it.

Is the concern here the new CH will reject an old stream because the old stream does not have the PCI state field? In that case can we generate a state structure that's all zeros to preserve the buggy behaviour?

[likebreath]

The concern is that we won't be able to backport this fix without breaking the live-upgrade, particularly for point releases of LTS version, where we intended to ensure they are live upgradable.

[rbradford]

I think we may be able to fix the issue in the backport in a different way - to me it looks like that the PCI capability data for the virtio devices is static - afaict the only place the offset and capability data is set is here:

        let configuration_cap = VirtioPciCfgCap::new();
        self.cap_pci_cfg_info.offset = self
            .configuration
            .add_capability(&configuration_cap)
            .map_err(PciDeviceError::CapabilitiesSetup)?
            + VIRTIO_PCI_CAP_OFFSET;
        self.cap_pci_cfg_info.cap = configuration_cap;

Perhaps we can just setup the capability and offset after the device migration.