Wrong PCIe devices get passed through to wrong qubes after adding new PCIe cards

[ddevz]

After installing qubes and setting up your passthrough PCI devices, if you add a new PCIe card, it can change the device numbers of the old cards. This will cause qubes to pass through the wrong devices to the wrong HVM qubes.

The device renumbering is obviously a hardware thing, and preferably there would be a way to make persistent names similar to how linux gives ethernet cards persistent names. If you remove eth0 from a linux system and put another ethernet card in, linux starts calling that new card eth1, not eth0. Linux probably determines uniqueness by MAC address, so there might not be any way to uniquely identify PCIe cards like that.

I expect this to become a bigger problem in the future when people are using sys-gui-gpu and video cards getting a new PCIe number means they will no longer pass through to sys-gui-gpu qube.

(this is related to, but different then #6587 )

[DemiMarie]

Ouch. This is actually a security concern: passing e.g. your primary SSD to sys-net would be bad.

The only solution I can think of is to identify devices by PCI bus path, which should be deterministic and enforced by hardware.

[unman]

This is a known issue with PCI ( at least it was ~4 yrs ago - I could be out of date.) Numbering can be affected by introducing new devices, BIOS/UEFI changes, etc. When I cared about these things we never rolled out hardware changes without testing for such changes. The bus path was not deterministic. Some cards will provide their own buses and bridges, which can really complicate numbering.

[DemiMarie]

@unman is there any sort of identifier associated with a physical slot on the motherboard somewhere, and which cannot change?

[unman]

I'm a little out of touch - I'll check back.

[Garbage4F]

I've experienced this issue whilst trying to add a 3rd GPU to this motherboard https://www.msi.com/Motherboard/X399-SLI-PLUS/Specification - in particular when 2 cards were nvidia and 1 ati all the pci IDs changed compared to a known good configuration with 1 less GPU. However going from 1 -> 2 GPUs with them in the white slots, they kept their IDs