PCI Passthrough fails critically

[ZYousef]

How to file a helpful issue

Qubes OS release

R4.1

Brief summary

Passing two PCIs to a single qubes makes it unstable.
The qubes fails to start occasionally due libxenlight being unable to reset PCI "even though PCI is set to no-strict-reset"

Steps to reproduce

Attach VGA and Audio PCI to a qube.
Set both PCI to no strict reset.

AppVM qubesd log:
Start failed: internal error: Unable to to reset PCI device X:::::: Internal error: libxenlight failed to create new domain "AppVM"

AppVM libxl-driver.log:
stubddom_xswait_cb: Domain X: Stubddom 5 for 4 startup: startup timed out.
domcreate_devmodel_started: Domain X: device model did not start -9

Expected behavior

qube boots up every time correctly.

Actual behavior

The qube fails sometimes with: Libxenlight failed to create new domain qube

[jandryuk]

With Xen 4.16 (OpenXT), I sometimes see the multiple passthrough failure like:

libxl_pci.c:1824:device_pci_add_done:Domain 24:libxl__device_pci_add failed for PCI device 0:0:1f.6 (rc -9)
libxl_pci.c:1824:device_pci_add_done:Domain 24:libxl__device_pci_add failed for PCI device 0:1:0.0 (rc -9)
libxl_create.c:1945:domcreate_attach_devices:Domain 24:unable to add pci devices

I just posted this patch to handle it:
https://lore.kernel.org/lkml/20220829151536.8578-1-jandryuk@gmail.com/T/#u

[marmarek]

I do see this issue from time to time too, even though qubes has this patch: https://github.com/QubesOS/qubes-vmm-xen/blob/xen-4.14/patch-libxl-add-pcidevs-to-stubdomain-earlier.patch. When it happens, it usually fails on the second device assigned to the VM, the first one is assigned successfully. I'm trying to collect more logging from QMP communication.

[jandryuk]

I had forgotten about https://github.com/QubesOS/qubes-vmm-xen/blob/xen-4.14/patch-libxl-add-pcidevs-to-stubdomain-earlier.patch. Yeah, that would be a different flow than what my patch addresses.

[ZYousef]

I found a workaround but I'm not sure what's going on. After investigating it looks like the second PCI falls out of the pci-assignable-list (xl pci-assignable-list), I set a script to add the pci at each boot the assignable list ( xl pci-assignable-add <second-pci-BDF> ). I've been testing the qube and it seems to be working fine ever since.