HVM PCI passthrough not working with devices with option/expansion ROM #2386

esheltone · Oct 20, 2016

Related issue: #1659
Tested on Qubes R3.2 using xen-4.6.3-21 package fixing above issue.

Qubes fails to start HVM appvms to with a PCI device with an option ROM has been assigned. In general, this seems to apply to most PCI/PCIE network adapters and many (or most) PCI/PCIE storage adapters. In my case, I encountered the same problem with two PCIE network adapters that each had an option ROM.

Here is output from 'sudo lspci -v' for the network adapter being passed through:

05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
    Subsystem: Intel Corporation Gigabit CT Desktop Adapter
    Flags: fast devsel, IRQ 19
    Memory at df0c0000 (32-bit, non-prefetchable) [disabled] [size=128K]
    Memory at df000000 (32-bit, non-prefetchable) [disabled] [size=512K]
    I/O ports at e000 [disabled] [size=32]
    Memory at df0e0000 (32-bit, non-prefetchable) [disabled] [size=16K]
    Expansion ROM at df080000 [disabled] [size=256K]
    Capabilities: [c8] Power Management version 2
    Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
    Capabilities: [e0] Express Endpoint, MSI 00
    Capabilities: [a0] MSI-X: Enable- Count=5 Masked-
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-40-5e-7f
    Kernel driver in use: pciback
    Kernel modules: e1000e

Here is part of the output from 'sudo lspci -xxx':

05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
00: 86 80 d3 10 00 00 10 00 00 00 00 02 00 00 00 00
10: 00 00 0c df 00 00 00 df 01 e0 00 00 00 00 0e df
20: 00 00 00 00 00 00 00 00 00 00 00 00 86 80 1f a0
30: 00 00 08 df c8 00 00 00 00 00 00 00 0b 01 00 00

The expansion ROM base address is at 0x30.

With that device (05:00.0) assigned to an HVM domain 'hvmdomain', 'qvm-start hvmdomain' sits for a while with "--> Loading the VM (type = HVM)..." and eventually fails out.

/var/log/xen/console/guest-hvm-hvmdomain-dm.log includes:

pcifront_watches: waiting for backend to get into the right state /local/domain/0/backend/pci/19/0
******************* PCIFRONT for device/pci/0 **********


backend at /local/domain/0/backend/pci/19/0
**************************
pcifront_watches: waiting for backend events /local/domain/0/backend/pci/19/0/state
xs_read_watch() -> device-model/18/command dm-command
dm-command: hot insert pass-through pci dev 
warning: pt_iomul not supported in stubdom 05:00.0
ERROR: PCI region size must be pow2 type=0x8, size=0xdf080000
close(0)

From there it is clear that QEMU has exited due to the above error (from looking at 'xl list' while it hangs, it is clear that Xen fails to realize that QEMU has errored out, which is a separate issue). As a result, the appvm never really gets started.

It looks like the issue is connected with the device's expansion ROM, as the "size" indicated in the QEMU error message (0xdf080000) is actually the address for the device's expansion ROM. I have not tracked down how the address and size for the BAR ended up getting mixed up (does QEMU access PCI config space directly, or is Xen providing this information?).

Some possible solutions?

find and fix the bug in qemu-traditional
modify the relevant code (whether QEMU or Xen) to ignore expansion/option ROMs entirely. I believe expansion/option ROMs will not work anyway with the ROMBios used by qemu-traditional, so maybe just drop them.
move to qemu-upstream. I believe this is not an issue with qemu-upstream, but I need to double-check. Obviously, there are a lot of challenges in making this happen.

The error message is being generated in pci_register_io_region() in tools/qemu-xen-traditional/hw/pci.c, which is being called by pt_register_regions() in pci-passthrough.c:

        pci_register_io_region((PCIDevice *)assigned_device, PCI_ROM_SLOT,
            pci_dev->rom_size, PCI_ADDRESS_SPACE_MEM_PREFETCH,
            pt_iomem_map);

The bad value for pci_dev->rom_size seems to be set in pci_generic_fill_info() in stubdom/pciutils-x86_64/lib/generic.c:

          u32 u = pci_read_long(d, reg);
          if (u != 0xffffffff)
            {
              d->rom_base_addr = u;
              if (flags & PCI_FILL_SIZES)
                {
                  u32 size;
                  pci_write_long(d, reg, ~0);
                  d->rom_size = pci_read_long(d, reg);
                  pci_write_long(d, reg, u);
                }
            }

esheltone · Oct 21, 2016

The error message is being generated in pci_register_io_region() in tools/qemu-xen-traditional/hw/pci.c, which is being called by pt_register_regions() in pci-passthrough.c:

        pci_register_io_region((PCIDevice *)assigned_device, PCI_ROM_SLOT,
            pci_dev->rom_size, PCI_ADDRESS_SPACE_MEM_PREFETCH,
            pt_iomem_map);

The bad value for pci_dev->rom_size seems to be set in pci_generic_fill_info() in stubdom/pciutils-x86_64/lib/generic.c:

          u32 u = pci_read_long(d, reg);
          if (u != 0xffffffff)
            {
              d->rom_base_addr = u;
              if (flags & PCI_FILL_SIZES)
                {
                  u32 size;
                  pci_write_long(d, reg, ~0);
                  d->rom_size = pci_read_long(d, reg);
                  pci_write_long(d, reg, u);
                }
            }

(does QEMU access PCI config space directly, or is Xen providing this information?).

Through xen-pciback/pcifront.

Some possible solutions?

find and fix the bug in qemu-traditional

-1

modify the relevant code (whether QEMU or Xen) to ignore expansion/option ROMs entirely. I believe expansion/option ROMs will not work anyway with the ROMBios used by qemu-traditional, so maybe just drop them.

This may be an option.

move to qemu-upstream. I believe this is not an issue with qemu-upstream, but I need to double-check. Obviously, there are a lot of challenges in making this happen.

@HW42 is working on this, based on your patches sent to xen-devel a
while ago: https://github.com/HW42/qubes-vmm-xen-stubdom-linux.
Actually, if you have some time, we could use some help ;)

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

marmarek · Oct 25, 2016

(does QEMU access PCI config space directly, or is Xen providing this information?).

Through xen-pciback/pcifront.

Some possible solutions?

find and fix the bug in qemu-traditional

-1

modify the relevant code (whether QEMU or Xen) to ignore expansion/option ROMs entirely. I believe expansion/option ROMs will not work anyway with the ROMBios used by qemu-traditional, so maybe just drop them.

This may be an option.

move to qemu-upstream. I believe this is not an issue with qemu-upstream, but I need to double-check. Obviously, there are a lot of challenges in making this happen.

@HW42 is working on this, based on your patches sent to xen-devel a
while ago: https://github.com/HW42/qubes-vmm-xen-stubdom-linux.
Actually, if you have some time, we could use some help ;)

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

Automated announcement from builder-github

The package xen-4.6.3-25.fc25 has been pushed to the r3.2 testing repository for the Fedora fc25 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r3.2-current-testing

QubesOS/qubes-issues

Join GitHub today

HVM PCI passthrough not working with devices with option/expansion ROM #2386

Comments

esheltone commented Oct 20, 2016

This comment has been minimized.

esheltone commented Oct 21, 2016

andrewdavidwong added bug C: core labels Oct 21, 2016

andrewdavidwong added this to the Release 3.2 milestone Oct 21, 2016

This comment has been minimized.

marmarek commented Oct 25, 2016

andrewdavidwong added the help wanted label Oct 27, 2016

marmarek modified the milestones: Release 3.2, Release 3.2 updates Nov 19, 2016

marmarek closed this in marmarek/old-qubes-vmm-xen@31a3a3d Jan 8, 2017

This comment has been minimized.

qubesos-bot commented Jan 8, 2017

qubesos-bot added the r3.2-fc25-cur-test label Jan 8, 2017

This comment has been minimized.

qubesos-bot commented Jan 8, 2017

qubesos-bot added the r3.2-dom0-cur-test label Jan 8, 2017

This comment has been minimized.

qubesos-bot commented Jan 8, 2017

qubesos-bot added the r3.2-fc23-cur-test label Jan 8, 2017

This comment has been minimized.

qubesos-bot commented Jan 8, 2017

qubesos-bot added the r3.2-fc24-cur-test label Jan 8, 2017

marmarek added a commit to marmarek/old-qubes-vmm-xen that referenced this issue Jan 9, 2017

This comment has been minimized.

qubesos-bot commented Feb 20, 2017

qubesos-bot added r3.2-fc23-stable and removed r3.2-fc23-cur-test labels Feb 20, 2017

This comment has been minimized.

qubesos-bot commented Feb 20, 2017

qubesos-bot added r3.2-fc24-stable and removed r3.2-fc24-cur-test labels Feb 20, 2017

This comment has been minimized.

qubesos-bot commented Feb 20, 2017

qubesos-bot added r3.2-fc25-stable and removed r3.2-fc25-cur-test labels Feb 20, 2017

This comment has been minimized.

qubesos-bot commented Feb 20, 2017

qubesos-bot added r3.2-dom0-stable and removed r3.2-dom0-cur-test labels Feb 20, 2017

qubesos-bot referenced this issue in QubesOS/updates-status Apr 2, 2017

marmarek closed this in `marmarek/old-qubes-vmm-xen@31a3a3d` Jan 8, 2017