Nested virtualization not working with VFIO devices.

[thomasbarrett]

Describe the bug
Nested virtualization does not seem to be working correctly with VFIO devices. When creating a L1 cloud-hypervisor VM with a VFIO passthrough device and a L2 qemu VM with a VFIO passthrough device, the L1 VM exited with an error originating in the VfioDmaMapping::map function. Specifically, the problem seems to be with mapping the VFIO bars into the L2 VM memory space. See error log below.

To Reproduce

Start L1 virtual-machine.

target/debug/cloud-hypervisor \
  --kernel /var/lib/cloud-hypervisor/CLOUDHV.fd \
   --cpus boot=12,topology=2:6:1:1 \
   --memory size=4G \
   --disk path=~/ubuntu.raw \
   --net "tap=" \
   --console off \
   --device path=/sys/bus/pci/devices/0000:01:00.0,iommu=on \
   -v

Start L2 virtual machine.

/usr/bin/qemu-system-x86_64 \
-M type=q35,accel=kvm -nographic \
-cpu host \
-smp 12 \
-m 1G \
-drive file=/var/lib/cloud-hypervisor/jammy-server-cloudimg-amd64.img,format=qcow2 \
-device vfio-pci,host=0000:00:04.0 \
-net none

Guest OS version details:
Ubuntu 22.04

Host OS version details:
Ubuntu 22,04

Logs
Note that 0xe000180000 is the address of the PCI BAR of the VFIO passthrough device in the L1 guest.

failed to convert guest address 0xe000180000 into host user virtual address

Linux kernel output:

[19056.136978] vfio-pci 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0023 address=0xbe109000 flags=0x0000]
[19056.136983] vfio-pci 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0023 address=0xbe109000 flags=0x0000]
[19056.136990] vfio-pci 0000:01:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0023 address=0xbe109000 flags=0x0000]

[thomasbarrett]

Notes:

• I was able to start a L2 guest using a virtio-blk device. I don't know if nested VFIO is even supposed to be a supported functionality (most if not all of the nested virtualization integration tests use virtio-pci devices.
• I tested this on both an Intel and an AMD hosts, so it doesn't appear to be a AMD vs Intel issue.

[thomasbarrett]

Hmm, it looks like this should be supported. This PR added support back in 2019. I don't know if there are any integration tests. This could have regressed if it is not frequently used.

[thomasbarrett]

After a little additional research, I found something interesting. cloud-hypervisor L2 guests work inside cloud-hypervisor L1 guests... but qemu L2 guests don't work inside cloud-hypervisor L1 guests.

[acarp-crusoe]

@thomasbarrett to confirm -
Cloud hypervisor L1 + cloud hypervisor L2 (VFIO) -> works
Cloud hypervisor L1 + cloud hypervisor L2 (Virtio-blk) -> works
Cloud hypervisor L1 + qemu L2 (Virtio-blk) -> works
Cloud hypervisor L1 + qemu L2 (VFIO) -> fails

[acarp-crusoe]

PR to resolve this here: #6297

[acarp-crusoe]

@thomasbarrett to confirm - Cloud hypervisor L1 + cloud hypervisor L2 (VFIO) -> works Cloud hypervisor L1 + cloud hypervisor L2 (Virtio-blk) -> works Cloud hypervisor L1 + qemu L2 (Virtio-blk) -> works Cloud hypervisor L1 + qemu L2 (VFIO) -> fails

To followup for clarity, the actual behavior of this bug was:

Cloud hypervisor L1 + cloud hypervisor L2 (VFIO) -> fails
Cloud hypervisor L1 + cloud hypervisor L2 (no VFIO) -> works
Cloud hypervisor L1 + qemu L2 (no VFIO) -> works
Cloud hypervisor L1 + qemu L2 (VFIO) -> fails

Both CH and qemu worked when no external devices were passed through over vfio. However, when devices were passed through over vfio, the L1 hypervisor would crash. This turned out to be a bug with cloud hypervisor's implementation of external dma, where it was unable to map mmio regions.

[rbradford]

Do we have a test that exercises this - should we have one?