AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it

[pqyptixa]

Qubes OS version:

R4.0

Affected component(s):

Debian 10 AppVM

---

Steps to reproduce the behavior:

So, I finally could pass a GPU to a VM with PCI passthrough some weeks ago, but for some reason, I can't get it to run with more RAM.
I asked #qubes_os @ freenode about this issue, and ended up making a guide on how to get GPU-passthrough done. I'm not going to post the complete guide here, so here's a link to it: https://paste.debian.net/1043341/ . I don't mind if someone else posts it here or elsewhere, but hopefully people will discuss it and improve it.
My system: Qubes 4.0 running on a AMD Ryzen 1600 CPU + MSI B350 Mortar motherboard (BIOS 1E0) + ASUS RX560 2GB GPU for the VMs + an old NVIDIA 610 GPU for dom0 (as secondary GPU).

Expected behavior:

AppVM runs just fine with > 3.5GB of RAM.

Actual behavior:

AppVM crashes with more than 3.5GB of RAM. The way it crashes depends on how much memory is assigned to it, but I usually see almost-instant kernel panics, systemd-init crashing, fsck failures and/or other random processes crashing, too.

General notes:

Other than removing "nomodeset" from the kernel command line, everything is set as default.

PS: link to the guide expired. Here a a couple of similar guides: https://github.com/Qubes-Community/Contents/blob/master/docs/customization/windows-gaming-hvm.md and https://neowutran.ovh/qubeos.html

[Jeeppler]

@pqyptixa how did you get the PCI passthrough to work? Would you be willing to explain your steps?

[pqyptixa]

@Jeeppler I left a link to the guide, but PCI passthrough "just worked" for me. I didn't need to mess with IOMMU groups or anything (in fact, "/sys/kernel/iommu_groups/" is empty in my system), simply passing the device by running "qvm-pci attach ..." worked.

[Jeeppler]

@pqyptixa thanks, for the explanation.

[alcreator]

This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has.

The code to pass a custom max-ram-below-4g value to the qemu command line does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to add this parameter.

It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal:

mkdir stubroot
cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz
cd stubroot
gunzip stubdom-linux-rootfs.gz
cpio -i -d -H newc --no-absolute-filenames < stubdom-linux-rootfs
rm stubdom-linux-rootfs
nano init

Before the line "# $dm_args and $kernel are separated with \x1b to allow for spaces in arguments." add:

SP=$'\x1b'
dm_args=$(echo "$dm_args" | sed "s/-machine\\${SP}xenfv/-machine\\${SP}xenfv,max-ram-below-4g=3.5G/g")

Then execute:

find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../stubdom-linux-rootfs
sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot/

Note that this will apply the change to all HVMs, so if you have any other HVM with more than 3.5G ram assigned, they will not start without the adapter being passed through.

Ideally to fix this libvirt should be extended to pass the max-ram-below-4g parameter through to xen, and then a calculation added to determine the correct TOLUD based on the total BAR size of the PCI devices are being passed through to the vm.

[pqyptixa]

@alcreator Awesome, thanks for replying!
I suspected QEMU was related to this, but I had no idea where to look for info, I'm really glad you replied!
I was able to make it work by adding your dm_args line to the init file with a small modification, so that it doesn't affect other VMs:

vm_name=$(xenstore-read "/local/domain/$domid/name")
SP=$'\x1b'
if [ x"$vm_name" == x"my_appvm" ]; then
    dm_args=$(echo "$dm_args" | sed "s/-machine\\${SP}xenfv/-machine\\${SP}xenfv,max-ram-below-4g=3.5G/g")
fi

Obviously this is ugly and hacky, but it works :D

Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)
BTW, does /usr/lib/xen/boot/stubdom-linux-rootfs ever get updated?

[alcreator]

/usr/lib/xen/boot/stubdom-linux-rootfs will get updated every time the "xen-hvm-stubdom-linux" package is updated.

[pqyptixa]

Just in case someone had problems with the modified script I posted above: the correct command is
vm_name=$(xenstore-read "/local/domain/$domid/name")
instead of
vm_name=$(xenstore-read "local/domain/$domid/name")
as I had posted originally.

[ThePlexus]

Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)

What is the security risk if Dom0 is driven by a different GPU, and not the one that is assigned to passthrough? OP seems to have 2 GPU cards, one for Dom0 and one for passthrough. IIRC, the risk to passthrough of GPU was being able to see Dom0 screen from the VM.

[pqyptixa]

Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)

What is the security risk if Dom0 is driven by a different GPU, and not the one that is assigned to passthrough? OP seems to have 2 GPU cards, one for Dom0 and one for passthrough. IIRC, the risk to passthrough of GPU was being able to see Dom0 screen from the VM.

I'm not an expert, but AFAIK, the problem is DMA. Not really sure about this, but AFAIU, an attacker could be able to access system memory.

[ThePlexus]

I dont claim to be an expert either, but from my understanding VT-d/IOMMU, assuming your firmware is trustworthy, should mean a PCI passthrough of a 2nd (unused) GPU to a virtual machine should have no access to Dom0 screen or RAM. From my limited understanding, the risk is passing through the primary GPU . maybe @rootkovska has more insight?