Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it #4321
Qubes OS version:
Debian 10 AppVM
Steps to reproduce the behavior:
So, I finally could pass a GPU to a VM with PCI passthrough some weeks ago, but for some reason, I can't get it to run with more RAM.
AppVM runs just fine with > 3.5GB of RAM.
AppVM crashes with more than 3.5GB of RAM. The way it crashes depends on how much memory is assigned to it, but I usually see almost-instant kernel panics, systemd-init crashing, fsck failures and/or other random processes crashing, too.
Other than removing "nomodeset" from the kernel command line, everything is set as default.
@Jeeppler I left a link to the guide, but PCI passthrough "just worked" for me. I didn't need to mess with IOMMU groups or anything (in fact, "/sys/kernel/iommu_groups/" is empty in my system), simply passing the device by running "qvm-pci attach ..." worked.
This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has.
The code to pass a custom max-ram-below-4g value to the qemu command line does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to add this parameter.
It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal:
Before the line "# $dm_args and $kernel are separated with \x1b to allow for spaces in arguments." add:
Note that this will apply the change to all HVMs, so if you have any other HVM with more than 3.5G ram assigned, they will not start without the adapter being passed through.
Ideally to fix this libvirt should be extended to pass the max-ram-below-4g parameter through to xen, and then a calculation added to determine the correct TOLUD based on the total BAR size of the PCI devices are being passed through to the vm.
@alcreator Awesome, thanks for replying!
Obviously this is ugly and hacky, but it works :D
Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)
referenced this issue
Oct 8, 2018
What is the security risk if Dom0 is driven by a different GPU, and not the one that is assigned to passthrough? OP seems to have 2 GPU cards, one for Dom0 and one for passthrough. IIRC, the risk to passthrough of GPU was being able to see Dom0 screen from the VM.
I'm not an expert, but AFAIK, the problem is DMA. Not really sure about this, but AFAIU, an attacker could be able to access system memory.
I dont claim to be an expert either, but from my understanding VT-d/IOMMU, assuming your firmware is trustworthy, should mean a PCI passthrough of a 2nd (unused) GPU to a virtual machine should have no access to Dom0 screen or RAM. From my limited understanding, the risk is passing through the primary GPU . maybe @rootkovska has more insight?
On Thu, Jan 10, 2019 at 09:06:46AM -0800, shamen123 wrote: I dont claim to be an expert either, but from my understanding VT-d/IOMMU, assuming your firmware is trustworthy, should mean a PCI passthrough of a 2nd (unused) GPU to a virtual machine should have no access to Dom0 screen or RAM. From my limited understanding, the risk is passing through the primary GPU.
Yes, that's right. One issue previously mentioned frequently about GPU passthrough was that it wasn't working together with qemu in stubdomain so the procedure included running qemu directly in dom0, which is a security risk. But it isn't the case here, in Qubes 4.0 - here it works with qemu still in stubdomain.