Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it #4321

Open
pqyptixa opened this Issue Sep 19, 2018 · 11 comments

Comments

Projects
None yet
6 participants
@pqyptixa
Copy link

commented Sep 19, 2018

Qubes OS version:

R4.0

Affected component(s):

Debian 10 AppVM


Steps to reproduce the behavior:

So, I finally could pass a GPU to a VM with PCI passthrough some weeks ago, but for some reason, I can't get it to run with more RAM.
I asked #qubes_os @ freenode about this issue, and ended up making a guide on how to get GPU-passthrough done. I'm not going to post the complete guide here, so here's a link to it: https://paste.debian.net/1043341/ . I don't mind if someone else posts it here or elsewhere, but hopefully people will discuss it and improve it.
My system: Qubes 4.0 running on a AMD Ryzen 1600 CPU + MSI B350 Mortar motherboard (BIOS 1E0) + ASUS RX560 2GB GPU for the VMs + an old NVIDIA 610 GPU for dom0 (as secondary GPU).

Expected behavior:

AppVM runs just fine with > 3.5GB of RAM.

Actual behavior:

AppVM crashes with more than 3.5GB of RAM. The way it crashes depends on how much memory is assigned to it, but I usually see almost-instant kernel panics, systemd-init crashing, fsck failures and/or other random processes crashing, too.

General notes:

Other than removing "nomodeset" from the kernel command line, everything is set as default.

@Jeeppler

This comment has been minimized.

Copy link

commented Sep 19, 2018

@pqyptixa how did you get the PCI passthrough to work? Would you be willing to explain your steps?

@pqyptixa

This comment has been minimized.

Copy link
Author

commented Sep 19, 2018

@Jeeppler I left a link to the guide, but PCI passthrough "just worked" for me. I didn't need to mess with IOMMU groups or anything (in fact, "/sys/kernel/iommu_groups/" is empty in my system), simply passing the device by running "qvm-pci attach ..." worked.

@Jeeppler

This comment has been minimized.

Copy link

commented Sep 19, 2018

@pqyptixa thanks, for the explanation.

@alcreator

This comment has been minimized.

Copy link

commented Sep 20, 2018

This is caused by the default TOLUD (Top of Low Usable DRAM) of 3.75G provided by qemu not being large enough to accommodate the larger BARs that a graphics card typically has.

The code to pass a custom max-ram-below-4g value to the qemu command line does exist in the libxl_dm.c file of xen, but there is no functionality in libvirt to add this parameter.

It is possible to manually add this parameter to the qemu commandline by doing the following in a dom0 terminal:

mkdir stubroot
cp /usr/lib/xen/boot/stubdom-linux-rootfs stubroot/stubdom-linux-rootfs.gz
cd stubroot
gunzip stubdom-linux-rootfs.gz
cpio -i -d -H newc --no-absolute-filenames < stubdom-linux-rootfs
rm stubdom-linux-rootfs
nano init

Before the line "# $dm_args and $kernel are separated with \x1b to allow for spaces in arguments." add:

SP=$'\x1b'
dm_args=$(echo "$dm_args" | sed "s/-machine\\${SP}xenfv/-machine\\${SP}xenfv,max-ram-below-4g=3.5G/g")

Then execute:

find . -print0 | cpio --null -ov --format=newc | gzip -9 > ../stubdom-linux-rootfs
sudo mv ../stubdom-linux-rootfs /usr/lib/xen/boot/

Note that this will apply the change to all HVMs, so if you have any other HVM with more than 3.5G ram assigned, they will not start without the adapter being passed through.

Ideally to fix this libvirt should be extended to pass the max-ram-below-4g parameter through to xen, and then a calculation added to determine the correct TOLUD based on the total BAR size of the PCI devices are being passed through to the vm.

@pqyptixa

This comment has been minimized.

Copy link
Author

commented Sep 20, 2018

@alcreator Awesome, thanks for replying!
I suspected QEMU was related to this, but I had no idea where to look for info, I'm really glad you replied!
I was able to make it work by adding your dm_args line to the init file with a small modification, so that it doesn't affect other VMs:

vm_name=$(xenstore-read "/local/domain/$domid/name")
SP=$'\x1b'
if [ x"$vm_name" == x"my_appvm" ]; then
    dm_args=$(echo "$dm_args" | sed "s/-machine\\${SP}xenfv/-machine\\${SP}xenfv,max-ram-below-4g=3.5G/g")
fi

Obviously this is ugly and hacky, but it works :D

Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)
BTW, does /usr/lib/xen/boot/stubdom-linux-rootfs ever get updated?

@alcreator

This comment has been minimized.

Copy link

commented Sep 20, 2018

/usr/lib/xen/boot/stubdom-linux-rootfs will get updated every time the "xen-hvm-stubdom-linux" package is updated.

@pqyptixa

This comment has been minimized.

Copy link
Author

commented Nov 4, 2018

Just in case someone had problems with the modified script I posted above: the correct command is
vm_name=$(xenstore-read "/local/domain/$domid/name")
instead of
vm_name=$(xenstore-read "local/domain/$domid/name")
as I had posted originally.

@shamen123

This comment has been minimized.

Copy link

commented Dec 6, 2018

Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)

What is the security risk if Dom0 is driven by a different GPU, and not the one that is assigned to passthrough? OP seems to have 2 GPU cards, one for Dom0 and one for passthrough. IIRC, the risk to passthrough of GPU was being able to see Dom0 screen from the VM.

@pqyptixa

This comment has been minimized.

Copy link
Author

commented Jan 10, 2019

Hopefully this will help other Qubes users! (Though here's a reminder: GPU passthrough is actually a security risk...)

What is the security risk if Dom0 is driven by a different GPU, and not the one that is assigned to passthrough? OP seems to have 2 GPU cards, one for Dom0 and one for passthrough. IIRC, the risk to passthrough of GPU was being able to see Dom0 screen from the VM.

I'm not an expert, but AFAIK, the problem is DMA. Not really sure about this, but AFAIU, an attacker could be able to access system memory.

@shamen123

This comment has been minimized.

Copy link

commented Jan 10, 2019

I dont claim to be an expert either, but from my understanding VT-d/IOMMU, assuming your firmware is trustworthy, should mean a PCI passthrough of a 2nd (unused) GPU to a virtual machine should have no access to Dom0 screen or RAM. From my limited understanding, the risk is passing through the primary GPU . maybe @rootkovska has more insight?

@marmarek

This comment has been minimized.

Copy link
Member

commented Jan 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.