Enabling kernel modesetting (KMS) for nvidia drivers in GPU PCI-passthrough HVM causes horrible performance

[Atrate]

Qubes OS release

Qubes OS 4.2

Brief summary

Enabling kernel modesetting (KMS) for nvidia drivers in GPU PCI-passthrough HMV causes horrible performance - framerate in games drops from a decently stable 60 fps to a decently stable 5-10 fps.

This happens regardless of distribution (tested on both Debian 12 and Archlinux).

Steps to reproduce

1. Create a HVM Debian 12/Archlinux qube with PCI passthrough (e.g. according to this great guide: https://neowutran.ovh/qubes/articles/gaming_linux_hvm.html)
2. Install the nvidia drivers
3. On Archlinux do nothing, as KMS is enabled by default, on Debian enable KMS (options nvidia-drm modeset=1 in /etc/modprobe.d/nvidia-drm-kms.conf)
4. Launch a GPU-intensive program, such as a game, on the dedicated GPU (e.g. with PRIME).

Expected behavior

Working reasonably fast, similarly to KMS disabled.

Actual behavior

Working very, very slowly. Even noticeable with glxgears.

Additional information

I haven't been able to test this with Fedora, as any attempts to install nvidia drivers there have failed.

Some software (like gamescope) simply refuses to work without KMS. Some games crash, too.

https://wiki.archlinux.org/title/Kernel_mode_setting

[LordMartron94]

I can confirm. KMS is a real cause of performance issues in Fedora 42 XFCE standalone HVM with NVIDIA GPU passthrough.

Adding the following to my GRUB fixed it:

GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX rd.driver.blacklist=nouveau nvidia-drm.modeset=0"

Now my Factorio FPS went from ~10/12 to the standard 60.