CUDA UVM Module Prevents DGPU Shut Down

[Tanner-Lin]

CUDA 6 added a new feature named Unified Virtual Memory, which requires a separate kernel module nvidia-uvm that depends on the main driver module. This module is automatically loaded (if not already) when a CUDA/OpenCL context is initiated.

If Bumblebee tries to shut down the DGPU after this happens, the UVM module would block the main NV driver module removal, and the following BBSwitch call would fail as well.

Please add code to handle the UVM module before disabling the main driver module.

[amonakov]

With module-init-tools or kmod (v14+, check with modprobe --version), you can provide a custom command for unloading the nvidia module. Create a file /etc/modprobe.d/nvidia-unload.conf with contents:

remove nvidia rmmod nvidia-uvm nvidia

This should allow modprobe -r nvidia to transparently remove both nvidia-uvm and nvidia.

Can you confirm this works for you?

[Tanner-Lin]

Tested on an Arch Linux install. Manually calling modprobe -r nvidia works as desired.

[amonakov]

Did that fix Bumblebee usage as well?

Instead of specifying remove nvidia ... clause, you can tell modprobe that nvidia-uvm is a soft-dependency of nvidia:

softdep nvidia post: nvidia-uvm

This will cause nvidia-uvm to be loaded after nvidia and unloaded before nvidia automatically.

Can you follow up with Arch maintainers to implement one of the solutions?

[amonakov]

I have filed a Gentoo bug here: https://bugs.gentoo.org/show_bug.cgi?id=506168

[Tanner-Lin]

The two methods only work with manual module unloading via modprobe -r. With both Bumblebee would still fail to shut down the DGPU.

Specifying remove nvidia ... doesn't hurt, but fails to solve the issue.

[user@host ~]$ optirun glxspheres64
Polygons in scene: 62464
Visual ID of window: 0x20
Context is Direct
OpenGL Renderer: GeForce GTX 765M/PCIe/SSE2
62.161759 frames/sec - 69.372523 Mpixels/sec
59.999007 frames/sec - 66.958892 Mpixels/sec
[user@host ~]$ cat /proc/acpi/bbswitch
0000:01:00.0 OFF
[user@host ~]$ optirun --no-xorg luxmark
[user@host ~]$ cat /proc/acpi/bbswitch
0000:01:00.0 ON
[user@host ~]$ lsmod | grep nvidia
nvidia_uvm             25990  0 
nvidia              10736148  1 nvidia_uvm
...

Using softdep is even worse. Now the UVM module is loaded every time, so DGPU shut down would fail even after pure OpenGL workload.

[user@host ~]$ optirun glxspheres64
Polygons in scene: 62464
Visual ID of window: 0x20
Context is Direct
OpenGL Renderer: GeForce GTX 765M/PCIe/SSE2
62.439642 frames/sec - 69.682640 Mpixels/sec
60.007762 frames/sec - 66.968662 Mpixels/sec
[user@host ~]$ cat /proc/acpi/bbswitch
0000:01:00.0 ON
[user@host ~]$ lsmod | grep nvidia
nvidia_uvm             25990  0 
nvidia              10736148  1 nvidia_uvm
...

Can someone point me to the relevant part in the Bumblebee source?

[amonakov]

Here.

It calls rmmod directly rather than modprobe -r. Sigh. I've fixed that on the "develop" branch; you can use bumblebee-git from AUR to test.

[Lekensteyn]

This change can break distributions which have a mismatch between the module filename and the module name. If modprobe nvidia-xxx gets loaded (/lib/modules/.../nvidia-xxx.ko), the module nvidia will appear. Then modprobe -vr nvidia complains that no such module exists if there is no modprobe config containing alias nvidia nvidia-xxx.

modprobe -vr nvidia-xxx does not error out, but neither does it remove a module. This was tested with kmod 13 by renaming a random module, run depmod and invoke the commands. Then compare against adding an alias confg.

At least Ubuntu will be affected by this.

[amonakov]

I'm pretty sure Ubuntu and Debian install a modprobe alias, but Bumblebee's modprobe blacklist may cause that alias to be ignored.

[Lekensteyn]

Last time I checked, that alias file is only enabled when the "alternative" for libGL.so is set to nvidia (instead of mesa).

[Tanner-Lin]

Amonakov's commit works well on my Arch install.

@Lekensteyn Will it be more reliable if Bumblebee keeps calling rmmod for unloading, but checks dependency via lsmod or something before that?

[amonakov]

@Lekensteyn, I think you're right, but I don't see a good reason to not install the alias unconditionally.

Unless they are willing to sanitize packaging, I think that just allowing to specify a custom unload command in the daemon config file is sufficient.

I wonder if they rename nvidia-uvm.ko to something like nvidia-uvm-331.ko too?

[amonakov]

Actually I don't see a good reason to rename the module from nvidia.ko in the first place. It caused noticeable pain for Debian users when it moved to nvidia-current.ko.

@Vincent-C, can you look into this and ask the Debian nVidia maintainer about avoiding the kernel module renaming?

[amonakov]

Can bumblebee-nvidia package add the appropriate aliases itself in the bumblebee.modprobe file?

[Lekensteyn]

That is a possibility, there is currently a bunch of blacklist lines.

...but looking at the lsmod output above, is that actually going to help? nvidia-uvm depends on nvidia. Won't this result in a "module already in use" error?