NVIDIA GPU is not supported since 4.8

[Thaodan]

After updating to linux 4.8 the nvidia driver says your gpu isn't supported when trying to access with primus:

[Okt18 12:54] bbswitch: enabling discrete graphics
[  +0,926684] nvidia: module license 'NVIDIA' taints kernel.
[  +0,000001] Disabling lock debugging due to kernel taint
[  +0,529426] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=none
[  +0,000037] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:13b0)
              NVRM: installed in this system is not supported by the 370.28
              NVRM: NVIDIA Linux driver release.  Please see 'Appendix
              NVRM: A - Supported NVIDIA GPU Products' in this release's
              NVRM: README, available on the Linux driver download page
              NVRM: at www.nvidia.com.
[  +0,000014] nvidia: probe of 0000:01:00.0 failed with error -1
[  +0,000053] nvidia-nvlink: Nvlink Core is being initialized, major device number 241
[  +0,000033] NVRM: The NVIDIA probe routine failed for 1 device(s).
[  +0,000002] NVRM: None of the NVIDIA graphics adapters were initialized!
[  +0,000002] nvidia-nvlink: Unregistered the Nvlink Core, major device number 241

uname:
uname -a Linux hellion 4.8.2-pf #1 SMP PREEMPT Tue Oct 18 10:19:55 CEST 2016 x86_64 GNU/Linux

[bluca]

What GPU is that?

[Thaodan]

Nvidia Quadro M2000M (in Dell M7510)

There are users that use Manjoro Linux reporting similar issues [1]

[1] https://forum.manjaro.org/t/bumblebee-not-working-under-4-8-kernel/10676/8

[olifre]

That seems to be the same I described here:
https://devtalk.nvidia.com/default/topic/971733/linux/nvidia-gtx-960m-not-supported-anymore-by-370-28-/?offset=5#4999717
In short: Until nvidia changes their driver, add pcie_port_pm=off to your kernel commandline (you will then not get everything from the new energy savings introduced by 4.8).

Note that this affects everybody who is part of the following group (see my original linked post):

Anybody using Kernel >=4.8, with a system for which the nvidia card is not the primary output (otherwise it could never enter D3), and a BIOS released >=2015. 

The most ugly thing of this story is that the kernel in any case uses pcie_port_pm=off if your BIOS / UEFI release date is <=2014.

EDIT: This also means it's not really a bumblebee issue, but a problem with kernel 4.8 and nvidia, which just happens to show up on laptops for which nvidia is not the primary card. Apparently I was the first to report this in their forums, but good to know I am not alone with this issue.

[Thaodan]

As far as I read the first commit revered in the forum post:
If the card is advertised as hotplug the pm runtime won't power it to d3 cold
and as bumblebee/optimus is somekind of a hotplug mechnism this should apply
to the card too or not?
Is it possible to set this flag in bbswitch or somewhere else?
Or would it be better to place this flag in the gpu driver?

[olifre]

Or would it be better to place this flag in the gpu driver?

Definitely that. That's since this also affects machines even without bumblebee.
If you just boot without starting bumblebeed and never loading bbswitch, the nvidia driver will still fail to detect the card - so the driver claiming the device (nvidia) should take care.
I guess it is even possible to do that in a better way than claiming to be hotpluggable (which the pcie interface to which the card is connected is not really...): The nvidia driver could check whether the card is in d3cold and power it up if necessary.

[Thaodan]

So its a "bug" that needs to be fixed in nvidia/nouveau.
But the problem is that nvidia is slow.
Would it be possible to do one of the solutions either in bumblebee or
bbswitch?

[olifre]

But the problem is that nvidia is slow.

AFAIK they do not even claim to support kernel 4.8 yet, so if we're lucky, it could already be in the next driver release.

Would it be possible to do one of the solutions either in bumblebee or bbswitch?

I'm not sure, but the bumblebee maintainers will know ;).

For an immediate "fix", the trick to add pcie_port_pm=off to kernel commandline is sufficient.

[Thaodan]

Ah I didn't noticed that you're not a maintainer (:

Should the dirty fix change something for the other devices in the "typical"
notebook?

[olifre]

Should the dirty fix change something for the other devices in the "typical" notebook?

It could lead to slightly higher power consumption since unused PCI bridges can not enter the deepest sleep state anymore.
However, since this (pcie_port_pm=off) was in any case what was done in kernel 4.7 and before, and it's also what the kernel 4.8 still does on any machine with a BIOS released before 2015, I don't expect this is significant on a "typical" machine.
But I'm not sure whether this is also true for the most modern Skylake / Broadwell machines, nowadays things start becoming more and more sensitive and power consumption stays high unless everything on the busses is in the deepest sleep states.

At least I can promise you things won't be worse over kernel 4.7 ;).

[Lekensteyn]

At least I can promise you things won't be worse over kernel 4.7 ;).

Very true, the boot option reverts to the pre-4.8 behavior.

Have you (via udev rules or some other "laptop mode tools") enabled runtime PM? You can check that by reading /sys/bus/pci/devices/0000:01:00:0/power/control. If it says "auto", then it is enabled. If it is "on", then I would expect it to have the same behavior as adding the boot option.

Btw, some laptops require the new 4.8 method or else may experience memory corruption (see the commit message of https://git.kernel.org/linus/692a17dcc2922a91c6bcf11b3321503a3377b1b1).

[olifre]

Have you (via udev rules or some other "laptop mode tools") enabled runtime PM? You can check that by reading /sys/bus/pci/devices/0000:01:00:0/power/control. If it says "auto", then it is enabled. If it is "on", then I would expect it to have the same behavior as adding the boot option.

Yes, runtime PM is active on that machine: I'm using laptop-mode-tools.
In addition, I have enabled PCIe-ASPM for all ports in the UEFI (it's one with unlocked features) and I'm also using "pcie_aspm=force".
Only after all this, I could achieve maximum battery runtime almost comparable to Windows on that laptop (sadly ASPM for the NIC and card reader is not working, even in Windows, so the machine saves significantly more if I turn off the full corresponding PCI port).

Btw, some laptops require the new 4.8 method or else may experience memory corruption

Thanks a lot for the link!

[Espionage724]

This doesn't seem to affect me with kernel 4.8.3 on Solus, but I don't use Bumblebee (I just pass everything to the NV GPU with xrandr):

Linux spinesnap 4.8.3 #1 SMP Thu Oct 20 11:50:13 UTC 2016 x86_64 GNU/Linux

[    9.786656] nvidia: loading out-of-tree module taints kernel.
[    9.786660] nvidia: module license 'NVIDIA' taints kernel.
[    9.793430] nvidia 0000:01:00.0: enabling device (0006 -> 0007)
[    9.793562] nvidia-nvlink: Nvlink Core is being initialized, major device number 247
[    9.793574] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  370.28  Thu Sep  1 19:45:04 PDT 2016
[    9.806149] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  370.28  Thu Sep  1 19:18:48 PDT 2016
[    9.814270] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[   11.790229] nvidia-modeset: Allocated GPU:0 (GPU-4d140bcd-71e6-3bb9-ed1c-033e9cf5bec2) @ PCI:0000:01:00.0
[   11.790322] nvidia-modeset: Freed GPU:0 (GPU-4d140bcd-71e6-3bb9-ed1c-033e9cf5bec2) @ PCI:0000:01:00.0

I have a GTX 960M in a Acer Aspire V Nitro VN7-792G laptop.

[Thaodan]

do you use nvidia prime? When yes you wont get the issue as your egpu isn't disabled before using it.

[Lekensteyn]

Even if you use nvidia, you might still run into issues if you enable runtime PM for all devices using a udev rule or using "laptop mode tools" before the nvidia driver is loaded (and have kernel 4.8+ without the pcie_port_pm=off parameter and a new enough laptop).

[Thaodan]

Using the pm-rework branch which enables using runtime pm is bbswitch
worksaround the issue that the nvidia driver doesn't handle runtime pm.

Could someone with an older laptop that doesn't use runtime pm and someone
else with an newer laptop if the newer version works/fixes the issue?