Undesired GPU offloading behaviour with hybrid (NVIDIA) graphics

[Ner0theHer0]

Apologies in advance as I'm sure part of this issue is the fault of NVIDIA.

Specs:
Host: Inspiron 16 Plus 7620 (no mux switch)
CPU: Intel i7-12700H
iGPU: Alder Lake-P Iris Xe
dGPU: GeForce RTX 3060 Mobile / Max-Q
NVIDIA drivers tried: 580.82.09, 580.119.02, 590.48.01

After a fresh COSMIC install, I noticed that a number of desktop related activities appeared to have poor performance on external monitors. This includes heavy stuttering/sub 30fps rendering when mousing over the desktop, mousing over full screen COSMIC apps (terminal, file explorer), window maximise/minimise/snap animations, top bar and dock auto-hide animations, etc.

Since I had come from 22.04 on x11, I assumed this was a wayland issue. However, I found I could not replicate the issue on gnome wayland and KDE plasma on Fedora 43. I did however find the same behaviour present in the Fedora COSMIC spin.

After a fair bit of digging and playing around with different installs, it appears that this issue comes down to how aggressively COSMIC seems to offload workloads to the dGPU. I found that gnome and KDE do not offload these desktop activities to the dGPU, whilst COSMIC does. When COSMIC offloads to the dGPU, it is often in the NVIDIA PowerMizer low performance level (level 0), which causes the aforementioned performance issues. When an application forces the dGPU to spin up a little more (performance level 2/3) the lag disappears. I noticed this when starting OBS to get a screen recording of the behaviour, which unfortunately pushed the GPU enough to keep it in levels 2 and 3.

It is worth noting that the dGPU seems very resistant to move to the performance level 2 and 3 even in COSMIC's high performance mode. It instead tends to stay in performance level 1 and results in constant sub 30fps outside of non COSMIC applications. In balanced mode, I cannot get the GPU to go higher than level 1 with normal desktop use, which means there is always stuttering. Worth noting here that between level 1 and 2 there is a huge jump in memory transfer rate, which appears to be part of the issue.

Right now there are a few soft workarounds, but nothing that is able to get a constant smooth experience in COSMIC. The following stops the GPU from going into level 0, but still has heavy stuttering if the GPU goes back to level 1 (which takes about 2 seconds):

• Install the 590 NVIDIA driver which has a fix for the "Preferred Mode" PowerMizer setting not working in 580
• Using nvidia-smi -lgc and nvidia-smi -lmc to force the gpu to stay at higher idle clocks. Unfortunately the bottleneck here appears to be the memory transfer speed, which isn't configurable. This just stops it from going back to level 0.
• Having a video or something that leverages the GPU playing on the laptop's internal monitor. Stupid, but it keeps the GPU awake so using the desktop is smooth

I'm sure you guys are bombarded with random NVIDIA issues since the full release, but either of the following would fix this issue:

• Allow the GPU offload aggressiveness to the configurable for the DE.
• Based on what I've read, the compositor should have some control over the dGPU power state. If possible, allow the DE to keep the dGPU above a certain power state when an external monitor is detected.

Unfortunately DE's are much lower level than what I'm familiar with, so apologies if either of those suggestions are simply not possible or are out of the DE's control. Thank you guys for all the hard work, the experience outside of this so far has been fantastic :)

[Drakulix]

Allow the GPU offload aggressiveness to the configurable for the DE.

Not feasible nor really what's going on here. COSMIC uses an entirely different model to facilitate copies between multiple GPUs on your system and will at times use the dGPU to render the framebuffer of outputs directly connected to it. The copies are likely what are causing the issue here, given your observation on the memory clock.

KDE at least from my knowledge is planning to do something similar in the long run, as it gives much greater control over certain resource usages, which can lead to drastically better performance in certain cases.

We won't rewrite COSMIC to use an older model, where everything is handled on the iGPU even when it makes no sense just to fix re-clocking issues on certain devices. The "aggressiveness" you are talking about is not as configurable as you might assume.

Based on what I've read, the compositor should have some control over the dGPU power state. If possible, allow the DE to keep the dGPU above a certain power state when an external monitor is detected.

That is sadly not true. We simply submit rendering jobs like every other application (though cosmic-comp creates what is called a high-priority context, but that doesn't seem to help in your case).

We don't have any additional way to keep the GPU in higher power states though.

Some additional notes:

• I am not seeing this issue on a multitude of nvidia laptops I have in use for testing, so this might also be linked to firmware in some cases.
• I do consider this a driver bug, that needs to be ironed out on nvidia's side.

EDIT:

• I have a PR in the works, that changes things around a bit when it comes to how quickly cosmic-comp is purging some GPU resources to allow the dGPU to enter deeper sleep states (something that obviously can not happen, if you have a display connected to it). On the off-chance that this influences the behaviour you are seeing in any way, I can give you a ping, once that PR is ready for testing.

[Ner0theHer0]

Thank you very much for that. I still have cosmic-greeter installed, so will keep an eye out and update this issue if there is an upstream nvidia fix at some point :)

[shawnbarton]

Just stopping by to +1 this issue.

Host: XMG Fusion E24
CPU: Intel i9-14900HX
iGPU: Raptor Lake-S UHD Graphics
dGPU: GeForce RTX 4070 Max-Q / Mobile
NVIDIA drivers tried: 580.173.02, 595.84 (open)