gamescope having CAP_SYS_NICE break the steam overlay in nested mode

[Oschowa]

I've seen that gamescope now re-nices itself and uses realtime priority compute if CAP_SYS_NICE is available, so I tried to add this capability to the gamescope executable with either:
sudo setcap cap_sys_nice+pie or
sudo setcap cap_sys_nice+pe
While this does give gamescope the ability to re-nice itself as confirmed by the log output, it seems to break the steam overlay and consequently steam-input when running games for steam in gamescope via launch options.
Removing the capability from the gamescope executable makes the overlay work again.

[Plagman]

Oh, thanks for catching that. It's likely causing LD_PRELOAD to get skipped, hm.

[emersion]

Hm, but LD_PRELOAD should work fine in the game right? (Because it doesn't have CAP_SYS_NICE?)

[Plagman]

We want the overlay hooked to gamescope, not the game, when running it as a sub-command. Otherwise the overlay isn't 1:1 screen-space, and you get input hooking at the wrong level.

[emersion]

Hmm, I see. I don't see an easy solution.

[Samsagax]

I tested OP's capabilities setting and seem to ignore all environment variables.

I usually do:

VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/intel_icd.x86_64.json gamescope -- glxgears

to use the iGPU (since Nvidia wont work) and with the set CAP_SYS_NICE capability it won't select the intel gpu so it wont start with nvidia errors.

Would it be better to use pam_limit module for renice itself?

[kasha695]

I'm also having a similar issue.

Entering sudo setcap 'CAP_SYS_NICE=eip' /usr/bin/gamescope in a Terminal allows gamescope to re-nice itself but whenever I launch a game through steam the overlay doesn't work.

Once I've set gamescope back to what it was before, sudo setcap 'CAP_SYS_NICE-eip' /usr/bin/gamescope, it works again through Steam no problem.

[dgmvecuador]

I have the same issue, is there a workaround to this??

[DisplayTalk]

Still happens here too.

[abigrock]

Running the setcap command shown in the Arch Wiki, I can confirm the steam overlay is broken for me as well.
I'm using the following launch options for games under steam:

gamescope -w 3440 -h 1440 -e -- gamemoderun %command%

[InsaneCake]

Same here. Overlay in gamescope doesn't work after executing sudo setcap 'CAP_SYS_NICE=eip' /usr/bin/gamescope. Works again after sudo setcap 'CAP_SYS_NICE-eip' /usr/bin/gamescope

[abigrock]

This has been driving me nuts so I looked into other methods to allow automatically renicing gamescope. I installed ananicy-cpp and created a simple rules file under /etc/ananicy.d/gamescope.rules with the following:

{"name": "gamescope", "nice": -20}

I then started/enabled the ananicy-cpp systemd service. Now I can launch a game using gamescope and the steam overlay still works. As expected gamescope is running at nice level -20 as it does when CAP_SYS_NICE=eip is enabled for it. Definitely more of a workaround than a full fix.

[igo95862]

This most likely because of the dynamic linker secure mode: https://man.archlinux.org/man/ld.so.8#Secure-execution_mode

Because linker detects the extra capabilities it will prevent LD_PRELOAD from working.

I wonder if running entire Steam with the CAP_SYS_NICE instead of setting capabilities on gamescope can be a work-around. (for some reason the overlay gets disabled in gamescope no matter the capabilities on my system)

[ammgws]

Just wondering, is CAP_SYS_NICE set on the Steam Deck when in game mode?

[Joshua-Ashton]

Yes, but the overlay on Steam Deck uses an overlay plane and doesn't hook into the game like on desktop

[ammgws]

If you launch gamescope from a tty on a desktop does it also do that, or is the only specific to the Steam Deck?