AppID detection for non-Steam games doesn't work in the Steam Flatpak

[doraskayo]

Description

The elaborate trick to get the AppID of a non-Steam relies upon the assumption that executed non-Steam processes are descendants (in a parent-child relationship) of the reaper process.

This assumption breaks in the Steam Flatpak, becasue the Flatpak backend of pressure-vessel relies upon flatpak-portal (through a DBus API) to spawn non-Steam games inside a Flatpak subsandbox. This means that non-Steam game processes aren't directly executed by pressure-vessel, and therefore aren't descendants of the reaper process.

Since the AppID of non-Steam game windows can't be detected, gamescope doesn't give them focus and they remain forever in the background.

How to reproduce

Setup

Flatpak requires a separate DBus session in order to spawn processes through flatpak-portal in the context of separate X displays (such as gamescope's).

So either create a separate desktop session for gamescope + Steam Flatpak in your session manager of choice, or use the following scripts:

$ cat ./run-steam.sh
#!/bin/env bash

dbus-update-activation-environment --systemd \
        DBUS_SESSION_BUS_ADDRESS \
        DESKTOP_SESSION \
        DISPLAY \
        XAUTHORITY \
        DESKTOP_SESSION \
        XDG_CURRENT_DESKTOP \
        XDG_MENU_PREFIX \
        XDG_SESSION_CLASS \
        XDG_SESSION_DESKTOP \
        XDG_SESSION_TYPE

flatpak run com.valvesoftware.Steam -steamos3 -steampal -steamdeck -gamepadui

$ cat ./run-gamescope.sh
#!/bin/env bash

dbus-run-session -- gamescope --steam -f -- ./run-steam.sh

Note: make sure to execute Steam with -steamos3 -steampal -steamdeck -gamepadui (you may need to opt-into the Steam Deck version of Steam by following this short guide). This is important and currently required because of ValveSoftware/steam-for-linux#8513. In short, only the Steam Deck version of the Steam UI currently supports focusing non-Steam game windows through gamescope at all.

Reproduction steps

1. Install the Flatpak version of Steam from Flathub.
2. Run it through gamescope in a new DBus session. (e.g., execute ./run-gamescope.sh above.)
3. Run a non-Steam game using official Proton builds or the Steam Linux Runtime.
4. Observe that the non-Steam game windows never gain focus.
   • You can use xwininfo -root -tree to confirm that the windows have actually opened. Just make sure to set the DISPLAY environment variable to gamescope's X display.

---

cc: @smcv

[misyltoad]

Steam Flatpak messing with the process tree in this way seems like the real issue here.

Both Gamescope and Steam make a lot of assumptions about the layout of the process tree, I imagine this breaks other things in Steam too.

Does killing games, and using Steam Input, other Steam API features even work in Steam Flatpak in non-Steam games because of this?..

[smcv]

Run a non-Steam game

Does this have to be a Windows non-Steam game via Proton, or does the Steam Deck branch of Steam use pressure-vessel for native Linux games too? My understanding was that native Linux non-Steam games do not normally use pressure-vessel.

Steam Flatpak messing with the process tree in this way seems like the real issue here.

There is no alternative to this. Flatpak apps are not allowed to create new user namespaces "below" the Flatpak sandbox, only in parallel, because if they could create new user namespaces, then they would be able to use that for a sandbox escape similar to CVE-2021-41133.

Does killing games, and using Steam Input, other Steam API features even work in Steam Flatpak in non-Steam games because of this?..

Killing games is special-cased for this arrangement: instead of immediately sending SIGKILL to the reaper and its children, Steam sends a SIGTERM to the pressure-vessel-launch process (or steam-runtime-launch-client in recent betas), which results in it communicating with flatpak-portal and asking it to send SIGTERM to pressure-vessel-adverb. The adverb (which is a subreaper, like Steam's reaper) sends SIGTERM to child processes, waits a short time for them to exit, and sends SIGKILL to any survivors.

I believe Steam Input also works.

pressure-vessel could wrap the Flatpak subsandbox in a reaper process if that would help, although it would still need to use pressure-vessel-adverb as a subreaper to get the process-killing behaviour.

[misyltoad]

How does Steam communicate for IPC if its in a separate sandbox?

[smcv]

It depends on the type of IPC in use (there are lots, and they interact with namespaces differently), but in general the answer is that Flatpak is instructed to share the IPC mechanism between the original sandbox (the one with the Steam client) and the subsandbox (the one with the game).

For kill() and similar process-ID-oriented mechanisms, flatpak/flatpak#4060 added an option to share the pid namespace between the Steam client and the subsandbox, and we use it.

For SysV IPC (semaphores and message queues), the Steam Flatpak app has --share=ipc, so the Steam client and the subsandbox both share the IPC namespace with the host system.

For filesystem-backed mechanisms (shared memory, AF_UNIX sockets, or pipes), flatpak/flatpak#4120, flatpak/flatpak#4093 and flatpak/flatpak#4124 share $XDG_RUNTIME_DIR, /tmp and /dev/shm (respectively) between the Steam client and the subsandbox. The subsandbox also shares ~/.steam with the Steam client (pressure-vessel does not attempt to put a security boundary between Steam and games).

Abstract AF_UNIX sockets are part of the network namespace, and Steam has --share=net, so the Steam client and the subsandbox both share abstract AF_UNIX sockets with the host.

[doraskayo]

@smcv:

Run a non-Steam game

Does this have to be a Windows non-Steam game via Proton, or does the Steam Deck branch of Steam use pressure-vessel for native Linux games too? My understanding was that native Linux non-Steam games do not normally use pressure-vessel.

You're right. The issue reproduces only when pressure-vessel is actually used, such as when using official Proton builds or the Steam Linux Runtime. I confirmed that this is also true with the Steam Deck branch of Steam. I clarified this fact in the issue description.

pressure-vessel could wrap the Flatpak subsandbox in a reaper process if that would help, although it would still need to use pressure-vessel-adverb as a subreaper to get the process-killing behaviour.

As far as I can tell, the non-Steam game AppId detection in gamescope looks at the command line used to execute its anscestor reaper process, presumably because there's no "sane" way to obtain the AppId in this case (yet?). Here's an example of such a command line, from my tests:

/home/dor/.var/app/com.valvesoftware.Steam/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=2929382885 -- /media/storage/SteamLibrary/steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point --verb=waitforexitandrun -- /media/storage/SteamLibrary/steamapps/common/SteamLinuxRuntime/scout-on-soldier-entry-point-v2 -- /media/storage/SteamExternal/glxgears/glxgears

Spawning an appropriate reaper process inside the subsandbox would indeed solve the issue. However, it must similarly have the "AppId" command line argument like the original for gamescope's logic to work. Generally, it may be best to have pressure-vessel pass the exact same command-line arguments to its reaper process as was passed to the original. For example, in the case above, like so: path/to/reaper SteamLaunch AppId=2929382885 -- ....