Grant access to Gamescope socket on Steam Deck

[rakonkido]

I encounter this error every time I launch a Dolphin game through Steam Deck's Game Mode UI:

I've encountered it many times before when launching games through other Flatpaks, like those for Lutris, Heroic, and other emulators. Recently, I found that this error went away after adding support for HDR on Steam Deck to Lutris Flatpak -- itself based on the method used for enabling HDR support on Heroic and Chiaki4deck. So I'm pushing this PR that I believe will give Dolphin Flatpak access to the Gamescope socket. I'm leaving this PR as draft for now until I've run the test build to confirm it works.

Similar to PRs I have made for RetroArch, PrimeHack, and PPSSPP, I have tested and confirmed that just adding the gamescope-0 socket is enough to make the error go away, with no apparent regressions, if you have the Gamescope Flatpak installed:

flatpak install flathub org.freedesktop.Platform.VulkanLayer.gamescope

[flathubbot]

Started test build 94776

[flathubbot]

Build 94776 successful
To test this build, install it from the testing repository:

flatpak install --user https://dl.flathub.org/build-repo/77523/org.DolphinEmu.dolphin-emu.flatpakref

[flathubbot]

Started test build 94779

[flathubbot]

Build 94779 successful
To test this build, install it from the testing repository:

flatpak install --user https://dl.flathub.org/build-repo/77527/org.DolphinEmu.dolphin-emu.flatpakref

[rakonkido]

Tested the latest build and confirmed that this fixes the issue with no apparent regression.

[flathubbot]

Started test build 94796

[flathubbot]

Build 94796 successful
To test this build, install it from the testing repository:

flatpak install --user https://dl.flathub.org/build-repo/77544/org.DolphinEmu.dolphin-emu.flatpakref

[cpba]

Thank you for testing, I don't have a steam deck and there's a surprising amount of steamdeck exclusive issues.

Just so I can understand it, could you please elaborate a bit further on the nature of the issue?

Does it always happen on steam deck? does it only happen with org.freedesktop.Platform.VulkanLayer.gamescope installed? (Is that autoinstalled on steamdeck?)

Does the error entirely prevent dolphin from working? Does it only affect the vulkan renderer?

And last, there are four commits for a single line change, would you mind if I squashed before merging?

[rakonkido]

Just so I can understand it, could you please elaborate a bit further on the nature of the issue?

It's pretty complex and I'm not familiar with the details of how exactly Valve got HDR10 working on Steam Deck OLED well before HDR was otherwise even fully supported in Linux, but you can read the Chiaki4deck thread starting from about here to get more details. My mental model (which might be slightly inaccurate) of what's going on is that:

• Everything in Steam Deck's Game Mode is running inside a fullscreen Gamescope window that presents itself to the application as a sandboxed desktop / virtual display.
• The HDR capability of the actual hardware display is detected in Gamescope and then exposed to applications running inside the virtual display it presents to the application.
• Applications that optionally support HDR10 query the display's capabilities. In this case, they query the Gamescope "display" and see that it's HDR10 capable, so they can attempt to render in an HDR10 colorspace.
• The Flatpak sandbox doesn't block applications from detecting the display's HDR10 capability, but it does seem to be blocking Flatpak applications from accessing the runtime extensions for 10-bit color that Valve exposes through Gamescope; this is fixed by installing the org.freedesktop.Platform.VulkanLayer.gamescope Flatpak.
• The Flatpak sandbox is also blocking applications from actually connecting to the gamescope-0 socket to engage the HDR10 mode. This is fixed by whitelisting xdg-run/gamescope-0.

Does it always happen on steam deck? does it only happen with org.freedesktop.Platform.VulkanLayer.gamescope installed? (Is that autoinstalled on steamdeck?)

The error message I posted a picture of only shows up if you install the Gamescope Flatpak but fail to whitelist xdg-run/gamescope-0. But you need both for HDR applications running through Flatpak, including Dolphin -- in fact, when I was testing this PR, I enabled HDR post-processing effects in Dolphin's graphics settings and that caused the Steam Deck OLED's HDR mode to be correctly engaged whenever I launched a Dolphin game.

Just whitelisting xdg-run/gamescope-0 makes no difference on its own -- if you fail to install the Gamescope Flatpak, games can try to render in HDR but then they will be tonemapped down to the SDR colorspace.

Unfortunately the Gamescope Flatpak is not autoinstalled. Steam Deck users of Dolphin, Heroic, Lutris, and other applications who want to play games in HDR will need to be educated that they need to also install the Gamescope Flatpak. So far I'm not aware of a better solution; bundling Gamescope inside the Dolphin Flatpak doesn't seem like a good idea since Dolphin and Gamescope don't necessarily update at the same cadence, which could introduce incompatibilities.

Btw, this PR's change is harmless for users who have no interest in running non-Steam HDR games. Everything will go on as usual for them.

Does the error entirely prevent dolphin from working? Does it only affect the vulkan renderer?

Only Dolphin HDR post-processing effects won't work without this PR, but other than that, the error doesn't prevent Dolphin from running (you can still press OK to "power through" the error). I believe the error is from the code in Gamescope that (IIUC) uses Vulkan to create a zero-copy / low-latency virtual display layer seen by the application. I vaguely suspect that all Flatpak games would need to have xdg-run/gamescope-0 whitelisted in order to fully take advantage of Gamescope's performance benefits...

And last, there are four commits for a single line change, would you mind if I squashed before merging?

Yes please do squash. The initial commits that I later reverted included changes to the environment variables, but I realized that these are only useful for applications like Lutris or Heroic that include options to actually run games inside their own nested Gamescope instance (e.g. for launching them inside a Gamescope window on your desktop).

[cpba]

Thank you, that was very helpful.

I'll try to add some explanation to the readme when I have some time.

[bbhtt]

I have approved the change but FYI, the gamescope socket is not fixed meaning there can be a gamescope-0 gamescope-1 sockets,. if two/more are running.

This makes it incompatible with static permissions of Flatpak as you can't predict the value.

But -0 should work in most cases, so I approved it.

[rakonkido]

Makes sense. Not sure of a better way to do it. I guess it should always be gamescope-0 when running in Steam Deck's Game Mode unless they change its design.