Games freeze after a few seconds and overlay doesn't work on NVIDIA Wayland

[Virkkunen]

System info:

• OS: EndeavourOS x86_64
• Kernel: Linux 6.10.8-1-cachyos
• DE: KDE Plasma 6.1.4 with Wayland
• CPU: AMD Ryzen 7 7800X3D
• GPU: NVIDIA RTX 3080, nvidia-open-beta-dkms 560.35.03-1
• egl-wayland 4:1.1.16-1
• gamescope 3.15.3-1

The system info shows what I'm running now, but this issue has been present since gamescope 3.12.xx, nvidia and nvidia-open 550 (with and without GSP firmware enabled). I also tried gamescope-nvidia from the AUR as it contains some patches for NVIDIA cards, but I have not noticed any difference for me, runs and freezes the same.

I'm trying to run my games with gamescope so I can play them in HDR, the problem is that they'll freeze seconds or minutes after started, requiring a restart. Audio keeps playing and on multiplayer games I am still connected according to my friends in parties, but no image and input works for me.

I've tried with Hunt: Showdown, Fallout 4, Deep Rock Galactic, TEKKEN 8, No Man's Sky and Deadlock (depsite not having HDR, still freezes). The games run perfectly fine without gamescope, all on Steam.

My launch parameters are:

gamescope -f -e -W 3840 -H 2160 -r 165 --hdr-enabled --force-grab-cursor -- %command%

Some people on Fedora 40 say that you don't need the --hdr-enabled flag as it will auto detect HDR, but it's not my case. The -e flag also doesn't do anything, although on gamescope 3.14.2 when I tried opening the overlay, I'd lose input on my game, as if the overlay was there but didn't correctly display, requiring me to press ESC to have input on my game again. I've also tried disabling the overlay from the game's settings on Steam, but the issue still happens.
I also tried with GAMESCOPE_WSI=0 and --backend sdl as people said this would fix the freezing on NVIDIA, but it didn't.

This is the console output while running the game:

------------------------------------------------------------------------------------------------------
Starting new game recording session for 594650
Video bit rate for game 594650 is 24.0 Mbps
Creating new encoder for codec 5
Encoder init succeeded, now have 1 encoder(s)
Refreshing process list for game 594650
1 game pids rooted at pid 45970:
- 45970 (will record this process tree)
GameScope focus changed to appID 0
Recording system audio
PulseAudio: Sink alsa_output.usb-GuangZhou_FiiO_Electronics_Co._Ltd_FiiO_USB_DAC-E10-00.analog-stereo is default
PulseAudio: Adding sink[63] alsa_output.pci-0000_01_00.1.hdmi-stereo with priority 1, monitor is alsa_output.pci-0000_01_00.1.hdmi-stereo.monitor
PulseAudio: Adding sink[68] alsa_output.pci-0000_11_00.1.hdmi-stereo-extra3 with priority 1, monitor is alsa_output.pci-0000_11_00.1.hdmi-stereo-extra3.monitor
PulseAudio: Adding sink[80] alsa_output.usb-GuangZhou_FiiO_Electronics_Co._Ltd_FiiO_USB_DAC-E10-00.analog-stereo with priority 3, monitor is alsa_output.usb-GuangZhou_FiiO_Electronics_Co._Ltd_FiiO_USB_DAC-E10-00.analog-stereo.monitor
PulseAudio: Recording on device alsa_output.usb-GuangZhou_FiiO_Electronics_Co._Ltd_FiiO_USB_DAC-E10-00.analog-stereo.monitor
Audio mix: start=19307485910, returned=0
Audio source [System Pulse]: init=0, adjustment=0, through=0, last_start=0, mixed=0, drop_before=0, drop_after=0
PulseAudio: Connected to device alsa_output.usb-GuangZhou_FiiO_Electronics_Co._Ltd_FiiO_USB_DAC-E10-00.analog-stereo.monitor (80, not suspended).
PulseAudio: Using sample spec 's16le 2ch 48000Hz', channel map 'front-left,front-right'.
[gamescope] [Info]  vulkan: selecting physical device 'NVIDIA GeForce RTX 3080': queue family 2
[gamescope] [Info]  vulkan: physical device supports DRM format modifiers
[gamescope] [Info]  wlserver: [backend/headless/backend.c:67] Creating headless backend
[gamescope] [Info]  xdg_backend: Seat name: 
[gamescope] [Error] vulkan: vkGetPhysicalDeviceFormatProperties2 returned zero modifiers for DRM format 0x38344241 (VkResult: 0)
[gamescope] [Error] vulkan: vkGetPhysicalDeviceFormatProperties2 returned zero modifiers for DRM format 0x38344258 (VkResult: 0)
[gamescope] [Info]  vulkan: supported DRM formats for sampling usage:
[gamescope] [Info]  vulkan:   AR24 (0x34325241)
[gamescope] [Info]  vulkan:   XR24 (0x34325258)
[gamescope] [Info]  vulkan:   AB24 (0x34324241)
[gamescope] [Info]  vulkan:   XB24 (0x34324258)
[gamescope] [Info]  vulkan:   RG16 (0x36314752)
[gamescope] [Info]  vulkan:   NV12 (0x3231564E)
[gamescope] [Info]  vulkan:   AB4H (0x48344241)
[gamescope] [Info]  vulkan:   XB4H (0x48344258)
[gamescope] [Info]  vulkan:   AB30 (0x30334241)
[gamescope] [Info]  vulkan:   XB30 (0x30334258)
[gamescope] [Info]  vulkan:   AR30 (0x30335241)
[gamescope] [Info]  vulkan:   XR30 (0x30335258)
[gamescope] [Info]  wlserver: Using explicit sync when available
[gamescope] [Info]  wlserver: Running compositor on wayland display 'gamescope-0'
[gamescope] [Info]  wlserver: [backend/headless/backend.c:17] Starting headless backend
[gamescope] [Info]  wlserver: Successfully initialized libei for input emulation!
[gamescope] [Info]  wlserver: [xwayland/server.c:107] Starting Xwayland on :1
[gamescope] [Info]  pipewire: stream state changed: connecting
[gamescope] [Info]  pipewire: stream state changed: paused
[gamescope] [Info]  pipewire: stream available on node ID: 108
[gamescope] [Info]  xwm: Embedded, no cursor set. Using left_ptr by default.
[gamescope] [Info]  vblank: Using timerfd.
[gamescope] [Info]  xdg_backend: PreferredMetadata: Red: 0.708 0.292, Green: 0.17 0.797, Blue: 0.131 0.046, White: 0.3127 0.329, Max Luminance: 1000 nits, Min Luminance: 0 nits, Max Full Frame Luminance: 351 nits
[gamescope] [Info]  edid: Patching res 800x1280 -> 3840x2160
Refreshing process list for game 594650
2 game pids rooted at pid 45970:
- 45970 (will record this process tree)
- 46121
The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning:          Unsupported maximum keycode 708, clipping.
>                   X11 cannot support keycodes above 255.
> Warning:          Could not resolve keysym XF86KbdInputAssistPrevgrou
> Warning:          Could not resolve keysym XF86KbdInputAssistNextgrou
Errors from xkbcomp are not fatal to the X server
GameScope focus changed to appID 0
Refreshing process list for game 594650
3 game pids rooted at pid 45970:
- 45970 (will record this process tree)
- 46121
- 46257
[gamescope] [Warn]  xwm: got the same buffer committed twice, ignoring.
Audio mix: start=19307485910, returned=2880209
Audio source [System Pulse]: init=19307510275, adjustment=0, through=2880689, last_start=2880209, mixed=2879520, drop_before=0, drop_after=0
Audio source [System Pulse]: Signal levels: -35.5dB, -47.3dB
GameScope focus changed to appID 0
Window title set to Steam
Window title set to
[gamescopereaper] [Info]  reaper: Parent of gamescopereaper was killed. Killing children.
[gamescope] [Info]  launch: Primary child shut down!
(EE) failed to read Wayland events: Broken pipe
Game 594650 created interface STEAMAPPS_INTERFACE_VERSION008 / 
Game 594650 created interface STEAMAPPTICKET_INTERFACE_VERSION001 / 
Game 594650 created interface STEAMUSERSTATS_INTERFACE_VERSION012 / 
Game 594650 created interface SteamController008 / 
Game 594650 created interface SteamFriends017 / 
Game 594650 created interface SteamInput006 / 
Game 594650 created interface SteamMatchMaking009 / 
Game 594650 created interface SteamUser017 / User
Game 594650 created interface SteamUser023 / 
Game 594650 created interface SteamUser023 / User
Game 594650 created interface SteamUtils007 / Utils
Game 594650 created interface SteamUtils010 / 
Game 594650 method call count for IClientUserStats::GetAchievement : 37
Game 594650 method call count for IClientUserStats::StoreStats : 3
Game 594650 method call count for IClientUserStats::GetStat : 12
Game 594650 method call count for IClientUserStats::DeprecatedPublic_RequestCurrentStats : 1
Game 594650 method call count for IClientAppManager::GetCurrentLanguage : 1
Game 594650 method call count for IClientAppManager::GetActiveBeta : 1
Game 594650 method call count for IClientAppManager::IsAppDlcInstalled : 53
Game 594650 method call count for IClientAppManager::GetAppContentInfo : 2
Game 594650 method call count for IClientApps::BGetDLCDataByIndex : 53
Game 594650 method call count for IClientApps::GetDLCCount : 1
Game 594650 method call count for IClientMatchmaking::GetLobbyOwner : 6
Game 594650 method call count for IClientMatchmaking::SetLobbyData : 1
Game 594650 method call count for IClientMatchmaking::GetLobbyMemberByIndex : 66
Game 594650 method call count for IClientMatchmaking::GetNumLobbyMembers : 73
Game 594650 method call count for IClientMatchmaking::CreateLobby : 1
Game 594650 method call count for IClientUtils::RecordSteamInterfaceCreation : 14
Game 594650 method call count for IClientUtils::InitFilterText : 1
Game 594650 method call count for IClientUtils::GetAPICallResult : 1
Game 594650 method call count for IClientUtils::GetAppID : 22
Game 594650 method call count for IClientUtils::GetImageRGBA : 1
Game 594650 method call count for IClientUtils::GetImageSize : 1
Game 594650 method call count for IClientFriends::GetFriendPersonaName_Public : 1
Game 594650 method call count for IClientFriends::RequestUserInformation : 1
Game 594650 method call count for IClientFriends::GetFriendByIndex : 52
Game 594650 method call count for IClientFriends::GetFriendCount : 2
Game 594650 method call count for IClientFriends::GetLargeFriendAvatar : 1
Game 594650 method call count for IClientFriends::GetPersonaName : 1
Game 594650 method call count for IClientUser::GetAuthTicketForWebApi : 1
Game 594650 method call count for IClientUser::GetAuthSessionTicketV2 : 1
Game 594650 method call count for IClientUser::GetAppOwnershipTicketExtendedData : 1
Game 594650 method call count for IClientUser::GetSteamID : 10
Game 594650 method call count for IClientUser::BLoggedOn : 8
Uploaded AppInterfaceStats to Steam
PulseAudio: Context connection terminated
Stopped game recording session
------------------------------------------------------------------------------------------------------
Game Recording - game stopped [gameid=594650]
Removing process 45970 for gameID 594650

The game froze a few seconds after the Audio mix: start=19307485910... line, I had to alt tab and force steam to quit it.

I also have a PROTON_LOG=1 log:
steam-594650.log

Sometimes on console I notice a lot of lines like "reaping gameoverlay" or something right before the game freezes, but it is not consistent.

For whatever it is worth, I have vk-hdr-layer-kwin6-git installed (as it's needed for HDR to work on mpv, and it does without issues) but I also tried removing it and playing, but games still freeze.

Maybe something is going on with (x)Wayland and the window it creates for the games, but I am clueless on how to proceed. I need help trying to troubleshoot this and providing better logs, so if needed please guide me on how to do so.

[valgusk]

does adding VKD3D_DISABLE_EXTENSIONS=VK_KHR_present_wait help? It helped with elden ring hdr for me but it was not a gamescope specific bug, since it also manifested on native wayland wine.

[Virkkunen]

does adding VKD3D_DISABLE_EXTENSIONS=VK_KHR_present_wait help? It helped with elden ring hdr for me but it was not a gamescope specific bug, since it also manifested on native wayland wine.

Well it seems that does solve the issue, thank you! I've only tried with Hunt at the moment, the game didn't freeze for 30 minutes now, but I'll still conduct more testing. Seems promising nonetheless.

EDIT: Haven't have a single freeze on the games I tested so far. Some running for 3 hours and not an issue, so I guess that's solved. However, you've mentioned that you had this same issue on native wayland wine, so I wonder who's responsible for analysing this? I want to open an issue if possible, or contribute if there's one already, on the proper channels for this case.

[brain-anti-freeze]

does adding VKD3D_DISABLE_EXTENSIONS=VK_KHR_present_wait help? It helped with elden ring hdr for me but it was not a gamescope specific bug, since it also manifested on native wayland wine.

Thanks for this ! I've had this freezing issue for ages with the DRM backend (it happens with either ENABLE_GAMESCOPE_WSI=1 or just enabling hdr output). Its a shame there is only a way to disable VK_KHR_present_wait in VKD3D since the issue also happens with DXVK.

[Virkkunen]

Closed this issue as it is not related to gamescope, but rather Wayland, Nvidia, Wine or something else.

[CapSel]

What if adding VKD3D_DISABLE_EXTENSIONS=VK_KHR_present_wait did not help in my case? I tried testing with glxgears and vkcube. Both freeze after few seconds.