Archlinux Segfault when launching applications

[Kagukara]

I'm using gamescope-git from the aur. With help I've been able to get the segfault debugging information.

The segfault happens when an application/game is launched with gamescope (the gamescope package from archlinux works fine). For this debugging information I used gamescope -- glxgear.

gdb /usr/bin/gamescope ~/core.96047.1000.11.gamescope-xwm.1700356873 
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/gamescope...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
(No debugging symbols found in /usr/bin/gamescope)                                                                                                                                           

warning: Can't open file /memfd:xwayland-shared (deleted) during file-backed mapping note processing

warning: Can't open file /memfd:pipewire-memfd:flags=0x0000000f,type=2,size=2312 (deleted) during file-backed mapping note processing

warning: Can't open file /dev/shm/wlroots-eiKJdD (deleted) during file-backed mapping note processing
[New LWP 96072]
[New LWP 96047]
[New LWP 96073]
[New LWP 96075]
[New LWP 96071]
[New LWP 96049]
[New LWP 96076]
[New LWP 96051]
[New LWP 96048]
[New LWP 96070]

warning: Section `.reg-xstate/96072' in core file too small.
                                                                                                                                                                                             --Type <RET> for more, q to quit, c to continue without paging--c
[Thread debugging using libthread_db enabled]                                                                                                                                                
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `gamescope -- glxgears'.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Section `.reg-xstate/96072' in core file too small.
#0  0x000055c579508eeb in ?? ()
[Current thread is 1 (Thread 0x7f78760136c0 (LWP 96072))]
(gdb) bt
#0  0x000055c579508eeb in  ()
#1  0x000055c57951d651 in  ()
#2  0x000055c57951eccf in  ()
#3  0x00007f788a2e1943 in std::execute_native_thread_routine(void*) (__p=0x55c57b443170) at /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104
#4  0x00007f7889fbd9eb in start_thread (arg=<optimized out>) at pthread_create.c:444
#5  0x00007f788a0417cc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) f 0
#0  0x000055c579508eeb in ?? ()
(gdb) f 1
#1  0x000055c57951d651 in ?? ()
(gdb) f 2
#2  0x000055c57951eccf in ?? ()
(gdb) f 3
#3  0x00007f788a2e1943 in std::execute_native_thread_routine (__p=0x55c57b443170) at /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc:104
Downloading source file /usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc
104	/usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc: Directory not empty.                                                                                                        
(gdb) f 4
#4  0x00007f7889fbd9eb in start_thread (arg=<optimized out>) at pthread_create.c:444
Downloading source file /usr/src/debug/glibc/glibc/nptl/pthread_create.c
444		ret = pd->start_routine (pd->arg);                                                                                                                                           
(gdb) f 5
#5  0x00007f788a0417cc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
Downloading source file /usr/src/debug/glibc/glibc/misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S
78		call	*%rdx		/* Call function.  */                                                                                                                                
(gdb) exit

EDIT:

System Information

• OS: Arch Linux
• WM: sway version 1.8.1
• Kernel: 6.6.1-arch1-1
• GPU: AMD Radeon RX 7900 XTX (radeonsi, navi31, LLVM 18.0.0, DRM 3.54, 6.6.1-arch1-1)
• GPU Driver: 4.6 Mesa 24.0.0-devel (git-fd80140723)

[chewi]

Not sure if it's the same issue, but 3.13.7 segfaults when starting Neverball under KDE on Gentoo, whereas 3.13.0 works fine.

[antheas]

Found the bug potentially if running under mutter or nested. Maybe it's a different bug though.

		// HACK! HACK HACK HACK
		// To avoid stutter when toggling the overlay on 
		int curLayer = frameInfo.layerCount++;

		FrameInfo_t::Layer_t *layer = &frameInfo.layers[ curLayer ];

		auto tex = vulkan_get_hacky_blank_texture();

		layer->scale.x = g_nOutputWidth == tex->width() ? 1.0f : tex->width() / (float)g_nOutputWidth;

The segmentation fault happens when accessing tex->width(), because that relies on g_output.temporaryHackyBlankImage which is set in function vulkan_make_output_images.

This function runs when BIsVRSession() || BIsHeadless() or !BIsSDLSession(). But when BIsSDLSession(), another function called vulkan_make_swapchain is called. That function does not set temporaryHackyBlankImage causing the seg fault. But that function runs repeatedly so it probably can not be set there.

However, by re-adding the initialization function in that if the compositor runs nested correctly.

// Oh no.
pOutput->temporaryHackyBlankImage = vulkan_create_debug_blank_texture();

at 3094 in ./src/rendervulkan.cpp.

Also, there's a race condition when launching gdp for debugging that causes another error. And another one where if you close the child of gamescope, it doesn't close. Don't know if I'll raise an issue about those though.

[DrShadow34]

Looks like today arch maintainers decided to update gamescope to 3.13.7-1, now I also have segmentation fault even when running
> $ gamescope -- glxgears

Downgrading to 3.13.3-1 solved this issue.

[sharkautarch]

I've also run into this issue after pulling in more recent commits into my fork
I ended up resetting my fork to a working commit HEAD at commit 04eb72b
(there may be a newer commit that also works, I just picked this commit to jump to because I remembered that it worked well)
try using this PKGBUILD which moves back to that commit: https://gist.github.com/sharkautarch/8a0cfac4207c23ec1b2d05ab1bf685f0

[simifor]

I found the offending commit, the regression was introduced with fe3dbc6, so the last good commit is 31ad691. I tested with a RX6600, kernel 6.6.1 and mesa git 898700ca647, I get the following log:

wlserver: [backend/headless/backend.c:68] Creating headless backend
vulkan: selecting physical device 'AMD Radeon RX 6600 (RADV NAVI23)': queue family 1 (general queue family 0)
vulkan: physical device supports DRM format modifiers
vulkan: supported DRM formats for sampling usage:
vulkan:   AR24 (0x34325241)
vulkan:   XR24 (0x34325258)
vulkan:   AB24 (0x34324241)
vulkan:   XB24 (0x34324258)
vulkan:   RG16 (0x36314752)
vulkan:   NV12 (0x3231564E)
vulkan:   AB4H (0x48344241)
vulkan:   XB4H (0x48344258)
vulkan:   AB48 (0x38344241)
vulkan:   XB48 (0x38344258)
vulkan:   AB30 (0x30334241)
vulkan:   XB30 (0x30334258)
vulkan:   AR30 (0x30335241)
vulkan:   XR30 (0x30335258)
vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
wlserver: Running compositor on wayland display 'gamescope-0'
wlserver: [backend/headless/backend.c:16] Starting headless backend
wlserver: [xwayland/sockets.c:63] Failed to bind socket @/tmp/.X11-unix/X0: Address already in use
wlserver: [xwayland/server.c:108] Starting Xwayland on :1
The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning:          Could not resolve keysym XF86CameraAccessEnable
> Warning:          Could not resolve keysym XF86CameraAccessDisable
> Warning:          Could not resolve keysym XF86CameraAccessToggle
> Warning:          Could not resolve keysym XF86NextElement
> Warning:          Could not resolve keysym XF86PreviousElement
> Warning:          Could not resolve keysym XF86AutopilotEngageToggle
> Warning:          Could not resolve keysym XF86MarkWaypoint
> Warning:          Could not resolve keysym XF86Sos
> Warning:          Could not resolve keysym XF86NavChart
> Warning:          Could not resolve keysym XF86FishingChart
> Warning:          Could not resolve keysym XF86SingleRangeRadar
> Warning:          Could not resolve keysym XF86DualRangeRadar
> Warning:          Could not resolve keysym XF86RadarOverlay
> Warning:          Could not resolve keysym XF86TraditionalSonar
> Warning:          Could not resolve keysym XF86ClearvuSonar
> Warning:          Could not resolve keysym XF86SidevuSonar
> Warning:          Could not resolve keysym XF86NavInfo
Errors from xkbcomp are not fatal to the X server
wlserver: [types/wlr_compositor.c:681] New wlr_surface 0x557d8934fbd0 (res 0x557d894d7ad0)
wlserver: [xwayland/server.c:273] Xserver is ready
pipewire: stream state changed: connecting
pipewire: stream state changed: paused
pipewire: stream available on node ID: 247
vulkan: Creating Gamescope nested swapchain with format 44 and colorspace 0
pipewire: renegotiating stream params (size: 1280x720)
ATTENTION: default value of option vk_khr_present_wait overridden by environment.
Selected GPU 0: AMD Radeon RX 6600 (RADV NAVI23), type: DiscreteGpu
[Gamescope WSI] Creating Gamescope surface: xid: 0x400000
[Gamescope WSI] Atom of T was wrong type. Expected XCB_ATOM_CARDINAL.
wlserver: [types/wlr_compositor.c:681] New wlr_surface 0x557d89388e80 (res 0x557d89435050)
wlserver: [types/wlr_compositor.c:681] New wlr_surface 0x557d89388980 (res 0x557d89435170)
[Gamescope WSI] Made gamescope surface for xid: 0x400000
[Gamescope WSI] Surface state:
  steam app id:                  0
  window xid:                    0x400000
  wayland surface res id:        5
  layer client flags:            0x0
  server hdr output enabled:     false
  hdr formats exposed to client: false
xwm: got the same buffer committed twice, ignoring.
The XKEYBOARD keymap compiler (xkbcomp) reports:
> Warning:          Unsupported maximum keycode 708, clipping.
>                   X11 cannot support keycodes above 255.
> Warning:          Could not resolve keysym XF86CameraAccessEnable
> Warning:          Could not resolve keysym XF86CameraAccessDisable
> Warning:          Could not resolve keysym XF86CameraAccessToggle
> Warning:          Could not resolve keysym XF86NextElement
> Warning:          Could not resolve keysym XF86PreviousElement
> Warning:          Could not resolve keysym XF86AutopilotEngageToggle
> Warning:          Could not resolve keysym XF86MarkWaypoint
> Warning:          Could not resolve keysym XF86Sos
> Warning:          Could not resolve keysym XF86NavChart
> Warning:          Could not resolve keysym XF86FishingChart
> Warning:          Could not resolve keysym XF86SingleRangeRadar
> Warning:          Could not resolve keysym XF86DualRangeRadar
> Warning:          Could not resolve keysym XF86RadarOverlay
> Warning:          Could not resolve keysym XF86TraditionalSonar
> Warning:          Could not resolve keysym XF86ClearvuSonar
> Warning:          Could not resolve keysym XF86SidevuSonar
> Warning:          Could not resolve keysym XF86NavInfo
Errors from xkbcomp are not fatal to the X server
[Gamescope WSI] Creating swapchain for xid: 0x400000 - minImageCount: 3 - format: VK_FORMAT_B8G8R8A8_UNORM - colorspace: VK_COLOR_SPACE_SRGB_NONLINEAR_KHR - flip: true
[Gamescope WSI] Created swapchain for xid: 0x400000 - imageCount: 3
[Gamescope WSI] Swapchain recieved new refresh cycle: 16.67ms
(EE) failed to read Wayland events: Broken pipe
zsh: segmentation fault (core dumped)  ./build/src/gamescope -- vkcube
[Gamescope WSI] getLargestObscuringWindowSize: xcb_query_tree failed for window 0x400000.                                                                            
[Gamescope WSI] getToplevelWindow: xcb_query_tree failed for window 0x400000.
[Gamescope WSI] canBypassXWayland: failed to get window info for window 0x400000.
[Gamescope WSI] getWindowRect: xcb_get_geometry failed for window 0x400000.
[Gamescope WSI] getWindowRect: xcb_get_geometry failed for window 0x400000.
[Gamescope WSI] getLargestObscuringWindowSize: xcb_query_tree failed for window 0x400000.
[Gamescope WSI] getToplevelWindow: xcb_query_tree failed for window 0x400000.
[Gamescope WSI] canBypassXWayland: failed to get window info for window 0x400000.
[Gamescope WSI] Creating swapchain for xid: 0x400000 - minImageCount: 4 - format: VK_FORMAT_B8G8R8A8_UNORM - colorspace: VK_COLOR_SPACE_SRGB_NONLINEAR_KHR - flip: false
[Gamescope WSI] Refusing to make swapchain (unsupported VkFormat) for xid: 0x400000 - format: VK_FORMAT_B8G8R8A8_UNORM - colorspace: VK_COLOR_SPACE_SRGB_NONLINEAR_KHR - flip: false

[simifor]

The issue is fixed on my side with 3.13.8