Does not create window (on X11)

[CyberShadow]

This is on a Lenovo T580 with an NVIDIA MX150 running Linux 6.0 / nvidia 525.60.11 / Xorg 21.1.4 / mesa 22.2.3. Trying to start anything does not create any windows. glxgears example:

$ gamescope -- glxgears
No CAP_SYS_NICE, falling back to regular-priority compute and threads.
Performance will be affected.
wlserver: [backend/headless/backend.c:82] Creating headless backend
vulkan: selecting physical device 'NVIDIA GeForce MX150': queue family 2
vulkan: physical device supports DRM format modifiers
vulkan: vkGetPhysicalDeviceFormatProperties2 returned zero modifiers for DRM format 0x3231564E (VkResult: 0)
vulkan: supported DRM formats for sampling usage:
vulkan:   0x34325241
vulkan:   0x34325258
wlserver: Running compositor on wayland display 'gamescope-0'
wlserver: [backend/headless/backend.c:18] Starting headless backend
wlserver: [xwayland/server.c:92] Starting Xwayland on :1
wlserver: [types/wlr_surface.c:748] New wlr_surface 0x555eb97405c0 (res 0x555eb9742000)
wlserver: [xwayland/server.c:250] Xserver is ready
pipewire: stream state changed: connecting
pipewire: stream state changed: paused
pipewire: stream available on node ID: 83
pipewire: renegotiating stream params (size: 1280x720)
src/nv_gbm.c:100: GBM-DRV error (nv_gbm_bo_create): DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY failed (ret=-1)

wlserver: [types/wlr_surface.c:748] New wlr_surface 0x555eb93513d0 (res 0x555eb9749d00)
src/nv_gbm.c:100: GBM-DRV error (nv_gbm_bo_create): DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY failed (ret=-1)

Error getting buffer
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
9609 frames in 5.0 seconds = 1921.797 FPS
10329 frames in 5.0 seconds = 2065.762 FPS
[...]
^Cgamescope: received kill signal, terminating!
xwm: X11 I/O error
[1]    10740 segmentation fault (core dumped)  gamescope -- glxgears
gamescope -- glxgears  186.90s user 52.69s system 208% cpu 1:54.70 total

Nothing on the web about DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY.

[Samsagax]

I think this is a duplicate of #498

[CyberShadow]

It's possible but the failure mode seems to be different (see error message), and also, the workaround set of environment variables discovered in #498 does not change anything in this case.

Also it's not clear to me in #498 if Xorg is running on NVIDIA or Intel, in my case I'm trying to start it from Xorg already running on NVIDIA (by using a xorg.conf with appropriate Device / Screen sections).

xorg.conf

Section "Files"
  ModulePath "/usr/lib/nvidia"
  ModulePath "/usr/lib32/nvidia"
  ModulePath "/usr/lib32/nvidia/xorg/modules"
  ModulePath "/usr/lib32/xorg/modules"
  ModulePath "/usr/lib64/nvidia/xorg/modules"
  ModulePath "/usr/lib64/nvidia/xorg"
  ModulePath "/usr/lib64/xorg/modules"
EndSection

Section "ServerLayout"
  Identifier "layout"
  Screen 0 "nvidia"
  Inactive "intel"
EndSection

Section "Device"
  Identifier "nvidia"
  Driver "nvidia"
  BusID "PCI:2:0:0"
EndSection

Section "Screen"
  Identifier "nvidia"
  Device "nvidia"
#  Option "AllowEmptyInitialConfiguration" "Yes"
#  Option "UseDisplayDevice" "none"
EndSection

Section "Device"
  Identifier "intel"
  Driver "modesetting"
  Option "AccelMethod" "none"
EndSection

Section "Screen"
  Identifier "intel"  
  Device "intel"
EndSection

[Samsagax]

Have you tried this?
$ DRI_PRIME=1 MESA_VK_DEVICE_SELECT=10de:1347 __NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia gamescope -- glxgears

Change the MESA_VK_DEVICE_SELECT variable to your Nvidia card.

[CyberShadow]

Yes, that is what I tried. It fails in exactly the same way.

Log with workaround from #498

$ lspci -nn | grep -i nvidia       
02:00.0 3D controller [0302]: NVIDIA Corporation GP108M [GeForce MX150] [10de:1d10] (rev a1)

$ DRI_PRIME=1  MESA_VK_DEVICE_SELECT=10de:1d10 __NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia gamescope -- glxgears 
No CAP_SYS_NICE, falling back to regular-priority compute and threads.
Performance will be affected.
wlserver: [backend/headless/backend.c:82] Creating headless backend
vulkan: selecting physical device 'NVIDIA GeForce MX150': queue family 2
vulkan: physical device supports DRM format modifiers
vulkan: vkGetPhysicalDeviceFormatProperties2 returned zero modifiers for DRM format 0x3231564E (VkResult: 0)
vulkan: supported DRM formats for sampling usage:
vulkan:   0x34325241
vulkan:   0x34325258
wlserver: Running compositor on wayland display 'gamescope-0'
wlserver: [backend/headless/backend.c:18] Starting headless backend
wlserver: [xwayland/server.c:92] Starting Xwayland on :1
wlserver: [types/wlr_surface.c:748] New wlr_surface 0x556f6e522e30 (res 0x556f6e524870)
wlserver: [xwayland/server.c:250] Xserver is ready
pipewire: stream state changed: connecting
pipewire: stream state changed: paused
pipewire: stream available on node ID: 88
pipewire: renegotiating stream params (size: 1280x720)
src/nv_gbm.c:100: GBM-DRV error (nv_gbm_bo_create): DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY failed (ret=-1)

wlserver: [types/wlr_surface.c:748] New wlr_surface 0x556f6e13c9b0 (res 0x556f6e52c570)
src/nv_gbm.c:100: GBM-DRV error (nv_gbm_bo_create): DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY failed (ret=-1)

Error getting buffer
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
9289 frames in 5.0 seconds = 1857.637 FPS
^Cgamescope: received kill signal, terminating!
xwm: X11 I/O error
[1]    86886 segmentation fault (core dumped)  DRI_PRIME=1 MESA_VK_DEVICE_SELECT=10de:1d10 __NV_PRIME_RENDER_OFFLOAD=1 = =  
DRI_PRIME=1 MESA_VK_DEVICE_SELECT=10de:1d10 __NV_PRIME_RENDER_OFFLOAD=1 = =    11.55s user 2.98s system 195% cpu 7.421 total

[CyberShadow]

DRM_IOCTL_NVIDIA_GEM_ALLOC_NVKMS_MEMORY goes away when I add the nvidia-drm.modeset=1 kernel parameter. Now it fails in the same spot but in a different way; I'll open a new issue.

Edit: Opened #708.