Using GPU Screen Recorder UI on niri

[bczhc]

tl;dr: All is not needed anymore. GSR-UI has added native wayland support using wlr-layer-shell's overlay layer now.

---

---

---

Old attempts trying to make gsr-ui work on niri:

gpu-screen-recorder-ui (referred to as gsr-ui below), a ShadowPlay like overlay for controlling gpu-screen-recorder. Note the latter one is a cli and is the core part of the screen recorder, and supports both X and Wayland. However the overlay app, is an X app and only supposed/guaranteed to work well on X. And indeed the fact - gsr-ui has several issues on niri.

For example gsr-ui will starts as a normal toplevel window on niri, and as a standalone column. Though gsr-ui has some transparency, by the nature of tiling compositors, in this case gsr-ui's background will be black or the background layer (aka. the wallpaper). To make it show the content behind it, gsr-ui should be floating. However the open-fullscreen false window rule doesn't work for this because somehow gsr-ui spawns its window first and then later, does a postponed fullscreen request. See issue: #3106.

Considering gsr-ui is still an X application, I'm able to tweak xwayland-satellite to make gsr-ui work better. Thanks there is an amazing tool: wl-proxy from @mahkoh, xwayland-satellite can be started as a wayland overlay layer.

Just starting a separate xwayland server (as overlay layer via wl-paper) and only dedicated to gsr-ui:

#!/bin/bash

wl-paper --layer overlay --margin-top -28 --keyboard-interactivity none xwayland-satellite :1 &
sleep 1
DISPLAY=:1 gsr-ui

..will solve the issue, and it makes gsr-ui work perfectly, almost.

Screenshot

---

Remaining issues are:

1. When the cursor is locked (e.g. playing a 3d game), starting gsr-ui will not warp the cursor to screen center, which I believe this is required for a better UX. However I think it's not that possible to be implemented in gsr-ui because gsr-ui is an X app, unless it can query something like, hey niri if the cursor is currently locked/grabbed?
2. If clicking Record - Start to start the recording, the gsr notification is displayed normally (the top-right corner, screenshot). However, if starting the recording via gsr-ui-cli toggle-record, the notification is at the top-left. Screenshot.
3. GSR notification ramdomly won't show on the recording starts when playing a fullscreen game. Don't know why and I haven't found a stable repro approach.

---

Other notes

In the wl-paper above, --margin-top -28 is to make the overlay occupy the whole screen otherwise it will only occupy the space below the bar (waybar, DMS bar etc.). Tweak -28 to find the proper value, or apply this patch to make wl-paper always occupy the whole screen:

diff --git a/apps/wl-paper/src/paper.rs b/apps/wl-paper/src/paper.rs
index a43ebaa..d5d6515 100644
--- a/apps/wl-paper/src/paper.rs
+++ b/apps/wl-paper/src/paper.rs
@@ -382,6 +382,7 @@ impl XdgSurfaceHandler for ClientXdgSurface {
             c.margin_bottom,
             c.margin_left,
         );
+        zwlr_layer_surface_v1.send_set_exclusive_zone(-1);
         zwlr_layer_surface_v1.send_set_keyboard_interactivity(c.keyboard_interactivity);
         zwlr_layer_surface_v1.send_set_anchor(
             ZwlrLayerSurfaceV1Anchor::TOP

--keyboard-interactivity none option is to let the overlay completely ignore key events. Currently seems gsr-ui has no key interactions except Esc (to exit GSR). So I just ignore all keys here, otherwise it may take the focus, causing issue like, some video games (like Minecraft) will show the menu on focus lose.

I had detailed talks on this with dec05eba today.

Some quotes from dec05eba

The main issue with niri I believe is that xwayland-satellite doesn't seem to respect override redirect windows and several other x11 window properties,
or niri ignores them. Most wayland compositors do (including tiling wayland compositors), but niri is the exception.
The gsr ui is forced to run as an xwayland application because otherwise games can get minimized
when the fullscreen overlay opens, but in a tiling wm that may not be an issue (depending on the tiling wm).
So I could maybe do a niri specific solution. I still believe that it may not work properly as the wayland protocol states that a fullscreen opaque window
shouldn't show the windows behind it (wayland "security" feature), so the gsr ui can never really work properly on wayland, unless the wayland compositors ignores the wayland protocol.

All of those issues are because niri/xwayland-satellite doesn't properly support x11 applications (specifically override redirect windows).
It works fine in all x11 window managers and other wayland compositors, even tiling wayland compositors
and gnome that doesn't support overlays (wayland doesn't actually support overlays, wlroots does and some wayland compositors support that instead).
So the question is just what kind of niri specific hacks you can do to make it work when niri doesn't properly support x11.
It would be better if niri/xwayland-satellite properly supported x11 applications. An x11 override redirect window is supposed to make the wayland
compositor not tile the window, in the same way an overlay window works.

The author's basic idea is it's niri/xwayland-satellite's lack of implementation of specific X protocols (like override redirect windows). And if niri/xwayland-satellite implements it, it should behave the same as Wayland's overlay layer.

Unfortunately wayland is such as a mess that gsr ui cant guaranteed work properly, which is why I hate wayland.

And sadly they are not a fan of Wayland.

:(

[ypurpl]

thank you for telling me about wl-proxy. Now i can finally minimize thunderbird to tray

[bczhc]

2026-05-08 update

I've filed this issue: Supreeeme/xwayland-satellite#429 for xwayland-satellite. If X11 override_redirect were implemented correctly using Wayland layer-shell, gsr-ui should work.

And I actually had luck with it. If anyone's not minding it, commit bczhc/xwayland-satellite@bac9faf is the AI-made fix for Supreeeme/xwayland-satellite#429.

If looking at gsr-ui's code, gsr-ui just doesn't start an OR (override_redirect) window on niri, for some reason I don't know. So the next step is to patch gsr-ui:

diff --git a/src/Overlay.cpp b/src/Overlay.cpp
index 053020b..a542d96 100644
--- a/src/Overlay.cpp
+++ b/src/Overlay.cpp
@@ -1163,11 +1163,7 @@ namespace gsr {
-        const Window x11_focused_window = get_focused_window(display, WindowCaptureType::FOCUSED, false);
-        const bool prevent_game_minimizing = gsr_info.system_info.display_server != DisplayServer::WAYLAND
-            || (x11_focused_window && is_window_fullscreen_on_monitor(display, x11_focused_window, *focused_monitor))
-            || is_wlroots
-            || is_hyprland;
+        const bool prevent_game_minimizing = true;
 
         const bool drm_cursor_pos = (!prevent_game_minimizing || is_wlroots || is_hyprland) && cursor_info;
         if(drm_cursor_pos)

After this, gsr-ui will try to start an OR window instead of a normal window on niri, and xwayland-satellite with that issue fixed will use overlay layer-shell for any OR window. Thus we can get gsr-ui fully work very well on niri. (No 2. and 3. of the "remaining issues" above are solved).

For the "remaining issues" no. 1, dec05eba stated:

As for the mouse warp, I dont think wayland (wlroots) supports querying if the pointer is locked or not.

Maybe more hack is needed for this.

But anyway dec05eba said they now have started to play around with wlr-layer-shell a bit and are trying to make gsr-ui work properly on niri.

Tl; dr GSR-UI solution for niri as of 2026-05-08

1. Patch xwayland-satellite and build

   git clone https://github.com/Supreeeme/xwayland-satellite
   cd xwayland-satellite
   # DISCLAIMER below: this patch is LLM-generated
   wget https://github.com/bczhc/xwayland-satellite/commit/bac9faf4e1a318c231f0757d9ac0fe0afd08cb2a.patch -O 0001.patch
   git am 0001.patch
   cargo build -r
   
   # Now replace the binary niri uses, for example do something like below,
   sudo rm /bin/xwayland-satellite; sudo cp target/release/xwayland-satellite /bin
   # Kill the running xwayland-satellite or restart niri to let it use the new one
   pkill xwayland

2. Patch gsr-ui

   # Clone gsr-ui
   git clone https://git.dec05eba.com/gpu-screen-recorder-ui
   cd gpu-screen-recorder-ui
   
   # (omitted - do the diff change mentioned above)
   
   # Build & install
   meson build && ninja -C build && sudo ninja -C build install

3. Add niri keybinds (one example):

   // GPU Screen Recorder hotkey
   // Alt+Z { spawn-sh "gsr-ui-cli toggle-show"; } (if gsr-ui's hotkey is disabled, use this)
   Mod+F10 { spawn-sh "gsr-ui-cli toggle-record"; }
   Mod+F9 { spawn-sh "gsr-ui-cli toggle-pause"; }    

---

2026-05-08 0358Z update again:

Oops I just noticed gsr-ui has this commit just pushed: https://git.dec05eba.com/gpu-screen-recorder-ui/commit/?id=353df088de909cc7926467623f5516a46d4d82d3

So, the gsr-ui patch mentioned above is not required anymore if you're using this. But XDG_CURRENT_DESKTOP is required to be exactly niri but not something like niri:GNOME because strcmp is used there.

2026-05-11 update:

dec05eba added native Wayland support for gsr-ui and gsr-notification, using wlr-layer-shell's overlay layer. This is a good new. Xwayland-satellite and all the hacky workarounds above are not needed anymore. After tons of email discussions and tests+bug reports gsr-ui finally did it, and it runs perfectly on niri now!

One last thing I want gsr-ui to do is warp the pointer to center+use "none" keyboard interactivity if the pointer is locked. I've added an ipc to niri to query if the pointer is currently locked and later i'll hack gsr-ui to achieve this.

2026-05-11 1313Z update:

One last thing I want gsr-ui to do is warp the pointer to center+use "none" keyboard interactivity if the pointer is locked. I've added an ipc to niri to query if the pointer is currently locked and later i'll hack gsr-ui to achieve this.

If anyone wants this also, there are the patches:

niri:
0001-ipc-Add-has-pointer-constraints.patch
0002-ipc-Add-niri-msg-action-set-pointer.patch

gsr-ui:
0001-Set-overlay-keyboard-interactivity-none-when-the-cur.patch

and use an extra new script to do gsr-ui toggle-show:

#!/bin/bash

locked="$(niri msg -j has-pointer-constraints | jq '.has_pointer_constraints')"
[ "$locked" = "true" ] && niri msg action set-pointer $((1920/2)) $((1080/2))
gsr-ui-cli toggle-show