PipeWire captures cursor incorrectly on Linux Wayland

[vchernin]

Operating System Info

Other

Other OS

Fedora 34 Silverblue

OBS Studio Version

27.0.0-rc6

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/88w7RayumWBJAQKA

OBS Studio Crash Log URL

No response

Expected Behavior

I expect the cursor to be positioned and sized correctly. The video OBS produces should ideally be 100% representative of what the display originally showed.

Current Behavior

PipeWire Window Capture:

The cursor is far from where it should be, and is too large.

PipeWire Display Capture:

Observer how the cursor moves slightly, relative to the red asterisk. The cursor is seemingly the correct size but is slightly off the correct position.

Steps to Reproduce

1. Attempt to screen share with either PipeWire capturer on Linux Wayland (OBS > 27).
2. Supported distros included any recent Fedora, Ubuntu 21.04+ or other distros with PipeWire 0.3 (I believe)
3. Observe cursor position and size in OBS Studio's preview.

Anything else we should know?

This issue occurs most noticeably with the PipeWire Window Capturer, but also occurs (albeit subtly) with the PipeWire Display capturer. The pictures above show this.

Note I am using 2 HIDPI displays (4k and 1440p 16:9, both set to 200% GNOME scaling, no fractional scaling). Because my setup is somewhat of an edge case there's a possibility this is an upstream PipeWire/GNOME/Wayland bug.

[GeorgesStavracas]

This is almost certainly a bug in Mutter / GNOME Shell, but just to make sure, would you please run this script [xdp-screen-cast-embedded-monitor.zip] and see if it works correctly? Unpack it, and run the xdg-screen-cast.py file. (Alternatively, you can try the PipeWire-based capture on another environment like KDE or any wlroots-based, and see if the issue is also hapenning there)

[vchernin]

@GeorgesStavracas So I can't reproduce the issue in OBS with your script. In fact screen sharing in that script seems to work nearly perfectly. Sometimes the share is a bit laggy, and sometimes with window sharing the aspect ratio is wrong (the output gets squished) but the mouse is always perfect.

Since it may actually be an OBS-specific issue I will try it out in Fedora 34 KDE wayland. Do you know if PipeWire and OBS flatpak will work in a distro running in live USB mode?

[GeorgesStavracas]

Right. Thanks for testing it. That means Mutter / GNOME Shell is indeed sending the wrong cursor information through PipeWire. OBS Studio merely renders what it receives, and since it's receiving slightly wrong information, it ends up with a slightly wrong result.

[carlocastoldi]

@GeorgesStavracas for me, on GNOME Wayland / Mutter, the pointer in the screen recording lags a lot if move the mouse very slowly. It seems like it start moving in the right direction and then goes back to the original position

It's not happening in OBS. Don't worry

[vchernin]

On KDE with Fedora 34, OBS via RPM Fusion doesn't have the mouse displacement issue. Rather the mouse doesn't show with window sharing. Also with display sharing the cursor is too small.

I made a table summarizing what I've noticed.

Cursor issues in OBS with Wayland PipeWire screen sharing:

	GNOME	KDE	wlroots
Window sharing	Cursor too large (300% scale?) Cursor very far from correct location	No cursor visible	?
Display sharing	Cursor slightly off position	Cursor too small (100% scale?)	?
OBS Package tested	Flatpak 27rc6	RPM Fusion 27rc5 (live USB)	?

OBS is running on a exclusively HIDPI (200% scaling) setup on Fedora 34.

[GeorgesStavracas]

Amazing. My exceptional goldfish memory forgot I was actually already reviewing a merge request against Mutter fixing this:

https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1866

I know this might be quite a stretch, but if you could test that merge request and let us know if it fixes the "GNOME" column of the table above, it would be fantastic. Otherwise, I'll eventually have to test it all and see if it's working, but it might take a few weeks 🙁

[vchernin]

@GeorgesStavracas So I tried to run Mutter with that patch and it doesn't appear to have done anything. Every issue in the GNOME column still occurs.

Since I'm on Fedora 34 Silverblue, I figured I should take the Mutter RPM, put the patch in the .spec, make sure all the patches were in the right directory and then build it. I installed the new RPM with rpm-ostree override replace, and then rebooted.

Is there anything else I could do to help test that patch out?

[GeorgesStavracas]

Thanks for testing it - that's really appreciated, specially when it's a non-trivial task. I'll try and reproduce this issue, and if I manage to, see what's going on. I still suspect the merge request I mentioned fixes it, but let's see.

[vchernin]

@GeorgesStavracas
Some good news, it appears I was actually not testing it properly. Now that I have, here is the updated table:

Visible preview issues in OBS with Cursor using Wayland PipeWire screen sharing with !1866.

	GNOME	KDE	wlroots
Window sharing	Cursor slightly off position Cursor "shakes" when moving window	No cursor visible	?
Display sharing	Cursor slightly off position	Cursor too small (100% scale?)	?
OBS Package tested	Flatpak 27.0	RPM Fusion 27rc5 (live USB)	?
!1866: latest commit	fa884b19	N/A	N/A

The cursor is no longer way too large and the cursor is relatively close to where it should be (to the point where only the keen-eyed will notice). Thank you again for helping find that patch. I might test it a bit more and then report my findings to the mutter discussion upstream.

If you're wondering how I managed to make this mistake, it appears that naming the patched RPM the same as the distro-provided mutter caused a problem. Simply giving my RPM a slightly different name seems to have gotten everything to work properly. I might report a bug to rpm-ostree as it should probably warn you if you try to do what I did...
Edit: I made an rpm-ostree issue, it really isn't related to OBS, but hopefully it will help someone in the future.

[RytoEX]

@GeorgesStavracas Does this mean we're waiting for the mutter merge request to be merged and distributed?

[GeorgesStavracas]

@RytoEX yes. Feel free to close this, there doesn't seem to have anything else to do on OBS side.

[vchernin]

Closing. Upstream patches like !1866 should mostly solve the issue for Mutter (GNOME). Feel free to discuss more upstream as that’s the ideal place to fix cursor issues.

[GeorgesStavracas]

Part of this issue was actually fixed by #4936