Improve Linux Wayland screen share dialog

[vchernin]

Steps to reproduce

1. Screen share on Linux Wayland.
2. If you're on 1.8.1 or earlier, ensure you're passing --enable-features=WebRTCPipeWireCapturer at runtime (this is fixed in develop).

What happened?

The dialog is confusing and hard to understand.The dialog attempts to show a preview of your window(s) but there isn't any.

To actually screen share you need to click the "Entire Screen" picture placeholder, and then click share. Both these steps add nothing useful to the experience when on Linux Wayland. The same is true if you try to share the "Application window" instead.

This issue is mainly because Wayland's method for screen sharing is very different from how X11 does it. Applications cannot assume they have access to view the screen, they must specifically request it from the host system. Element needs to be aware of that difference and not show a redundant dialog.

This issue similarly affects another Electron app, Jitsi Meet.

What did you expect?

So the whole dialog "Share Content" is probably best skipped for Wayland, there's no need to have 2 share buttons. When users click share they should be sent right to the PipeWire/Portal dialog.

So to recap:
Current steps:

1. Click circular share button in video meeting box.
2. Click image with no actual picture.
3. Click rectangular share button.
4. Let host use PipeWire/Portal dialog to actually share.

Proposed steps for Linux Wayland users:

1. Click circular share button in video meeting box.
2. Let host use PipeWire/Portal dialog to actually share.

Operating system

Linux

Application version

Element version: 1.8.1 Olm version: 3.2.3

How did you install the app?

Flatpak, identically affects DEB

Have you submitted a rageshake?

No

[SimonBrandner]

So I don't how would we determine whether we're running Wayland... Do you perhaps know of an Electron app that already does this?

[vchernin]

Well the output of echo $XDG_SESSION_TYPE will output wayland if on Wayland.

But no I don't know of any Electron apps that change behaviour to remove the redundant dialog if on Wayland. For further context, Jitsi Meet electron and Slack's electron app are 2 other Electron apps that can functionally use PipeWire (that I know of). I recall Slack having this same issue as here when I was able to try it (requires paid plan or trial).

Discord, Skype, Teams and Signal are all broken for various reasons (usually very outdated Electron version or non-standard WebRTC implementation from my experience).

I feel like Electron itself could be smarter here, so maybe that should investigated. An Electron fix might be a better solution so it can be solved for Jitsi and Slack (and others) in one go.

I'll see if I can find more info, but if not I might ping some experts, notably Jan Grulich who AFAIK wrote a lot or most of the PipeWire WebRTC implementation. There's an annoying issue in Chromium with dual PipeWire pop-ups, but that doesn't really seem to exist for Electron.

[vchernin]

I poked around in the WebRTC repo a bit, maybe this is useful?

It seems they're using XDG_SESSION_TYPE as part of their desktop_capturer code itself. How feasible is it for Element to just not show its dialog in some cases based on a Wayland check?

I'm not exactly sure how this could be done in Electron itself though without modifying each app. Neither am I sure what changes Electron makes to the WebRTC capturer code (if any). Maybe we need an Electron patch for 2 "backends" for when screen sharing is called: one for the "standard" use case (i.e. what most Electron apps assume they should do with X11), and another for the "wayland" use case where any custom app dialog should be skipped.

I am really not familar enough with any of the relevant codebases to help much more than giving random conceptual ideas though...

[SimonBrandner]

The current way Electron apps do screen-sharing is that they ask for a list of available sources and create their own dialog to choose one. This is completely different from how Wayland does this (AFAIK) which makes me quite unsure as to how this would be fixed in Electron itself. What I would hope for is at least a helper function to tell you whether we're on Wayland but I guess we can do that ourselves somehow

[SimonBrandner]

Chromium doesn't let you use your own dialog, so things are quite straightforward there I assume

[vchernin]

The current way Electron apps do screen-sharing is that they ask for a list of available sources and create their own dialog to choose one. This is completely different from how Wayland does this (AFAIK)

Yeah, I think this is correct, under Wayland the app indeed won't have a list of available sources. I think that needs confirmation though.

[vchernin]

@grulja sorry for the direct ping, but what do you know about app-created dialogs with PipeWire in Electron?

There is a seemingly universal issue for Electron sharing with PipeWire, where Electron apps continue trying to use the "X11 way" of screen sharing, meaning they're under the assumption they should be able to see open windows and displays. The user is faced with a now redundant app-created dialog that adds confusion and extra clicks.

See the screenshot above and here: jitsi/jitsi-meet-electron#608.

In my mind, when a user clicks the share button in an app like Element, they should be taken right to the PipeWire/XDG Desktop Portal dialog.

Would it make sense for Electron itself to be patched so apps are prevented from creating their own dialog when under a Wayland session?

[SimonBrandner]

Would it make sense for Electron itself to be patched so apps are prevented from creating their own dialog when under a Wayland session?

As far as I can tell there isn't much Electron can do besides the helper function because the way things work at the moment is that there is a method to get the sources and a method to request a media stream from one of them. I can't really see what Electron could do differently here 🤷‍♂️

[grulja]

@grulja sorry for the direct ping, but what do you know about app-created dialogs with PipeWire in Electron?

There is a seemingly universal issue for Electron sharing with PipeWire, where Electron apps continue trying to use the "X11 way" of screen sharing, meaning they're under the assumption they should be able to see open windows and displays. The user is faced with a now redundant app-created dialog that adds confusion and extra clicks.

This is basically same problem we have with Chromium. Everytime a user wants to share a screen he has to go through the dialog presented by Chromium and there is no way to skip it even though it is redundant as they want to keep same UI/UX experience across all platforms. With Chromium you even get the portal dialog twice as one is used for preview in the Chromium dialog and the second one is for the actual content on the web page itself.

See the screenshot above and here: jitsi/jitsi-meet-electron#608.

In my mind, when a user clicks the share button in an app like Element, they should be taken right to the PipeWire/XDG Desktop Portal dialog.

Would it make sense for Electron itself to be patched so apps are prevented from creating their own dialog when under a Wayland session?

Yes, that would be the best solution, however, I have no idea how this all is implemented in Electron and if it's something easy to do and if it's something upstream would agree with. I tried to propose same change for Chromium and that change was rejected.

[vchernin]

Yes, that would be the best solution, however, I have no idea how this all is implemented in Electron and if it's something easy to do and if it's something upstream would agree with. I tried to propose same change for Chromium and that change was rejected.

I have reported this to Electron: electron/electron#30652
Hopefully I did a good job explaining it, if there is anything to improve I can edit it.

I really hope they accept skipping the app-created dialog at an Electron-level, that is the best solution and maybe it could even help convince Chromium to do the same for their dialog. We'll see I guess.

For Element issue triagers: this is considered an Electron issue unless the Electron maintainers think otherwise. But until a fix is done this issue will affect Element so I think this bug should be left open.

[vchernin]

As far as I can tell there isn't much Electron can do besides the helper function because the way things work at the moment is that there is a method to get the sources and a method to request a media stream from one of them. I can't really see what Electron could do differently here

So... how about patching Electron's get sources method itself? So if an app calls get sources, but a Wayland session is detected, to just immediately request a media stream on behalf of the app?

From what I can tell the method is desktopCapturer.getSources, I found some docs. I am not really familar with Electron's codebase but maybe here or here is what would need to be patched?

It feels hacky but surely there's a sane way that idea could be implemented... as otherwise the solution is to get every app to change their implementation themselves, and that will take a long time if Wayland support in Electron is any precedent.

[SimonBrandner]

So... how about patching Electron's get sources method itself? So if an app calls get sources, but a Wayland session is detected, to just immediately request a media stream on behalf of the app?

I am not sure how would that look/solve the problem..

[vchernin]

Hm.. could you point me to where in Element it tries to share and create the dialog?

All I could find in the desktop and web repos was this.

If we know what Element tries to call when screen sharing, it might become clearer on how Electron could "trick" Element to not show the Element-created dialog.

[SimonBrandner]

@vchernin, may I suggest we move to a DM (@simon.brandner:envs.net) or to the #element-dev room?

[novocaine]

Related: #728

[Thatoo]

Whereas sharing screen works perfectly well with element-web on Wayland, it doesn't work at all on element-desktop on Wayland.
Wayland becoming by default on all major Linux OS distribution, it's becoming quiet an issue.

[yajo]

I've recently uninstalled element desktop and installed elements web as a pwa. It works much better.