screencast: let the user select the output

[emersion]

Right now we have a CLI option, which is less than ideal. Replace this with a basic UI, probably slurp-based.

xdg-output can be used to obtain output geometry in logical coordinates. A list of rectangles can be fed to slurp on stdin.

[tmccombs]

Replace this with a basic UI, probably slurp-based.

It would be nice to have something like this for screenshots as well. I also think it would be good to have a way to select whether to use a rectangle, a window, or the full output (or in the case of screenshot, the full screen).

[jvanbruegge]

I think would also be interesting to investigate how to offer firefox (and other webrtc clients) a list of screens to share. Usually you can select the screen in the browser itself

[danshick]

@jvanbruegge I don't think this is possible without a change to the WebRTC codebase. In fact, the whole reason that chromium still hides pipewire behind a build and a feature flag is the "dialog spam" of having to do output selection once in the xdp implementation and a second time in the browser. jgrulich, I believe, has been working on reducing the dialog complexity, but I don't know if those changes landed.

From an old chat with jgrulich on #pipewire:

originally, I had a patch that it didn't show the chromium dialog, it immediately showed the portal dialog and only if you cancelled it, then the Chrome dialog appears with option to share a tab, but Chromium developers didn't like it and they wanted to keep the Chromium dialog as it is

on Chromium side, I merged "Screen" and "Window" tabs into one, because we don't know what the user will choose and it doesn't make sense to create two capturer classes, then once you pick something in the portal dialog, it will show preview in the Chromium dialog and user just needs to confirm it, then you shouldn't get the portal dialog again, but you should see the screen cast content directly on the web page

Old (on KDE, I think):

https://jgrulich.fedorapeople.org/chromium-dialog-hell.webm

Fixed:

https://jgrulich.fedorapeople.org/chromium-sharing-screen.webm

Ironically, we don't experience the double request issue, because we don't allow for output selection at all at the time of invocation, but this will be an issue when we do, until this patch lands.

[danshick]

The big difference being that WebRTC itself used to enumerate all screens and windows, and now xdp does it. There's no mechanism in the d-bus api to pass that info to the consumer. The more I think about it, it would not even just be a patch to WebRTC, but to the definition of the xdp screencast portal too.

What I'm thinking for a final selector is a fullscreen overlay similar to slurp, but one which only lets you select entire outputs. I think it should be both keyboard and mouse activated. This can be done with the layer shell protocol, but I haven't gotten around to writing anything just yet.

That said, I'm open to suggestion and PRs.

[tmccombs]

a fullscreen overlay similar to slurp, but one which only lets you select entire outputs

so it wouldn't be possible to capture just part of the screen?

[emersion]

a fullscreen overlay similar to slurp, but one which only lets you select entire outputs

slurp allows you to provide a set of rectangles on stdin that the user can choose from.

[danshick]

@emersion The problem with slurp is that it only outputs coordinates, and for screencasting output selection, we need the output name or ID. Otherwise I would have just done something similar to this and called it a day (but with sway-ipc).

swaymsg -t get_outputs | jq -r '.[] | select(.active) | .rect | "\(.x),\(.y) \(.width)x\(.height)"' | slurp

I suppose we could get both outputs and their coordinates, invoke slurp, get coordinates back, and map it back to outputs. But, I think it'd be nice to also include a little overlay that explains what's going on, how to select an output, etc... A user invoking slurp has a better grip on the causal relationship of what they're seeing than someone who clicked "share my screen" in the browser. The crosshair cursor and greyed out selection could be a bit confusing without the context.

[danshick]

@tmccombs
Screen casting will (likely) never handle anything other than entire outputs. There's no mechanism in wlroots for getting only a single application's buffer.

Screenshots, OTOH, could totally have region selection, as exemplified by slurp + grim. The question is, should invoking a screenshot portal immediately invoke slurp by default? Should a user-selected region be the typical behavior? I'd be curious to see how this works in other portal implementations (gtk, kde). My guess is that the portal should just provide a full screen capture, and the consuming application should do the cropping as needed.

[tmccombs]

Screen casting will (likely) never handle anything other than entire outputs.There's no mechanism in wlroots for getting only a single application's buffer.

For single applications buffers, yes. although this is quite unfortunate and a regression from X. However, wlroots does appear to support capturing a region with the "capture_output_region" request in the wlr-screencopy-unstable-v1 protocol. With that it would be possible to use sway ipc to get the region for a particular window, and then capture that region. Although, handling the case of the window moving during a recording would be tricky, since you would need to watch for movement of the window.

I'd be curious to see how this works in other portal implementations (gtk, kde)

I believe gnome opens a screenshot dialog, although I don't remember exactly what the options are in the dialog.

[danshick]

Yeah, the window movement issue is the blocker for this for sure. It also isn't possible at all for dmabuf based casting, which we would like to implement eventually.

[vesim987]

I did a very dirty hack to test that: vesim987@45d5b42 but it is not working too well, especially on chromium. It is executing the setup_outputs function three times, so slurp is executed multiple times too. I don't think that there is a good approach for that w/o fixing the WebRTC spec for allowing output selection.

[danshick]

There is already some work being done on the "dialog hell" problem as mentioned here[0]. There are a set of patches that seek to improve the situation [1][2] which are currently under review, (and, as a side effect, also improves the situation for #34 by supporting alpha channel pixel formats).

The bigger problem with the idea behind your "dirty hack" is its reliance on swaymsg, which makes this patch sway-specific. We want xdpw to work for all wlroots based compositors. Using the data from the output protocol keeps things agnostic.

[0] https://bugs.chromium.org/p/chromium/issues/detail?id=682122#c58
[1] https://webrtc-review.googlesource.com/c/src/+/160649
[2] https://chromium-review.googlesource.com/c/chromium/src/+/1942327

[vesim987]

@danshick

The bigger problem with the idea behind your "dirty hack" is its reliance on swaymsg, which makes this patch sway-specific. We want xdpw to work for all wlroots based compositors. Using the data from the output protocol keeps things agnostic.

I know, doing that with only slurp is easier, there is even a list of the outputs(ctx->output_list) available, but I didn't know about the fixes to WebRTC and I wanted to make something fast to test this. I will try to create some proper patch around the weekend.