xdg-desktop-portal helper implementations

[orowith2os]

Wayland doesn't allow arbitrary clients to manipulate the compositor, and it also offloads the task of implementing sensitive tasks like screen and input capturing to the compositor in xdg-desktop-portal.

Smithay should ideally provide some tools that help to implement the following portals:

Global keybinds should be able to have the UI handled by the compositor, but it's not too bad to have them handled by each compositor.

ScreenCast could definitely use some assistance here, and the UI for it is handled by the compositor; I think Smithay could just provide some PipeWire details that the compositor forwards to the requesting application.

RemoteDesktop and InputCapture might need some implementation in Smithay too, like libei support.

Screenshot should be fine, assuming Smithay supports the wlroots protocols used for the wlroots portal implementation?

[PolyMeilex]

Well it depends on the compositor, one could just use wlroots portal, its just a matter of having all protocols it uses supported in the compositor.

One could also write compositor specific portal, like cosmic does.

[Drakulix]

There is no issue about this topic, because smithay is a compositor framework, not a compositor (anvil is not meant to be daily driven) and for sure not a desktop environment.
And a xdg-desktop-portal implementation falls into the responsibility of the latter.

We are open to support more wlr-protocols to make using the xdg-desktop-portal-wlr easier however. However supporting that or our own is already absolutely possible with a compositor built upon smithay.

[orowith2os]

I believe using the wlroots portal would involve having all the same limitations, including only being able to screen share whole screens.

Smithay would ideally include a portal implementation that most compositors using it can benefit from, and would probably help to not duplicate work.

For things like filepickers (the FileChooser portal), though, those are best managed by the compositors themselves.

[Drakulix]

Screensharing in particular is a privacy concern, that needs to be handled by individual compositors, which makes abstracting quite hard.
However if that is your goal (specifically supporting sharing things other than outputs), there is the new ext-screencopy protocol, which can handle arbitrary sources and needs implementations. Some abstractions for it in smithay are definitely in scope.

[ids1024]

https://flatpak.github.io/xdg-desktop-portal/#gdbus-org.freedesktop.portal.ScreenCast lets the application express what it wants to record with a bitmask of MONITOR/WINDOW/VIRTUAL, a multiple boolean, and a cursor_mode setting. But doesn't seem to have any way to express what particular monitor or window to capture. So if I understand correctly a full-featured implementation should provide a UI to select what to capture, which seems generally out of scope for anything Smithay itself would provide.

It would be good to have a PR with an implementation of ext-screencopy, though I guess that wouldn't be merged until the protocol is accepted upstream and it's not clear that will happen soon or without significant further changes.

[PolyMeilex]

This actually reminds me of an idea that I had at some point of a crate that would simplify the process of writing a portal, aka. abstract ugly details and let the consumer implement anything that might be compositor specific.

But that's not strictly smithay related, although could live under the umbrella.

Also haven't done proper assessment of what can actually be abstracted, and if it actually makes sense.

[ids1024]

I think that's basically the goal of bilelmoussaoui/ashpd#108.

For https://github.com/pop-os/xdg-desktop-portal-cosmic, most portal functionality should be handled alright by the generic xdg-desktop-portal-gtk, and the main thing I've been wanting to improve is the APIs pipewire-rs provides for dealing with streams. The dbus part is much simpler than the pipewire part, for the ScreenCast portal.

[orowith2os]

Update, should be easy for what should be handled by Smithay.

Global keybinds might be able to use some assistance from Smithay and finishing the implementation on the compositor.

ScreenCast could definitely use some assistance here, and the UI for it is handled by the compositor; I think Smithay could just provide some PipeWire details that the compositor forwards to the requesting application.

RemoteDesktop and InputCapture might need some implementation in Smithay too.

Screenshot should be fine, assuming Smithay supports the wlroots protocols used for the wlroots portal implementation?

[orowith2os]

Updated the main issue to contain the implementations in Smithay to help. I'm also more knowledgeable on the technical details, so I'm ready to go on this one now too.

[Drakulix]

Global keybinds might be able to use some assistance from Smithay and finishing the implementation on the compositor.

Keybindings are currently only implement in anvil and then handled by smithay's KeyboardHandle::input function, which allows filtering out key-events to be handled by the compositor.

Some functionality to make tracking global keybindings easier and generate the filter from would fit very well here.

ScreenCast could definitely use some assistance here, and the UI for it is handled by the compositor; I think Smithay could just provide some PipeWire details that the compositor forwards to the requesting application.

This needs two things:

• A protocol to facilitate getting the screen-contents for the portal. If someone wants to use xdg-desktop-portal-wlr, this has to be wlr-screencopy right now, but ext-screencopy would definitely be preferred by us in the long-term (once that lands).
• UI to select the source. This can easily be implemented inside the portal, given that smithay supports wlr-layer-shell. It could also be implemented inside a compositor, but integration of GUI toolkits is definitely not in scope of the smithay-crate. (We have toolkit support in external crates though, e.g. smithay-egui exists.)

So on the compositor side the big issue is some abstraction for a protocol to capture screen contents.

RemoteDesktop and InputCapture might need some implementation in Smithay too.

Those are definitely much harder. I think the best cause of action would be to first implement these directly in a compositor and then later evaluate which parts of such an implementation could fit into smithay. I don't think we want to support either of those in anvil.

Screenshot should be fine, assuming Smithay supports the wlroots protocols used for the wlroots portal implementation?

We don't, because that is essentially the same as screen-casting. This is handled by wlroots with the same protocol - wlr-screencopy.
Screen capture is just constant screen-shotting at the displays framerate. So this is essentially the same issue.

---

Additionally I would love to see a library to make it easier to write a xdg-desktop-portal implementation for smithay-based compositors.
As @ids1024 already pointed out, there is ashpd as a library for handling the dbus-protocol side of the portal. I would imagine an additional library providing the communication to the compositor, e.g. through the necessary wayland-protocols like ext-screencopy.

[orowith2os]

A protocol to facilitate getting the screen-contents for the portal.

Is this absolutely needed? Why can't Smithay provide some API that a compositor uses internally to get the window contents, at the very least until ext-screencopy is merged? The current APIs should handle this bit already too.

ext-screencopy might even be irrelevant here, windows just need to be sortable and Smithay should ideally provide some preimplemented PipeWire stuff that just gets tossed to the appropriate portal backend.

Those are definitely much harder. I think the best cause of action would be to first implement these directly in a compositor and then later evaluate which parts of such an implementation could fit into smithay.

libei should be done in Smithay, from there the portals themselves should probably be done on the compositor side of things to see what else should be simplified by Smithay.

[Drakulix]

Is this absolutely needed? Why can't Smithay provide some API that a compositor uses internally to get the window contents, at the very least until ext-screencopy is merged? The current APIs should handle this bit already too.

Smithay does provide an api to copy screen contents, but every xdg-desktop-portal is a separate executable. So there needs to be a way to communicate between the compositor based on smithay and the portal.
For wlroots-based compositors this is always wlr-screencopy, for gnome this is some internal dbus protocol (because their compositor does already speak dbus anyway) and kde has it's own private wayland protocol (if I remember correctly).

A compositor based on smithay could of course implement any form of communication their developers desire, but given every compositor already has a wayland socket, using something like wlr-screencopy or ext-screencopy is relatively easy to do.
The actual portal implementation using pipewire and dbus needs to be in a separate executable anyway, so why lift all that logic into smithay as well?

This is the reason most smithay-based compositors choose the implement either wlr-screencopy or a private protocol (usually derived from a recent ext-screencopy draft, so that switching over will hopefully be easy in the future). They need a stop-gap solution.

Edit: Note that you are free to just do all the pipewire stuff directly in your compositor and just proxy that through your portal-implementation. I just don't think we want to introduce to relatively large dependencies like pipewire and zbus in smithay itself.
But the APIs to get the screen-contents and everything else should be part of public api anyway and not require some access to any smithay internals (after all all these compositors figured out to use them to implement said protocols). If anything is missing, I am very happy to help make the API easier to work with for that use-case. It shouldn't assume any way of transferring the captured pixel data.

[Drakulix]

libei should be done in Smithay, from there the portals themselves should probably be done on the compositor side of things to see what else should be simplified by Smithay.

Yes, libei support is definitely of interest. But somebody has to write bindings for that first... (given enough time, this person will likely be me...)

[PolyMeilex]

Is this absolutely needed?

Technically no, kwin, mutter, weston all use pipewire for that, it's simply a question of: do you want to follow wlroots like philosophy of delegating everything into separate processes and making sure there is no dbus in the world.

Personally I wanted to use pipewire directly in my compositor (duh in my free time I am even working on RIIR of libpipewire compatible with calloop and with the same API as wayland-rs), and just expose nodes whenever portal needs them, but I'm no longer working on my compositor, so my stance is kinda void 😄.

EDIT: that being said, I'm still working on plug and play pipewire solution for smithay compositors, because I simply like the idea of exposing outputs or windows as video nodes that can be seen and wired through something like helvium.

[Drakulix]

Technically no, kwin, mutter, weston all use pipewire for that, it's simply a question of: do you want to follow wlroots like philosophy of delegating everything into separate processes and making sure there is no dbus in the world.

Some remarks and additions:

• I don't think xdg-desktop-portal-weston actually exists. But weston definitely has its own capture protocols, but those seem to return wl_buffers: https://gitlab.freedesktop.org/wayland/weston/-/blob/main/protocol/weston-output-capture.xml
• They do have a pipewire integration though, that might be for the build-in remote-desktop stuff though.
• kwin has a private wayland-protocol, but that returns pipewire nodes: https://wayland.app/protocols/kde-zkde-screencast-unstable-v1 This is probably closest to what @orowith2os was imaging?
• mutter has a dbus protocol to attach pipewire streams to screencast sessions: https://gitlab.gnome.org/GNOME/mutter/-/blob/gnome-40/src/org.gnome.Mutter.ScreenCast.xml

If you want to go the kwin route @orowith2os (assuming I didn't misunderstood you), I would suggest to:

• Do a prototype directly in smithay, if you rather do that than building your own compositor right away and handling it in there.
• Do a draft PR to discuss it. Though I am pretty sure my stance on the subject would be, that this should probably live in it's own crate.
• I am not sure if there is interest in the smithay community to make that a proper smithay project or even part of the smithay repo itself, but that is something to be best discussed in a draft PR and not my place to decide.
  • (At least @PolyMeilex seems to be very much interested and I can see why such an abstaction could be useful and beneficial to the smithay ecosystem.)

[orowith2os]

If you want to go the kwin route @orowith2os (assuming I didn't misunderstood you), I would suggest to:

I was suggesting more the GNOME way, where both compositor and portal speak the same protocol, so the portal can easily pass the data along. But having it implemented in a Wayland protocol like KDE would be nice too, and would cover everything I want!

Do a draft PR to discuss it. Though I am pretty sure my stance on the subject would be, that this should probably live in it's own crate.

How easy would it be to have a separate crate under the Smithay banner that does exactly this, as an add-on to the core Smithay crate? Would having it as an opt-in module in the core crate be ideal? Though, I figure that's the point of discussing in a draft PR, so I'm more just asking your opinion on the matter.

[Drakulix]

I was suggesting more the GNOME way, where both compositor and portal speak the same protocol, so the portal can easily pass the data along. But having it implemented in a Wayland protocol like KDE would be nice too, and would cover everything I want!

Also possible, but I would personally not suggest adding another protocol, if you can send the same information over the wayland-protocol. GNOME has other reasons to include dbus in their shell anyway, as they handle a lot of different stuff in their monolithic gnome-shell binary.

How easy would it be to have a separate crate under the Smithay banner that does exactly this, as an add-on to the core Smithay crate? Would having it as an opt-in module in the core crate be ideal? Though, I figure that's the point of discussing in a draft PR, so I'm more just asking your opinion on the matter.

That is essentially what I had in mind. I think building something like that should be relatively straight-forward. And if there is interest in maintaining that from the wider community, I am totally fine adopting such a crate under the smithay umbrella.

[PolyMeilex]

How easy would it be to have a separate crate under the Smithay banner that does exactly this, as an add-on to the core Smithay crate?

I would say it's no different to (wayland) screencopy-like protocols, everything needed to capture a screen is already public API, because users have to have a way to implement screen copy protocols (Smithay does not provide one for now, probably will once ext_screencopy is a thing), said crate could use same APIs as users do.

[ids1024]

I was suggesting more the GNOME way, where both compositor and portal speak the same protocol, so the portal can easily pass the data along.

From a performance standpoint, at least, this should be unnecessary. If the xdg-desktop-portal backend uses something ext-screencopy or wlr-screencopy, the portal can pass the SHM or dmabuf file descriptors between the Wayland protocol and pipewire without any significant overhead.

[orowith2os]

From a performance standpoint, at least, this should be unnecessary.

I know the performance is fine, I'm just worried about the compositor needing to implement all of the bits and pieces. It's less work for everyone if Smithay just passes through PipeWire stuff to the compositor, or provides examples so compositors don't have to do too much to get going.