App to App Media Sharing

[tytan652]

Related to Goal: Peer-to-peer media sharing

Usecases

Case n°1

Applications like SnekStudio want to be able to share a stream (maybe even various) to other applications (e.g. OBS Studio) alike to what Spout2 and Syphon allows to do on respectively Windows and macOS with almost zero-latency, so mostly "PipeWire + DMA-BUF" for Linux.

We can also point out Gamescope that is a Wayland compositor that want to enable sharing in nested scenarios where a xdp implementation is not a solution (obsproject/obs-studio#9607).

Case n°2

Others applications want to be able to provide a virtual camera (e.g. OBS Studio) to be used as a camera in other applications.
Some of them already does by using v4l2-loopback which require kernel space API to be used.

As an example, OBS Studio plans to move features to PipeWire (obsproject/obs-studio#7998), but the actual proposed implementation requires an access to the host PipeWire.

Case n°3

Also some applications might want to act as virtual microphone (device to send their own audio stream) alike to one of Easy Effects features but without requiring direct access to the host PipeWire.

Idea of a solution

Note for the following:

• Shown sequence diagram are not strictly sequence diagram
• I didn't deep dive into how PipeWire nodes should be handled, I might do it at the same time that I work on a proof of concept of case n°1

For case n°1

For this use-case, we need to enable applications to provide screen cast sources (virtual) to the ScreenCast portal of the xdp implementation.
So the xdp stack will have to connect to the application point to point D-Bus to make it a screen cast sources provider.

Note: Name in italic are just temporary name

It is also imaginable to simplify the per-application implementation by providing a library that provide most of the basic of the implementation.

Thought about audio on those Screencast

Also related to Goal: Window screencast with audio.

Since the application provide its own video stream, the application could also provide a audio one paired with the latter.

This could improve the Gamescope sub-usecase, and maybe also allow to some power users of OBS Studio to "stream" an OBS session to another locally without having to rely to solution like NDI or DeckLink if implemented in OBS. Yes, chained OBS Studio setup like that do exist.

For case n°2

I didn't give enough thought about it, and like I said earlier I need to dive a little more on how PipeWire works with Portals.

For this one, we need to enable applications to give a video stream to xdp to make it a camera available through the Camera portal.

There is no expectation that the virtual camera can do more than an actual camera, so format-wise it should be like a real camera.

Reminder: Name in italic are just temporary name

For case n°3

This cases is alike to the case n°2 but relies on Goal: Audio portal/#1142.

But unlike case n°2, the naming "virtual microphone" is just used as a figure of speech.

[Mikenux]

From a UX point of view.

For case 1, I think there could be two possible experiences:

1. Direct sharing of the streams from the sharer app. The user is presented with an "Open with" type dialog box with apps that support receiving streams from sharers. Permission to share the streams is granted when the user has selected an app and clicked “Open”. In the dialog box, we can use more appropriate phrasing and words than “Open (With)”.
2. Grant access to the streams from the requesting app. The user is presented with the source selector from the app with which to share the streams. If this is secure, sharers can expose their streams without asking the user. If not, apps capable of sharing their streams report that they are able to do so. Then, when the selector is called from the requester, two possibilities, depending on security: (1) send a signal to open compatible apps so that they share their streams, (2) list open compatible apps and send the signal when the app is selected. In short, the ability for an app to provide its content to other apps is done from the selector, since it is the portal.

There may be an option to share or not share audio from a stream.

For case 2, either we display a dialog asking to add the camera (confirming its creation) and it is directly available for other apps in a selector, or it can be the same as in the second experience of case 1 but for a virtual camera.

[tytan652]

Direct sharing of the streams from the sharer app. The user is presented with an "Open with" type dialog box with apps that support receiving streams from sharers.

1. We can't know in advance which app can/will do what since those have access to all portals by design, so no "Open with".
2. The plan for the new portal is to be a "provider" to other portal (ScreenCast, Camera, Audio) without adding over-complexity to them, only those portals UI/UX will be used including the Access one.

If this is secure, sharers can expose their streams without asking the user.

This is not how permission and the ScreenCast portal works:

• if the permission is unset, the user must be asked if the app can share
• if it's not a session restoration the user must be asked with a selector.

There may be an option to share or not share audio from a stream.

Out of scope, those of thought for the quoted goals, this is to talk about in another discussion about adding audio to the ScreenCast portal.

[Mikenux]

In the second experience, I talk about implicitly granting access to streams from the relevant portals (called "source selector" because we select sources with them). I mean we want to avoid "Allow %s to provide screencast sources".

[tytan652]

I mean we want to avoid "Allow %s to provide screencast sources".

The application must explicitly have permission separately from and before the ScreenCast dialog, there is no point to avoid it.
If the permission is not given, the portal implementation must not be aware of the provider.

[Mikenux]

This is wording that should ideally be avoided, not to not grant permission. Here, it is not common, but it is about displaying in the screencast UI a list of open applications having self-declared (or using any other valid hints) to be able to give their content. Clicking on a listed application means that you are implicitly requesting and giving permission for that application to provide its content. This is, I believe, more or less the same for the OpenURI portal: this dialog is displayed from application A, but you are implicitly giving permission to application B to open a link or file that the application A "has".

Also, I think if the streams are pipewire sources, then they can be presented directly in the screencast UI. I don't think you're granting permission to a portal to access things, because the portal is the one granting access.

[tytan652]

I began by saying that the whole permission block in case n°1 is there because I thought allowing explicitly the sharing app to share would a good idea, the permission looks valid only if asked while the sharing is declaring itself to xdp and not later in the xdp impl ScreenCast selection. If various consider it unnecessary with discussions and agreement, it will be removed.

In my opinion, you are just asking for the removal of this block.

but it is about displaying in the screencast UI a list of open applications having self-declared (or using any other valid hints) to be able to give their content

This portal will not enforce UI/UX guidelines to xdp impls, if they want to show an apps list and then a sources list, or just a sources list. It's up to their design guidelines.
Xdp impls will just be made aware of the sharing apps (if allowed) by xdp, and it will be up to them to show them and their sources with their own UI/UX.

Clicking on a listed application means that you are implicitly requesting and giving permission for that application to provide its content.

"implicit" is different from "explicit", for me the former has no value as a permission. And like I said before the xdp impl have directly access to the sources list.

Also, I think if the streams are pipewire sources, then they can be presented directly in the screencast UI.

Nope, in the ScreenCast portal, PipeWire streams can be created after the source happen to be requested by the xdp impl, they don't have to exist beforehand.

[Mikenux]

Could you describe case 3 further? Is it, for example, an app that applies voice effects and then shares the resulting audio stream?

[tytan652]

This will used as a sort of changelog if I happen to update my illustrations.

Changes to Pseudo-Sequence Diagram for App to App ScreenCast between v1 and v2:

• Replaced application interface names
• Removed the "maybe ?"
• Various wording changes
• Changed how how the xdp request application sources

[tytan652]

Changes to Pseudo-Sequence Diagram for App to App ScreenCast between v2 and v3:

• Application ScreenCast Session design is overhauled
  • When submitted as a provider, no session is created but the D-Bus name is watched and the token is stored for later
    • If the D-Bus disappear, the token is destroyed
    • Potential future addition: Checking if the D-Bus name has the needed interface
  • Sources list is given by the application ScreenCast interface (not the session)
    • If the token is wrong, the provider is ignored
  • The Applicatrion ScreenCast Session is created if a application stream happens to be selected

[tytan652]

Changes to Pseudo-Sequence Diagram for App to App ScreenCast between v3 and v4:

• Move to a point to point D-Bus connection design for the providing app
  • I want to thank Christian Hergert for releasing Collaborating on Builder where I discovered this
  • Just in case, it's Unix Domain Sockets as a transport on the D-Bus spec
  • Removes all to token complexity
  • Removes all the same App ID or D-Bus name stuff since it's now mostly file descriptors

NOTE: I'm working on a POC with point to point to see if it's viable. If it's not, I'll go back to v3 I guess.

[tytan652]

PR for case n°1: #1554

[nlovlyn]

I'm excited to see this change make get PRed! My use case is under n1 for gamescope and screensharing a game on steamdeck (game-mode) to either Chrome for Jitsi or through the discord client.

[swick]

The problem I have with the overall architecture is that it's very complex. If we made pipewire available inside the sandbox and let the portals just control the "permissions" for pipewire, we could clients simply have tag their video sources for consumption of other apps and let pipewire session manager make it available to other clients (if they have the permission).

[tytan652]

I will begin with that case n°1 must not make consumer application add new code paths, it has to go through the ScreenCast API at the end.

For producer apps, I'm willing to listen but nobody really discussed about architecture for more than 1 year so unfortunately expect some kind of tenacity from me.

While I agree the architecture is complex, please understand you'll need to be more thorough on your vision/explanation.

And moreover PipeWire is not an easy API/ecosystem at all, do not expect me to understand everything right away.

[BasiqueEvangelist]

Another usecase for this is letting users stream only one particular Chrome/Firefox/whatever tab, like how it's possible in intra-browser screen capture

[tytan652]

Idea of improvement for case n°1, make xdp behave like an aggregator of providers act as 1 provider with an interface meant for backends.
This could avoid having backend implements p2p providers handling.

[tytan652]

This change where the provider session is managed:

1. End up with ScreenCast CreateSession having a scenario where no Request/Response is emitted.
2. Create a portal interface meant for media managed by xdg-desktop-portal accessed by impls another way (here the interface to get all app sources).
3. Throw a _in_thread_func function in GTask to send the request to avoid 1. scenario.

[hoshinolina]

Just wanted to throw in that I'm also interested, as this change is needed to implement a Spout2-on-Proton/DXVK to PipeWire/OBS bridge reasonably (which would allow the existing Windows Spout2 sender ecosystem to send video to Linux apps when running on Proton). @tytan652 already has some baseline code for the portal/enumeration bits that he shared with me, and I'm building on it to fill in the PipeWire/texture sharing bits to make it work.

[Conan-Kudo]

This seems like overkill for this problem? Why not just directly interact with PipeWire? Accessing the PipeWire socket and pushing something through it is well-trodden ground (there's even code in Weston to do this with its pipewire backend).

[tytan652]

"Path of least resistance" is a problem to be concerned of, if you want to give "power tools" you need to ensure that not every app and user will use those, and ensure that user is informed of the risk (e.g. remind the user every time the app opens).

You need to create a UI/UX that allows the usual flow to exist without making the "power one" the go to.
And this is a hard problem to solve.

[hoshinolina]

I don't think reminding the user every time the app opens makes any sense. That would defeat the whole purpose of making it reliable and automatable.

Yes, making a UI/UX to handle a subset of common use cases safely and intuitively is hard, and I'm not saying that shouldn't happen. But clearly we don't all have infinite time and ability to push things forward, so there has to be some pragmatism. "Perfect is the enemy of good."

There is the ability to meet a large set of use cases in a less intuitive way, with less code and development/debate time. And there is the ability to meet an important subset of use cases in a more intuitive way, with more code and development/debate time. Both are important. I don't think we should give up on doing things "the clean way", and I also don't think doing things "the clean way" means we shouldn't also do "the ugly but powerful way".

I know you and others here think punching holes in the sandbox with unrestricted PipeWire access is bad. I personally don't care about the sandbox and would be perfectly happy to give relevant applications full PipeWire access on my systems, but that's just my personal use case. That's why I'm proposing a middle ground, where the application gets an unpermissioned PipeWire node that every other app with full PipeWire access can connect to, on request by the user through the portal. So no security is lost for users who do not explicitly choose to invoke this functionality.

[hoshinolina]

I have been investigating the permission system. It looks like PipeWire has a special "L" permission that allows a third party to create links between nodes that can't see each other. This means that the client PW node does not need to be given any special permissions, rather it's the manager side that needs an extra bit. The only thing the XDG portal side has to allow is a setup with zero streams.

In other words, the only change needed in xdg-desktop-portal is deleting this check. Surely this shouldn't be very controversial. Portal backends could then simply have a UX for such an unconnected node, and then it's up to the user to use a client with full PW access to manage the connection.

This also obviates any security concerns since by definition no security decision is being made by or through the portal. It is up to the user to use PW superpowers to do what they want.

[tytan652]

I repeat the scope of this discussion:

enabling application to share stream to other apps through the same flow as monitor and windows

I will not extend the scope of this discussion and like I said:

it is not about how the flow is unfit to more "professionnal"/"power user" usecase. This part should be discussed elsewhere.

xdg-desktop-portal is an API, so it's behavior is documented.
A screencast session started without stream is a screencast session that is to be closed by the implementation since the implementation failed to start it.

[hoshinolina]

OK, fair enough.

From what I've seen of the ScreenCast interface after digging deeper, it is unsuitable for professional use cases as it exists today (in particular the restore functionality, which I just discovered is more flawed than I thought and cannot be relied upon safely).

Since the goal here is to not to touch that interface for sinks at all or make any other changes to it, even simple ones, that would make this use case more reliable/possible, then I don't think this particular discussion is of interest to me any more. I'm sure if this moves forward it will be very nice for casual usage, but I can't in good faith recommend it for production usage when it has such fundamental limitations and discussion of how to improve them is seen as out of scope.