Adopt rustdesk' libdrmtap

[Pesc0]

Select Topic Area

Feature Request

Body

Hi, i am curious to know if you are aware of this library
https://github.com/rustdesk-org/libdrmtap

what do you think of it?
could it be useful for sunshine? would it improve the kms backend?
is there any interest in adopting it?

i am currently struggling trying to setup a headless virtual machine with sunshine streaming.
kms capture would be the only proper way to stream the whole desktop. no getting locked out by permission prompt. no issues with login screens...

i know currently the kms backend is not recommended in favour of xdg, but are there really any drawbacks to using it? would this library make things easier sunshine-side, to make kms a first class citizen?

thank you in advance

[ReenigneArcher]

@psyke83 @Kishi85 thought you both would be interested in this topic.

[psyke83]

The library might be useful in general, but the enhanced use cases you might be imagining won't work for Sunshine simply because we don't run as a global service. How can a per-user Sunshine service run during a greeter login? Even if we force via enable-linger, Sunshine can't initialize when the system tray is enabled because there's no appropriate session to attach to. What happens if user X runs Sunshine but logs into user Y's session? I can imagine a lot of issues such as this.

Aside from that, having a proper forked permission helper could enhance security compared to our drm worker, but I disagree with the notion that it's possible to make KMS capture a first-class citizen. KMS grab will always have the potential to cause conflict with compositors. Having said all that, libdrmtap might improve our KMS support, but I personally wouldn't be interested to implement this myself.

[Kishi85]

Good summary on the caveats.

I'm having some concerns regarding that permission helper binary as it's access restrictions seem to be just using filesystem permissions if I'm understanding the readme correctly. Therefore if a user has access to/can execute the binary it could potentially be used to capture another users session on a screen utilizing the helper directly.

Overall this might improve KMS capture (slightly?) but I'm not sure if it will fix all issues, especially since compositors like KDE/KWin see KMSgrab as unsupported and won't test for it or fix any issues with it that occur in the future (e.g. using KWIN_USE_OVERLAYS=1). So grabbing the KMS framebuffer directly will still be a hacky workaround either way even if we use libdrmtap.

Personally it's not something I'm seeing myself be able to properly implement or check as I'm having personal and work-related things coming up in the next months that will take up most of my time. I'm also mostly running on kwingrab (or portalgrab) these days and those are good enough for my use-case.

[Pesc0]

Thank you for your response.

we don't run as a global service

I thought sunshine had the option to run as a system service? That would fix the issues you mentioned right?

KMS grab will always have the potential to cause conflict with compositors

Could you briefly explain what are the possible issues here?

[psyke83]

Sunshine runs via a systemd user unit on Linux and is intended to run with minimum needed privileges (CAP_SYS_ADMIN is only used by KMS). You might be thinking of Windows where it's run as a system service. I've never used RustDesk, but I assume that the application model works better with libdrmtap as it's designed to run system wide.

KMS grab is inherently a hack that can always break compositors. An example is KWin's recent overlay support which I believe was optional for 6.7 (activated via KWIN_USE_OVERLAYS=1) but will be enabled by default for 6.8. This causes major KMS corruption issues on my system, and reporting such bugs upstream will do nothing but frustrate developers who can't control how such hacks interact with their code. My opinion is that we should be focusing our energies on improving the Pipewire and Wayland-based capture methods instead.

I understand that the Portal token setup is an annoyance, but KDE already has kwingrab or a permanent portal bypass available. It's feasible that GNOME could be persuaded to add a secondary permission system for their Portal implementation as well... otherwise there's probably room for improvement in terms of making our Portal restore token negotiation more resilient.

[Kishi85]

I understand that the Portal token setup is an annoyance, but KDE already has kwingrab or a permanent portal bypass available. It's feasible that GNOME could be persuaded to add a secondary permission system for their Portal implementation as well... otherwise there's probably room for improvement in terms of making our Portal restore token negotiation more resilient.

For GNOME/Mutter it should be possible to do something similar as we do for KDE/KWin with kwingrab but instead of using a wayland extension (zkde_screencast_unstable_v1/kde_screencast_v2 in the future) GNOME/Mutter seems to expose what xdg-portal-gnome is using for portalgrab via it's DBus Interface org.gnome.Mutter.ScreenCast. There's at least https://github.com/fzwoch/obs-gnome-screencast/blob/master/gnome-mutter-screencast.c implementing it this way.

[Pesc0]

Sunshine runs via a systemd user unit on Linux

Yeah, but my question is: running as a system service is not an option?

major KMS corruption issues

I don't use KDE, but i'm curious what causes such issues.
Isn't kms capture below the compositor, aka accessing the framebuffer directly?

I understand that the Portal token setup is an annoyance

it's not just an annoyance, in a VM setup you risk locking yourself out completely. And even an unsafe shutdown or system crash corrupts the token and re-asks for permission, which is a deal breaker in such a setup.

That's why i was focused on kms, because it provides an alternative path below the compositor.
The login screen can be bypassed by enabling autologin, so it's not an issue.
Since I am on gnome, it would be very interesting to try out this screencast functionality that Kishi is talking about. If i'm not mistaken it is used by gnome remote desktop as well. I've seen it somewhere already.

Thank you for your replies :)

[psyke83]

A global service is a non-starter, as it would be too insecure and circumvent the whole point of a drm worker thread or forked privileged helper as used by libdrmtap. The service model Sunshine uses simply doesn't work in the way you seem to want, where you can capture the greeter and all user sessions.

Screencasting would depend on the compositor to be running in the per-user session, so even if we implemented a GNOME-specific screencast method, it wouldn't work on the login screen. All I can suggest is that you use autologin and always have a VNC or RDP connection available in case the Portal token becomes stale.

[Pesc0]

Thanks for the suggestion, i had not thought about having another program as a fallback.
As i said the login is a minor issue solvable by enabling autologin.
I would be very interested in the gnome screencast capture method though.