Feat: Wayland flatpak input support | Remote desktop portal

[sahilyeole]

Fixes #5949

Description:

Utilise the remote desktop portal interface (provided by xdg desktop portal) when uinput is unavailable for input capture on wayland.

Advantages:

Wayland input is supported similar to uinput on:

1. Flatpak
2. While dev/debugging (cargo/flutter run)

Supports:

It supports input from:

• Desktop
• Mobile
• Physical keyboard mobile

Events:

• Key inputs
• Mouse movements, clicks, and scroll (x & y axis)

Tested:

Remote: ubuntu, arch (kde, gnome)
Local: mac, linux, ios, android

Note:

During testing, I found an issue that occurs even without the changes of this PR (also occurs on 1.2.3):

• Mobile -> ubuntu wayland flatpak stream freezes but input still works (rdp input; enabled by this PR).

Build for test/review:

rustdesk-1.2.4-x86_64_wl_rdp.flatpak

Test videos:

rdp_desktop.mp4

rdp_mobile.mp4

[rustdesk]

remote desktop portal

So we can remove uinput? what limit does remote desktop portal have?

[rustdesk]

@fufesou please help review this.

[sahilyeole]

remote desktop portal

So we can remove uinput? what limit does remote desktop portal have?

Yes, we can remove it. I could not find any limitations.
Teamviewer is only using the remote desktop portal for input.
Though I would like to know @fufesou's opinion on it.

[rustdesk]

remote desktop portal

So we can remove uinput? what limit does remote desktop portal have?

Yes, we can remove it. I could not find any limitations. Teamviewer is only using the remote desktop portal for input. Though I would like to know @fufesou's opinion on it.

@fufesou please confirm and test. If so, let's remove uinput to keep cleaner code.

[rustdesk]

@sahilyeole have you tested appimage? can it accept input with rdp?

[sahilyeole]

No, I haven't tested appimage yet.
I will check in a few hours (currently away from my pc).

[sahilyeole]

https://github.com/rustdesk/rustdesk/blob/master/src%2Fserver%2Fuinput.rs#L549-L551

uinput currently doesn't support scroll on x-axis.
rdp_input supports it, I have also tested it.

[fufesou]

@sahilyeole I've tested on Ubuntu 22.04.
20.04 does not support org.freedesktop.portal.RemoteDesktop and org.freedesktop.portal.ScreenCast by default.

The input works fine.

But the image may not be transferred after some time.

I've also tested the latest nightly build of the master branch. The images transferring is fine.

aa.mp4

[sahilyeole]

Yes, I noticed it once in a while on Ubuntu. I will look into it soon.

Meanwhile, if you want to test more about input you can try on arch. Stream works on arch without freezing.

[rustdesk]

But the image may not be transferred after some time.

@sahilyeole why does your PR for keyboard/mouse impact screen capture?

[sahilyeole]

@sahilyeole why does your PR for keyboard/mouse impact screen capture?

I had to rearrange API calls in pipewire.rs for remote desktop API to work along with screencast API.
I made sure the stream is not affected. Also, the stream works the same as earlier on deb version of this PR. My changes should've not specifically affect flatpak on ubuntu 🤔. I'm now rechecking if there's anything I missed.

---

@sahilyeole have you tested appimage? can it accept input with rdp?

Appimage on wayland gives this error while connecting (with or without this PR changes). It does not show video stream.
If this error is solved input may work.

---

20.04 does not support org.freedesktop.portal.RemoteDesktop and org.freedesktop.portal.ScreenCast by default.

My bad I didn't test 20.04, now working on it.

[fufesou]

My bad I didn't test 20.04, now working on it.

I don't think it's necessary to test Ubuntu 20.04 since it's old and the default desktop doesn't support org.freedesktop.portal.RemoteDesktop and org.freedesktop.portal.ScreenCast.

[sahilyeole]

I've also tested the latest nightly build of the master branch. The images transferring is fine.

@fufesou can you please try nightly flatpak again?
It is also freezing on ubuntu, same as this PR.
I also checked 1.2.3, same case.
It doesn't always happen, so when you checked it might have not happened.

---

I don't think it's necessary to test Ubuntu 20.04 since it's old and the default desktop doesn't support org.freedesktop.portal.RemoteDesktop and org.freedesktop.portal.ScreenCast.

So currently the connection to ubuntu 20.04 wayland must also be failing anyway since we need org.freedesktop.portal.ScreenCast for video stream.

[fufesou]

@fufesou can you please try nightly flatpak again?
It is also freezing on ubuntu, same as this PR.
I also checked 1.2.3, same case.
It doesn't always happen, so when you checked it might have not happened.

Yes, the nightly flatpak has this issue.

This issue seems happen when the images are not changed, though it doesn't always happen.

bb.mp4

[sahilyeole]

So do I need to do any other code changes?
Since the stream freeze issue is not related to this PR.
I checked the logs I could not find anything particular for stream freeze, I will debug more after this PR is ready.

Flatpak on arch can be tested for rdp_input capture without stream freeze.

[rustdesk]

AppImage?

[rustdesk]

Code is not beautiful in this PR. Also please review all unwrap again.

[fufesou]

I've tried comment

https://github.com/rustdesk/rustdesk/blob/master/libs/scrap/src/wayland/pipewire.rs#L242

But the flatpak issue persists.

There's another strange issue. My VM (Ubuntu 22.04, Wayland) is easily to get stuck after running for a while after "rustdesk-1.2.4-x86_64.flatpak" is installed.

It does not always happen. I need to test more.

[sahilyeole]

AppImage?

Appimage has this issue:

@sahilyeole have you tested appimage? can it accept input with rdp?

Appimage on wayland gives this error while connecting (with or without this PR changes). It does not show video stream.
If this error is solved input may work.

It does ask the user for permission for remote interaction. If the stream works the input must also work.

[sahilyeole]

Thanks for the suggestions. I've made the changes.
It is working the same as before.