Keyboard mapping mismatch with connection from Android to Debian Wayland

[kochstefan]

Bug Description

There is a keyboard mapping mismatch bug with rustdesk 1.2.1 when connect from Android to Debian Wayland.
This issue does not affect Android to Debian X11.

Examples:

• With a German keyboard y and z are swapped => If Wayland and Android keyboard layout is both US mapping of z and y is correct
• Write capital letters does not work using the Shift button from the Android virtual keyboard. The Shift button shortcut from rustdesk has to be used.
• Special characters and numbers from Android virtual keyboard sub pages does not work.

It seems to be that the virtual keyboard keys are mapped as keys from a hardware keyboard and no translation support is given. Using the virtual keyboard like a hardware keyboard is not sufficient, because it uses sub pages and is missing many keys on the main page.

How to Reproduce

Running rustdesk service on Debian 12 or Testing system using Wayland with KDE. Running rustdesk client on Android 13.

Expected Behavior

Android virtual Keyboard is translated in the correct way. There should be a switch to select translation mode or hardware visual mapping mode on Android as well as on desktop systems.

Operating system(s) on local side and remote side

Android 13 -> Debian Testing Wayland

RustDesk Version(s) on local side and remote side

1.2.1 -> 1.2.1

Screenshots

Not needed.

Additional Context

No response

[rustdesk]

tracked by #3565

[rustdesk]

@sahilyeole

[sahilyeole]

I ran into #2798 error while connecting android/mac -> debian 12 kde plasma wayland. Xfce working.
remote side:

I will try to reproduce this keyboard mapping issue on gnome wayland soon.

---

Reproduced in android -> debian gnome wayland, working on it.

[sahilyeole]

I ran into #2798 error while connecting android/mac -> debian 12 kde plasma wayland. Xfce working.

This comment can be ignored. kde wayland is working for me now.

[sahilyeole]

Found solution for these 2:

• Write capital letters does not work using the Shift button from the Android virtual keyboard. The Shift button shortcut from rustdesk has to be used.
• Special characters and numbers from Android virtual keyboard sub pages does not work.

Reproduced on ios too.
Mobile -> wayland seems to be using only layout mode and evdev keyboard mappings which are only limited to physical keyboard events.

// android virtual keyboard (not working)
log: send keyboard data Keyboard(KeyUp(Layout('G')))
log: send keyboard data Keyboard(KeyUp(Layout('C')))
// android physical keyboard (simulator) (working)
log: send keyboard data Keyboard(KeyDown(Shift))
log: send keyboard data Keyboard(KeyDown(Layout('d')))

So I have added a shift key simulation like a physical keyboard for those characters which are not available in evdev mappings, inside of else block below this line

rustdesk/src/server/uinput.rs

Line 346 in c5a864e

}

with this shift key simulation, the virtual keyboard characters are turning into physical keyboard events. I tried it for capital letters and a few special characters.

Wondering if there's any other better solution.
@fufesou can I use Raw(code) here instead of Layout('character') like you have used it for map keyboard mode?
I tried but it was printing incorrect characters, maybe I tried it wrong.

• With a German keyboard y and z are swapped => If Wayland and Android keyboard layout is both US mapping of z and y is correct

Also, for this no clue yet. Since its using physical keyboard only combinations.
Maybe if Raw(code) is possible then it may work.

[fufesou]

@sahilyeole Maybe you can try the following methods:

1. Implement the map mode for mobile

rustdesk/flutter/lib/models/input_model.dart

Line 137 in c5a864e

if (isDesktop && keyboardMode == 'map') {

2. Try to find the keycode for the key event, depending on the current input method. Or input keys through Xwaylalnd.

But Wayland does not enable an input method engine (IME) by default, you may need to install one and then

https://en.opensuse.org/SDB:Enable_input_method_in_Wayland

I'm not sure if rdev can be used when different IMEs are enabled.
fcitx and ibus may support input through Xwayland.

Uinput is not necessary if Xwayland can be used.

You may need more tests.

[rustdesk]

Please test the latest nightly build

[mikekgr]

Please test the latest nightly build

The today's nightly release is working fine to me (Windows 10 => Android 13 phone)