Physical keyboard to android support #6097

mcfans · 2023-10-19T07:05:46Z

Description

Support KeyEvent input in android side. Fix issue(#3556)
/claim #3556

Detailed Design

Android doesn't provide an API for send key event to the whole system, since most scenario that needs keyboard input are text input. I used android accessibility service to find current focused text input widget and set text to it. I create an invisible text input widget in InputService, and set current focusing accessibility node info's text and selection to it, send the key event to it then retrieve it's new state and set the new text back to the current focusing node.In this way, we don't need to handle what different key will do to text on our side, just use system default implementation.
Use getInputMethod() API, then send key event to it.

Other Approach

Register an IME to system then use that to send key stroke to system. WIFIKeyboard use this approach. But Android system can only have one IME at a time, receive side will not able to use their on screen IME while using RustDesk, that would make features like chatting not work.

Changes

Modified build.gradle to generate java code from message.proto for usage of KeyEvent in Kotlin code. KeyEvent is a struct with many field, pass its' fields through ffi is complicated, so I pass serialized data through ffi.
Updated rdev, add android key code, make keyboard mode(map) work.
Added canRetrieveWindowContent to accessibility_service_config.xml

Test

Tested on:
iOS -> Android,
macOS -> Android(map mode),
windows -> Android(legacy mode)

Compatibility

Android receive side needs update to this version to work.
iOS control side needs update to this version to show a soft keyboard.
Desktop version doesn't need update for legacy keyboard mode, need update for map keyboard mode.

A Video

IMG_1634.MOV

mcfans · 2023-10-19T07:07:00Z

/claim #3556

rustdesk · 2023-10-19T07:09:59Z

Cargo.toml

@@ -64,7 +64,7 @@ default-net = "0.14"
 wol-rs = "1.0"
 flutter_rust_bridge = { version = "1.75", features = ["uuid"], optional = true}
 errno = "0.3"
-rdev = { git = "https://github.com/fufesou/rdev" }
+rdev = { git = "https://github.com/mcfans/rdev" }


Please merge to https://github.com/fufesou/rdev
@fufesou please review.

I will open a PR for that.

Finish this first

@fufesou Could you take a look at this?

rustdesk · 2023-10-19T07:12:46Z

flutter/android/app/src/main/kotlin/com/carriez/flutter_hbb/InputService.kt

+    @RequiresApi(Build.VERSION_CODES.N)
+    fun onKeyEvent(data: ByteArray) {
+        val keyEvent = KeyEvent.parseFrom(data);
+        val handler = Handler(Looper.getMainLooper())


can not parse it in rust?

It's serialized on rust side by purpose, because I want to use generated KeyEvent class on Kotlin side instead of pass the fields of it on rust side through ffi that would be a lot of parameters and less convenient.

rustdesk · 2023-10-19T07:13:06Z

@fufesou please have more review.

rustdesk · 2023-10-19T07:27:42Z

Have you tested different languages? e.g. German, French. And combination key, e.g. alt + key, will generate special character for different language.

mcfans · 2023-10-19T07:35:45Z

Have you tested different languages? e.g. German, French. And combination key, e.g. alt + key, will generate special character for different language.

Not yet. I will try it later.

fufesou · 2023-10-19T14:04:24Z

👍 👍 👋

Win -> Android , en->en, the charaters are all correct for both "Map mode" and "Translate mode".

android.input.mp4

Some possible improvements

Input events will have a side effect that will set the cursor to the end. So the arrow control doesn't work, the cursor always moves to the end.
The input does not work on the input box on the web page. Tested chrome browser.

Mouse click event issue. The specified position of the text cannot be clicked and range selection cannot be made. This has nothing to do with keyboard input, but I always want to click the mouse to a specified location.

avc permission issues

I'm not familiar with avc issues in android.

They're not related to the keyboard issue. But if you're familiar with it, it's good.

======================= update =============================

I see the secure context is scontext=u:r:untrusted_app:s0:c120,c257,c512,c768.

I'm trying to find out:

What are affected
How to set a domain as a trusted app.

mcfans · 2023-10-19T16:45:58Z

I found a new way to do this. Accessibility service has a method called getInputMethod, we could then send key event directly to it.This could solve previous problem.

Input events will have a side effect that will set the cursor to the end. So the arrow control doesn't work, the cursor always moves to the end.

Old way can only set text and text selection separately, if application doesn't do some special text handling, this is fine.But if app would change text after text set, then our text selection data would be wrong. So I ignored text selection data for correctness. In this new way I leave this to system to handle, arrow key works fine.

The input does not work on the input box on the web page. Tested chrome browser.

This is because findFocus gave a wrong input widget, send event to IME also fix this.

mcfans · 2023-10-19T16:53:31Z

Have you tested different languages? e.g. German, French. And combination key, e.g. alt + key, will generate special character for different language.

I tested combination key, I could enter exclamation mark(shift + 1). I couldn't test different languages for now because I found out my android device doesn't support other languages. Need some help on that.

YangCheung · 2023-10-20T07:45:21Z

flutter/android/app/src/main/kotlin/com/carriez/flutter_hbb/InputService.kt

+    @RequiresApi(Build.VERSION_CODES.N)
+    fun onKeyEvent(data: ByteArray) {
+        val keyEvent = KeyEvent.parseFrom(data);
+        getInputMethod()?.let { inputMethod ->


getInputMethod added in API level 33

You may have some modification to make it compatible with lower level target device. @mcfans

I checked the wrong field, I thought we were using API 33. My bad. So I just keep this for higher version and add the old code back for lower API?

I am not familiar with kotlin about how to call higher API on high target device but still can run on low target device.

My android device may be the lower version. The input does not work now, and it reports:

D/ffi ( 3011): librustdesk::server::connection: call_main_service_key_event fail:Java exception was thrown

Could you try it again? I added old code back with a version check.

The keyboard input works now.

fufesou · 2023-10-20T13:31:46Z

Have you tested different languages? e.g. German, French. And combination key, e.g. alt + key, will generate special character for different language.

The "Translate mode" works fine.

For the "Map mode" (fr -> fr), the the characters are different. It seems the default en-use keyboard layout is used.

Maybe it's better hide the "Legacy mode" and "Map mode" if local is desktop and remote is Android.

======================== update =============================

Maybe you need to use the scancode in the KeyEvent when using the "Map mode".

https://developer.android.com/reference/android/view/KeyEvent#KeyEvent

"Map mode" means using the "position code" (scancode) to control the key inputs.

With "Map mode", people can control the peer as the physical keyboard is connected directly to the peer.

rustdesk · 2023-10-21T07:13:25Z

"Map mode" means using the "position code" (scancode) to control the key inputs.

👍

fufesou · 2023-10-24T05:47:57Z

@mcfans 👍

Do you have a plan to implement the "Map mode"?
It's an important keyboard mode.

mcfans · 2023-10-24T06:31:51Z

I am trying to implement Map mode these days.But there are two problems. First we can only send a keycode that matches US keyboard layout, Android keycode only has those keys. Set scancode of a keyevent is ineffective base on my experiments. Second I only found ways to send key events to application, not send to input method. On Android a hardware keyboard event is sent to input method, then the input method would figure out correct text to insert into application. I failed to find a way to do the same thing. However I have not tested what text would be inserted if the OS language matches input method, because my android device couldn't do that. If that could work, we could add map mode back.

mcfans · 2023-10-24T06:32:37Z

I fixed translated mode and legacy mode in those commits.

fufesou · 2023-10-24T06:46:40Z

what text would be inserted if the OS language matches input method

The scancode should be handled by the input method. It should have nothing to do with the OS language.

I only found ways to send key events to application, not send to input method.

I'm not familiar with Android. But it seems you are right, the event should be sent to the system or input method.

mcfans · 2023-10-24T06:57:49Z

If we can send key event to the system, everything would be perfect.

mcfans · 2023-10-24T07:46:06Z

So should we merge this?

rustdesk · 2023-10-24T07:51:59Z

AnyDesk's map mode work?

mcfans · 2023-10-24T08:18:14Z

I tested it. It doesn't work correctly with 1:1 mode on and local input method is fr, its output is US layout keys. Same behavior with my implementation.

fufesou · 2023-10-29T13:00:15Z

I've tried the ID edit in RustDesk.
The key input also does not work.

fufesou · 2023-10-29T15:14:41Z

src/ui_session_interface.rs

@@ -216,6 +216,11 @@ impl<T: InvokeUiSession> Session<T> {

    pub fn get_keyboard_mode(&self) -> String {
        let mode = self.lc.read().unwrap().keyboard_mode.clone();
+        if self.peer_platform() == crate::PLATFORM_ANDROID {


👍

This commit seems better than
#6168

It's independent of the UI.

But the logic here is the same as in flutter_ffi.rs. Maybe it's better to merge the code.

if session.peer_platform() == "Android" && mode == KeyboardMode::Map { return SyncReturn(false); }

The logic in flutter_ffi.rs is used to hide map mode in UI, ui_session_interface.rs is used to override keyboard mode saved in local storage. It’s two part of the same problem.

It’s two part of the same problem.

Yes, but if something changed, you have to change to two places, which is error prone.

They are doing almost the same filtering, maybe you can extract the filter code, or some func like should_replace_mode_to(conds, mode) -> Option<mode>

I agree, that's better. Working on it.

# Conflicts: # src/server/connection.rs

mcfans · 2023-10-30T04:11:34Z

I made the cursor works for API level lower than 33 just like AnyDesk.

fufesou · 2023-10-30T15:11:42Z

@mcfans Hi, what architecture did you use for testing? It's strange that I can't run x86_64 on my emulator.

fufesou · 2023-10-30T15:42:26Z

With the latest commit, I can't enter "Backspace" "Enter" "Left/Up/Right/Down".

mcfans · 2023-10-30T15:49:14Z

@mcfans Hi, what architecture did you use for testing? It's strange that I can't run x86_64 on my emulator.

For API level 34, I am using arm64 apk on x86 emulator. It’s okay for Android 11 and above. For API level 28, I am using arm64 apk on arm64 emulator. I have a M2 MacBook, it’s arm64, so it’s fine. You probably need —split-per-abi when running flutter build. That will build an apk that contains arm64 only. Without this flag, output apk seems have 3 architecture, dyld trying to open .so in x86 directory, but nothing in there.

mcfans · 2023-10-30T15:57:00Z

With the latest commit, I can't enter "Backspace" "Enter" "Left/Up/Right/Down".

Which API level are you using?

rustdesk · 2023-10-31T13:14:45Z

c2703d2
flutter and bridge updated to new versions.
If you can not run on mac, please rm -rf macos/rustdesk.xcodeproj, pub get/upgrade,

…selection

rustdesk · 2023-11-07T03:34:14Z

@mcfans please fix the conflicts. I will merge then.

# Conflicts: # Cargo.lock # src/server/connection.rs

rustdesk · 2023-11-07T04:39:29Z

src/server/connection.rs

+                            log::debug!("call_main_service_key_event fail:{}", e);
+                        }
+                    } else {
+                        log::debug!("encode key event fail:{}", encode_result.err().unwrap());


Can we remove this unwrap? BTW, space between : and {.

It's safe here, but I changed it to a match block anyway.

basilgello · 2023-11-07T04:57:59Z

Thanks for so awaited feature! :)

rustdesk · 2023-11-16T13:26:51Z

#6417 big regression failure.

Magustructor · 2024-01-17T19:17:14Z

I don't know if here's the right place for reporting it but I have a problem with using keyboard in Windows 10 (R. ver.1.2.4)-> Android 11 (R. ver. 1.2.4) scenario.
The keyboard generally works properly except Enter key.
The Enter moves the cursor to the next line instead of confirming/accepting text when used in any search field like google.
I tried both legacy and translation modes.

Am I doing something incorrectly or it's a bug?

rustdesk · 2024-04-15T12:59:13Z

This feature is ready in 1.2.4, current night build, https://github.com/rustdesk/rustdesk/releases/tag/nightly

mcfans added 5 commits October 17, 2023 22:12

feat: support android keyboard input

bbc2417

feat: legacy mode android keyboard support

22165ec

feat: map mode and translate mode receive side

b3e2ab0

feat: keyboard map mode control side

28c1180

feat: modifier key

26e77ba

rustdesk reviewed Oct 19, 2023

View reviewed changes

mcfans mentioned this pull request Oct 19, 2023

Add android keycode fufesou/rdev#9

Merged

algora-pbc bot mentioned this pull request Oct 19, 2023

physical keyboard desktop -> mobile #3556

Closed

feat: use current input method to send key

9a903a1

YangCheung reviewed Oct 20, 2023

View reviewed changes

fix: add old code back for old version

d900f2c

mcfans added 2 commits October 24, 2023 13:34

fix: use seq for translate mode and use commitText

dd33c0e

fix: hide map mode if peer is android

cbe9b9c

fufesou reviewed Oct 29, 2023

View reviewed changes

mcfans added 2 commits October 29, 2023 23:32

Merge remote-tracking branch 'upstream/master'

7b24835

# Conflicts: # src/server/connection.rs

fix: update text selection for API level lower than 33

70dd3f3

fix: unified keyboard check logic in common.rs

6fdce63

chore: add some log

9521ac6

mcfans added 8 commits November 1, 2023 00:06

fix: set focus and FLAG_RETRIEVE_INTERACTIVE_WINDOWS

47d57dd

fix: set focusable before request focus

100967c

fix: use onKeyDown to dispatch key event

0a94b74

fix: set text before update accessibility node

c49853e

fix: set same rect for correct layout and navigation and set correct …

b155cd9

…selection

fix: add page up/down keymap

409d5b1

fix: set correct flag

6d82724

fix: add proguard rules, avoid protobuf generated classes got obfuscated

e474b59

Merge remote-tracking branch 'upstream/master'

b522de3

# Conflicts: # Cargo.lock # src/server/connection.rs

rustdesk reviewed Nov 7, 2023

View reviewed changes

chore: use a match instead of unwrap error

32a29a5

rustdesk merged commit 5577731 into rustdesk:master Nov 7, 2023
1 of 8 checks passed

sahilyeole mentioned this pull request Nov 9, 2023

Keyboard Input not working #6338

Closed

Physical keyboard to android support #6097

Physical keyboard to android support #6097

Conversation

mcfans commented Oct 19, 2023 • edited

Description

Detailed Design

Other Approach

Changes

Test

Compatibility

A Video

mcfans commented Oct 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcfans Oct 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rustdesk commented Oct 19, 2023

rustdesk commented Oct 19, 2023 • edited

mcfans commented Oct 19, 2023

fufesou commented Oct 19, 2023 • edited

👍 👍 👋

Some possible improvements

avc permission issues

mcfans commented Oct 19, 2023 • edited

mcfans commented Oct 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rustdesk Oct 20, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fufesou commented Oct 20, 2023 • edited

rustdesk commented Oct 21, 2023

fufesou commented Oct 24, 2023 • edited

mcfans commented Oct 24, 2023

mcfans commented Oct 24, 2023

fufesou commented Oct 24, 2023

mcfans commented Oct 24, 2023

mcfans commented Oct 24, 2023

rustdesk commented Oct 24, 2023

mcfans commented Oct 24, 2023

fufesou commented Oct 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcfans commented Oct 30, 2023

fufesou commented Oct 30, 2023

fufesou commented Oct 30, 2023

mcfans commented Oct 30, 2023

mcfans commented Oct 30, 2023

rustdesk commented Oct 31, 2023

rustdesk commented Nov 7, 2023

rustdesk Nov 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

basilgello commented Nov 7, 2023

rustdesk commented Nov 16, 2023

Magustructor commented Jan 17, 2024

rustdesk commented Apr 15, 2024 • edited

mcfans commented Oct 19, 2023 •

edited

mcfans Oct 19, 2023 •

edited

rustdesk commented Oct 19, 2023 •

edited

fufesou commented Oct 19, 2023 •

edited

mcfans commented Oct 19, 2023 •

edited

rustdesk Oct 20, 2023 •

edited

fufesou commented Oct 20, 2023 •

edited

fufesou commented Oct 24, 2023 •

edited

rustdesk Nov 7, 2023 •

edited

rustdesk commented Apr 15, 2024 •

edited