[WIP]Support inject unicoe text

[pangliang]

Refer to the Vysor method, add a custom ime to enter unicode characters

• add IME 'ScrcpyIME' to handle InjectText
• add switch cmd 'ctrl+e' to use ime, 'ctrl+shift+e' to use injectKeyEvent
• change original handler behavior:
  • change text_input to send all types char
  • when enable IME, injectKeyEvent skip Letter or Digit or Space char, InjectText to handler it
  • when disable IME, InjectText skip all text, injectKeyEvent to handler it

[WIP]

• ime build script
• enable/disable ime by cmd and scrcpy liefcycle
• auto install ime apk

[AndroidDeveloperLB]

This won't work, because the moment you disconnect the device, the app will still be the default keyboard.
Plus, sometimes we do want to have the default keyboard...
Isn't there another workaround for this?

[pangliang]

4 SDL Events here: SDL_TEXTINPUT, SDL_TEXTEDITING, SDL_KEYDOWN, SDL_KEYUP
https://wiki.libsdl.org/SDL_TextInputEvent

I think inputtext event and key event are two completely different events and should not be converted to each other.
The inputtext event is only generated when the input method (IME) window is show.
The IME window is only show when edit text area(ui) is focus

when android device edit text area is focus, You definitely want to enter text, you care about the hold text, not a single character
otherwise, for example when you are playing a game; you want to know when the keyboard button is down and up; you will care about the SDL_KEYDOWN and SDL_KEYUP events

Scrcpy splits text into character arrays to handle:

private int injectText(String text) {
        for (char c : text.toCharArray()) {
            injectChar(c)
        }
}

but, a lot of text is 2 or even 4 bytes of unicode string, which cannot be split into a single char.

this commit work for me, ctrl+e to use customer IME to input chinese, ctrl+shift+e to use the original injectKeyEvent to play game

BTW,
InputMethodService. onWindowShown/Hidden on(Un)BindInput can help to automatic switching of inputtext handler and keycode handler
but, it need to create some link from ime to scrcpy app, I think it is too complicated, it is enough for me to use ctrl+e

[pangliang]

vysor use a Custom IME method to enter text
https://www.vysor.io/

[rom1v]

Thank you for your work.

Supporting non-ASCII characters is a feature which is requested a lot (especially for Chinese or other non-ASCII languages), so that's great 👍

I did not have time to review or test yet (I would like to finish things to be fixed for v1.11, and maybe investigate the new audio API first), but it's in my TODO list :)

I think inputtext event and key event are two completely different events and should not be converted to each other.

Yes, I agree. It's just a constraint from key injection.

However, on the computer, when you press a for example, you receive both the key events (A pressed, A released) and the text event ("A"). If we forward both, there will be two a inserted, so we still have to decide which ones must be forwarded.

when android device edit text area is focus, You definitely want to enter text, you care about the hold text, not a single character
otherwise, for example when you are playing a game; you want to know when the keyboard button is down and up; you will care about the SDL_KEYDOWN and SDL_KEYUP events

Exactly, that's #87. And on dev branch, a new option had been added to "prefer text events", as a mitigation (#650): c916af0

(Btw, you should target dev branch instead of master)

but, a lot of text is 2 or even 4 bytes of unicode string, which cannot be split into a single char.

This is even worse than that, this method only works for ASCII text (on 7 bits) (note that char is an unsigned 16 bits in Java, so in theory 2 bytes unicodes could be supported). For example, injecting é does not work (but there is a workaround so we can inject some accented characters).

Questions:

• what happens when you unplug the device? do you need to manually change the IME?
• does it work when you copy-paste chinese text directly (Ctrl+c on the computer, Ctrl+v in scrcpy) while the IME is enabled?

[pangliang]

@rom1v Thank you for your reply and endure my poor English. :)

However, on the computer, when you press a for example, you receive both the key events (A pressed, A released) and the text event ("A"), so we still have to decide which ones must be forwarded.

Yes, I agree. The main problem is that we can't know which component(device, android ui) is being focused.

SDL provides two methods, let developers choose whether to send inputtext event.

• SDL_StartTextInput(): only InputText event, no keyEvent
• SDL_StopTextInput(): only keyEvent event, no InputText

But, the scrcpy app(computer side) does not have an EditText component. So, just on the scrcpy side, it’s difficult to decide when to start and to stop textinput.
On the device(android) side, if we can know that the inputtext view is the focus, we can make a decision. Fortunately, IME can know this thing. use InputMethodService class onWindowShown() and onWindowHidden() callback function

But unfortunately, I am trying hard to get IME and scrcpy server to communicate directly with each other. This won't work. app_process app can send broadcast but can not recv.
So we have to find a way. For example, create a tcp link from IME to scrcpy app (computer side). IME send InputMethod status to app and app forward to server.

This is too complicated ! In order to enter text, this is too expensive.

The scrcpy achieves great features with the most succinct code.
So I think this should also be kept simple. let users make this choice is simple.

what happens when you unplug the device? do you need to manually change the IME?

As it is now, yes. But as I said above, this can be smarter.

does it work when you copy-paste chinese text directly (Ctrl+c on the computer, Ctrl+v in scrcpy) while the IME is enabled?

yes, it works

[rom1v]

Merge branch 'dev' into input_method

Oh, you should not do that, but rebase instead, to get a clean reviewable patchset in the end. :)

[AndroidDeveloperLB]

Is it possible to do it using root instead of an additional keyboard app?

[AndroidDeveloperLB]

Closed because implemented?
How can I use it without changing keyboard app?