Text input rework

[kekekeks]

Unfortunately, we can't just ask OS to give us a character alongside with a key press event. Most complex text input systems compose characters from multiple key presses and we shouldn't try to interpret these key presses at all, because they are not intended by the user to be interpreted in application-specific way. So we need a way to indicate if a control expects text input at all or wants just raw key events. Our iOS/Android backends do that by checking if focused control is a TextBox.

Another problem is "preedit" or "marked" text.
GTK app with preedit:

Winforms app without preedit

See this video: https://vimeo.com/211472594

As you can see, input method interacts with text input area to display incomplete text sequences. This is done with XIM callbacks and NSTextInputClient (we partially implement NSTextInputClient to get text input at all).

That's quite a lot of methods and it's not quite possible to properly express them using routed events. Avalonia is built on the routed event model, so I propose to determine input handler (if any) by sending GetTextInputHandlerEventArgs and expect TextInputHandler property to be set if someone wants text input. Then everything text-related will be passed directly to said handler.

For now it will contain only one method, so we could solve at least the first problem and lay the foundation for text input system interaction

Old OnTextInput will be removed.

@AvaloniaUI/core

[grokys]

@kekekeks yeah, good points. I'm no expert on this, so forgive my ignorance, but I have a few questions:

• Regarding sending an event (GetTextInputHandlerEventArgs) rather than having, say, an interface which exposes the handler: is this because you forsee the need to add such a handler to arbitrary controls? From my limited understanding I would have thought the need for such a text handler would be something decided when writing the control itself?
• I'm fine with removing TextInput, but could it maybe be seen as a "higher level" event? i.e. TextInput would be invoked by the text input handler for simple scenarios?

[kekekeks]

The problem is that we can't handle raw KeyDown/KeyUp events and text input events at the same time. We have to choose which ones we need in the particular control. So we can't have this event on an arbitrary control, unfortunately.

Custom event for selecting the control allows something other than currently focused control (it's parent control, for example) to handle input events.

[Gillibald]

https://www.x.org/releases/X11R7.6/doc/libX11/specs/XIM/xim.html

[grokys]

@Gillibald could we have some context for that comment?

[Gillibald]

That's the specification for the XIM protocol and should help understanding how input methods are implemented. @kekekeks mentioned this in gitter and maybe it helps to better understand how other systems handle this with some kind of IME.

https://docs.microsoft.com/en-us/windows/desktop/dxtecharts/using-an-input-method-editor-in-a-game

Even when this states it is for game integration it should give some insides of the windows solution.

WPF uses this https://docs.microsoft.com/de-de/windows/desktop/TSF/text-services-framework

[grokys]

Thanks @Gillibald - that gives me plenty of reading material ;)

[kekekeks]

We could use some kind of a "virtual" text store that only contains preedit sequence (manipulated by XIM, TSF and Cocoa) and current cursor position (manipulated by our textbox). That will keep our public API more stable since the API surface will be smaller.

[kekekeks]

Wayland's text-input protocol - https://github.com/wayland-project/wayland-protocols/blob/298d888ac718eae57ff2245d373d4327074506ea/unstable/text-input/text-input-unstable-v3.xml

It requires "surrounding text", which doesn't seem to be exactly compatible with "virtual text store" approach.

Also see content hints and content purposes here: https://github.com/wayland-project/wayland-protocols/blob/298d888ac718eae57ff2245d373d4327074506ea/unstable/text-input/text-input-unstable-v3.xml#L192 They seem to be intended for on-screen keyboards on mobile devices.

[Gillibald]

This is the Android API: https://developer.android.com/guide/topics/text/creating-input-method

Just some thoughts:

If current focused element can receive text input it is registered with the current input method. Input method itself is responsible for intercepting key events. Key events that are not handled by the input method reach the focused element.

The goal for Avalonia's Api should be simplicity. If an input method is available let it handle text composition if not let the control handle key events. Each control that can receive text input needs to register itself with the current input method if one is available and unregister when focus is lost.

There has to be a composition buffer that allows us to show a preview of the currently typed text.

An input method might be used for auto completion.

Maybe it is better to have a default input method implementation that processes the raw key events like the current text input event. That way we only have to deal with one way of processing text input.

This also allows us to support a virtual keyboard.

[kekekeks]

My current idea is to let routed event system to select the input method client (some control on the event route should set InputMethodClient property on the routed event) and set some input properties like keyboard type (normal, digits only, email, url, etc).
The rest should be handled by ITextInputClient implementatiin provided by the text input control. That interface is required to handle preedit, marked text, selection, surrounding text and other queries required by the input method.

[Gillibald]

There can only be one active input method per application. So if a control gets keyboard focus it can be associated with the current input method. I don't see why we would want to have a routed event for input method selection. Do you think there could be multiple input methods within the visual tree?

[kekekeks]

For input method client selection. It's needed so the focused control itself doesn't need to implement ITextInputClient and can delegate that to another class. It also needed for scenarios where there is some control that have child controls and wants to act as ITextInputClient for them, i. e. a rich text editor that can have buttons, images and other content inside. Basically it allows to implement some scenarios that are currently implemented by previewing TextInput event on children

The workflow would be somewhat like this:

1. Focus gets changed
2. Avalonia triggers RequestTextInputClient routed event on focused element
3. Avalonia checks TextInputClient property, if it's null, it assumes that current focused control doesn't support text input and asks the platform backend to disable IME
4. If TextInputClient isn't null, Avalonia checks if it maches the previous one, if it does, no action is needed
5. Avalonia asks the platform backend to enable IME and passes it ITextInputClient instance it got from the routed event

[Gillibald]

That's the most flexible approach. Ideally, we only have to deal with an input method rather than having to process raw key events in the control that supports text input. To only have one code path we could have a default implementation that processes key events like the current implementation. So a keystroke is directly committed to the text buffer.

To achieve this we would walk up the visual tree until we reach the root. If no input method was registered we use the default implementation. This would keep things simple for TextBox, etc.

[kekekeks]

Yes, platform backend should still act like it has a complex input method even if it just translates keystrokes to symbols with 1-to-1 mapping. Also, the change of the current input method shouldn't affect the client control in any way other than resetting the current preedit text.

To achieve this we would walk up the visual tree until we reach the root. If no input method was registered we use the default implementation. This would keep things simple for TextBox, etc.

I kinda lost you here. Are you sure that you aren't mixing up input methods (platform-specific or application-specific way of inputing text, e. g. keyboards, popup-assisted keyboard, virtual keyboard, voice recognition, handwriting recognition, etc) with input method client (control-side interface that's required to interact with the input method, ITextInputClient in the proposal).

Do you mean that we select ITextInputClient by walking up through the visual tree? That's why I've proposed to use routed events, they are doing exactly that and allow to override the choice made by the child controls.

[Gillibald]

Every control that handles text input should implement ITextInputClient. A control that implements ITextInputClient will then try to select an input method via visual tree traversal aka routed event.

If the control finds an input method implementation it registered itself with the found input method.

When it could not find an input method implementation it uses the default that acts like the current TextBox implementation with an empty composition buffer because it directly commits to the text buffer.

[kekekeks]

If the control finds an input method implementation it registered itself with the found input method.

Input method is something provided by the OS (or by a TopLevel-hosted overlay). One can't find it in the visual tree. It is also pretty much selected by the OS, not the app (some platforms allow some control over input method selection, but not a complete one).

Please, take a look at Qt5 input method API. So far it's the only one that properly works across all desktop and mobile platforms. I don't think why we shouldn't follow the same idea.

[Gillibald]

If you don't want a customizable input method implementation you don't have to have a routed event to look for some input method. As you write there can be multiple implementations. One could be some IME, one a virtual keyboard, one just auto-complete, etc. and no not all of them are OS-specific. It is possible to have a managed implementation etc.

The input method implementation is just a black box that communicates with the current
ITextInputClient

[driver1998]

Current status with IMEs before proper IME support lands:
Windows: sort of works, you can type stuff using IME, but the compose UI will show up on the upper left corner instead of the Textbox
Linux: does not work at all, types out as exact keystrokes (#5111)
macOS: unknown

[wuzlai]

Current status with IMEs before proper IME support lands:
Windows: sort of works, you can type stuff using IME, but the compose UI will show up on the upper left corner instead of the Textbox
Linux: does not work at all, types out as exact keystrokes (#5111)
macOS: unknown

Excuse me, I just did the MAC test, and found that MacOS supports input method to input Chinese. IME seems to be ok, but the TextBox receives a confused value. Map it out for reference.

[ryancheung]

If someone is going to write Windows IME support code, https://github.com/ryancheung/ImeSharp this could be a reference, it handles both IMM32 and TSF API.

[kekekeks]

@wuzlai use a font that has Chinese characters. https://github.com/anthonyfok/fonts-wqy-microhei is a good one
WenQuanYiMicroHei-01.ttf.zip

FontFamily="avares://ControlCatalog/Assets/Fonts#WenQuanYi Micro Hei"

[wuzlai]

@wuzlai use a font that has Chinese characters. https://github.com/anthonyfok/fonts-wqy-microhei is a good one
WenQuanYiMicroHei-01.ttf.zip

FontFamily="avares://ControlCatalog/Assets/Fonts#WenQuanYi Micro Hei"

thank you ,you are right.Another solution is to call the system font directly.
FontFamily = "Microsoft YaHei,Simsun,苹方-简,宋体-简"

[robloo]

[RDY] win32 ime #6223