Key handling improvements

[MrJul]

What does the pull request do?

This PR implements an alternate version of #11797 (as discussed with the team).

It ensures that KeyEventArgs.Key is correctly translated according to the current keyboard layout, with QWERTY fallback for non-latin layouts.
This fixes shortcuts on non-Windows platforms for users of non-QWERTY latin keyboard (e.g. French AZERTY), while still retaining existing latin based shortcuts for non-latin keyboard (e.g. Russian JCUKEN).

Note that it's only valid for digit and alphabetic keys. Users shouldn't rely on Key for punctuation symbols, as it's implementation and keyboard layout dependent.
In the future, maybe we could obsolete Key and provide an alternative enum with alphanumeric keys that are propertly mapped and supported on all platforms.

Additionally, two new members are implemented on KeyEventArgs:

PhysicalKey

This property indicates which physical key has been pressed. It's layout-independent and should be used when the physical position of a key on the keyboard matters (for example, WASD player movement in a game). It corresponds to the position of the key on a QWERTY keyboard when possible.

This doesn't reuse the Key enumeration, which has too many virtual key members and is hard to use for this purpose. Instead a new PhysicalKey enum is introduced, with members from https://www.w3.org/TR/uievents-code/

KeySymbol

This property returns the symbol corresponding to the pressed key, if available, while taking the current keyboard layout and modifiers (shift, caps lock, num lock) into account.

Example

While pressing the key that matches the letter Z on a QWERTY keyboard, the properties have the following values:

Keyboard Layout	PhysicalKey	Key	KeySymbol
QWERTY (English)	Z	Z	z
AZERTY (French)	Z	W	w
QWERTZ (German)	Z	Y	y
JCUKEN (Russian)	Z	Z	я

Implementations

Win32

Win32 already translated virtual keys automatically.
It provides scan codes in the lParam for WM_KEYDOWN: a scan code to PhysicalKey map has been added.
Symbols are coming from a call to ToUnicodeEx.

macOS

macOS only provides device-independent key codes (physical keys).
These are translated to characters according to the current keyboard layout (which is the key symbol), and the characters are mapped to keys as best as possible, with QWERTY fallback.

X11

X11 provides device-independent key codes, and allows translating them to symbols.
The X core functions do the right thing only when you only have a single keyboard configured (or if the current one is the first) as it doesn't handle multiple keyboards.
With the Xkb extension enabled (which should be almost everywhere, it's been the default for years), we have access to new functions that correctly handle the current layout. Avalonia now makes use of these Xkb functions (XkbLookupKeySym), and falls back to the X core functions if they're unavailable (XLookupString).

Browser

The physical key comes from the event's code property with fallback to key (because browsers can't follow recommandations apparently).
The key comes from the event's key property, with fallback to QWERTY.

Android

The symbol directly comes from the event's DisplayLabel.
Physical keys aren't handled yet: I'm waiting for an USB-A to USB-C adapter to arrive to properly test with a physical keyboard, as it wasn't clear what translation came from the emulator itself. It will be another PR, and is considered low priority as most mobile users use the touch screen.

VNC

Key symbols come from the VNC protocol directly (they're X11 keysyms).
The PhysicalKey is always None as we have no way to know it.

Dead keys

Dead keys have different behavior depending on the platform:

• On Windows, they're correctly reported by the OS and Avalonia reports them in KeyDown.
• On macOS, they're correctly reported by the OS but Avalonia filters them (Dead key doesn't trigger KeyDown on macOS #12483).
• On X11, they're not reported as they're filtered by XFilterEvent. They're only reported if dead keys are disabled (no XCreateIC call or a call with XIMPreeditNone), which isn't useful. There's not much we can do here.
• On the browser, they're reported as Dead: we can't get the original symbol.

New public API

Base

• Avalonia.Input.KeyEventArgs.KeySymbol (property)
• Avalonia.Input.KeyEventArgs.PhysicalKey (property)
• Avalonia.Input.PhysicalKey (enum)

Those members are explained above.

• Avalonia.Input.PhysicalKeyExtensions.ToQwertyKey (method)

ToQwertyKey is here to allow platforms to have a quick fallback from PhysicalKey to Key if they don't handle multiple keyboard layouts.

Previewer

• Avalonia.Remote.Protocol.Input.KeyEventMessage.PhysicalKey (property)
• Avalonia.Remote.Protocol.Input.KeyEventMessage.KeySymbol (property)

New properties for the previewer implementations to use in future updates, to match the new KeyEventArgs.
Existing implementations obviously don't pass this new member, which won't cause any problems to the BSON deserializer, resulting in PhysicalKey = PhysicalKey.None and KeySymbol = null in this case.

Headless

• Avalonia.Headless.HeadlessWindowExtensions.KeyPress (method)
• Avalonia.Headless.HeadlessWindowExtensions.KeyRelease (method)

New overloads of the existing KeyPress/KeyRelease extension methods that take physicalKey and keySymbol parameters.

Win32

• Avalonia.Win32.Input.KeyInterop.PhysicalKeyFromVirtualKey (method)
• Avalonia.Win32.Input.KeyInterop.GetKeySymbol (method)

The KeyInterop conversion class is public as it was requested in #10322.
Two new methods have been added to convert from WM_KEYDOWN-like lParam/wParam to the physical key and key symbol.

Obsoletions

• Avalonia.Input.Raw.RawKeyEventsArgs.ctor (constructor)

A new constructor taking the physical key and key symbol has been added and used everywhere: the existing constructor is now obsolete to encourage implementations to provide correct values.
This should only impact platforms outside of the core that choose to enable Avalonia's private APIs.

• Avalonia.Headless.HeadlessWindowExtensions.KeyPress (method)
• Avalonia.Headless.HeadlessWindowExtensions.KeyRelease (method)

The existing overloads that don't take the physical key and key symbol are now obsolete, to encourage callers to provide correct values.

Misc

• Cleaned up and optimized WindowsKeyboardDevice.
• Removed the outdated regen.sh file in Avalonia.Native (the regen has been happening on build for a while).
• Fixed X11 XLookupString declaration, which had an incorrect out parameter, potentially overwriting stack values.
• Added nullable annotations to X11Window.Ime.cs.

Found issues

This are pre-existing issues I found while testing:

• macOS: dead key is filtered by Avalonia (Dead key doesn't trigger KeyDown on macOS #12483)
• macOS: KeyDown event isn't triggered if there's handled text input, which isn't coherent with all other platforms (issue to open)
• macOS: Key.Fn{Left|Right|Up|Down}Arrow exist but there's no path getting to them
• VNC text input doesn't work for non-ASCII characters (VNC: allow non-ASCII text input #12573)

Fixed issues

• Fixes Invalid key mapping with azerty keyboard and ctrl modifier #2049
• Fixes AZERTY keyboard mapping to QWERTY Key event on macOS #9424
• Fixes X11KeyTransform doesn't respect keyboard layouts #10123
• Fixes Hotkey issue in Browser (azerty vs querty) #11065
• Fixes [Spec] Key Handling #11797

[Gillibald]

This should also fix disabled IME on Mac because keys are translated without the InputContext

[sards3]

I like this improvement. But can we also add the physical scan code? That could be useful in some scenarios.

[MrJul]

I like this improvement. But can we also add the physical scan code? That could be useful in some scenarios.

It's unlikely, as each platform as its own notion of scan code: they're not portable, and are of different types (for example the closest thing in the browser is the code string). However, we could expose platform-specific public method to translate from a scan code to a PhysicalKey and vice-versa if necessary. It's already the case for Windows.

[sards3]

However, we could expose platform-specific public method to translate from a scan code to a PhysicalKey and vice-versa if necessary. It's already the case for Windows.

That would be great.

[grokys]

macOS: KeyDown event isn't triggered if there's handled text input, which isn't coherent with all other platforms (issue to open)

I think this should be fixed by #12560 ?

[MrJul]

I think this should be fixed by #12560 ?

Indeed it does, and even fix the dead key handling too!
But there seems to be new issues with IME after this change, I'm opening an issue to track them. Edit: #12571

[avaloniaui-team]

You can test this PR using the following package version. 11.0.999-cibuild0038694-beta. (feed url: https://nuget-feed-all.avaloniaui.net/v3/index.json) [PRBUILDID]

[Mrxx99]

If this is 0.10 backport candidate, would it not make sense to also be 11.0.x backport candidate?

[grokys]

If this is 0.10 backport candidate, would it not make sense to also be 11.0.x backport candidate?

Possibly... This is in the grey zone between bug fix and feature. My only concern for a 11.0.x backport would be that it might need more testing than a simple bug fix.

Note that for 0.10.x we only have one version number free, so bug fixes and features all get squashed into that.