Broaden the scope of the specification to support web IDEs

[spoenemann]

The current draft states the following non-goals:

• Identifying a keyboard layout or locale
• Being able to convert from a code value + modifiers into the key value that would be generated by the current locale and layout with those modifiers present.
• Being able to convert from a key value into the set of code values required to generate that value, given the current locale and layout.

I understand that for the currently described user scenario, keypress instructions in games, this is acceptable. There is another scenario that is unsufficiently supported by web APIs: command keybindings in web IDEs. In this other scenario, the three stated non-goals are exactly the missing piece that would allow to handle keybindings for IDE commands properly. This can be explained best with an example.

Example

The Visual Studio Code keybinding for the command "Toggle Line Comment" is Ctrl+/ on Windows and Cmd+/ on Mac with US keyboard layout. On many other keyboard layouts, the / character does not have its own key, so the keybinding has to be changed to something else. For German keyboards, VS Code has solved it by using Ctrl+# on Windows and Cmd+Shift+7 on Mac (Shift+7 produces the character / on German keyboards).

VS Code is an Electron application and uses the native-keymap package to determine the keyboard layout (see microsoft/vscode#713). Since that is an OS-native library, it cannot be used in a web application.

Rationale

If we would have a web API that allows us

...to convert from a code value + modifiers into the key value that would be generated by the current locale and layout with those modifiers present,

a web IDE could identify the complete keyboard layout and thus adapt the configured keybindings to the user's keyboard. This is exactly the kind of information that is provided by the native-keymap package for VS Code.

The inverse mapping, that is the ability

...to convert from a key value into the set of code values required to generate that value, given the current locale and layout,

would be useful, too, in that context, though it could also be derived from the mapping above and thus does not add any new information.

In case a web IDE would like to display the detected keyboard layout to the user,

identifying a keyboard layout or locale

would be necessary.

In summary, web IDEs need some means to query the mappings from code values to key values for all combinations of the Shift and Alt modifiers. The other extensions can be useful, but can be regarded as optional.

Proposal

The current proposal of the Keyboard interface could be extended so the getLayoutMap() function takes an optional first argument with the structure { shiftKey: boolean, altKey: boolean }. If such an argument is given, it returns the KeyboardLayoutMap that considers those modifiers, otherwise it behaves as described in the current proposal.

Background Information

As a committer of Eclipse Theia, my main motivation is to improve support for international keyboard layouts in that web IDE. Theia is currently used in

• Gitpod (TypeFox)
• Yangster (Ericsson)
• Eclipse Che (Red Hat)
• Microclimate (IBM)
• Cloud Shell (Google)
• Mbed Studio (ARM)

Of course other web IDEs could also greatly benefit from better keyboard layout support in the web API, for example

• Cloud9
• Codeanywhere
• CodePen
• Coder
• CodeSandbox
• Janitor
• JS Fiddle
• JupyterLab
• Koding
• Particle
• repl.it
• StackBlitz

Web IDEs are emerging very quickly, which shows that there is a high demand for them. However, the web APIs are not prepared for this kind of user scenario when it comes to keyboard events.

Privacy Considerations

The impact on users' privacy is basically the same as for the currently proposed API (see #8). Of course adding modifiers to the keyboard layout information increases the fingerprint in terms of the amount of data, but I assume it does not change much about the identifiability of a certain keyboard because most keyboards should be already fully identified by their character layout without modifiers. This assumption needs to be verified, though.

How to Proceed?

I am submitting this issue because the current keyboard-map proposal is already quite close to the needs of web IDEs. The missing piece are the modifier keys.

If you think what I describe would be better put into a separate draft, I'm fine with that. My intention is to start a discussion on this so eventually we can improve the user experience for web IDEs.

[Keavon]

Our use case is a 2D raster and vector graphics editor application called Graphite which will support desktop and web. For the web version, we are running into limitations of the web API because of the lack of this proposed feature.

I implemented some heuristics to guess based on a combination of event.key and event.code values but it isn't sufficient. We need the ability to query what key will come from hitting each key on the keyboard with each modifier key pressed (including the Option key on Macs, which acts as an alternate symbol key). That is necessary for both input mapping (to detect what relevant symbol the user actually is pressing regardless of modifier keys) and display of the keyboard shortcuts in the UI.

Without this proposed addition to the spec, it is essentially impossible to properly build localized web apps with nontrivial keyboard shortcuts that work outside the US.

For example, a user pressing AltW (OptionW on a Mac) will end up typing an event.code of KeyW and an event.key of Σ. On the US layout, it's fine to use the code, but on a French layout, KeyW should be a Z not a W! On a French layout, there is no way to determine which key the user pressed while holding the Option modifier. This is also true with the Shift modifier for any key that doesn't have a lowercase and uppercase version of the same symbol (where .toLowerCase() would be handy), like the number row that puts numbers and symbols on the same keys. It's impossible to know where a certain symbol is, and whether it's behind a modifier or not, for every keyboard without the proposed API addition.

Our graphics editor application is one example amongst many of a professional-oriented web app that needs full control over keyboard shortcuts. Other examples can include Figma, a hypothetical web app port of Blender and other similar tools... basically all digital content creation tools. That's in addition to the use case presented in the proposal: IDEs.