Ideas & Thoughts on Design for Keyboard Input

[ivakam]

As of 0.13, Bevy primarily uses physical keys when reading input via KeyCode (which is the recommended method since ButtonInput<Key> isn't fully supported and is a bit unergonomic due to the Character variant). This makes things very rough for anyone not using a US QWERTY keyboard.

As an example, I use a Nordic layout and I have no idea which KeyCode to use to map my + key. I could, of course, look it up, but I should not have to do that. There are also cases where layouts have different amounts of keys, or where it's not clear how to map the physical positions of the keys due to variations in how the layout is staggered, the size of keys, etc.

I am aware this change was brought on by changes to winit, but ideally Bevy would have its own design spec in terms of keyboard input, and how winit is used is an implementation detail. The goal of this thread is to gather ideas and thoughts on what an RFC for a keyboard input spec would look like. I'm no expert on this by any means, so anyone who has experience with this is very much encouraged to share any ideas you might have.

These are some of the points I think are important to address:

• The primary way of reading keyboard input should be in terms of logical keys. That is, the "final" key code after things like layout, OS-level remapping, layers, shift/alt/... modifiers, etc. are considered. This is the only way that things end up making sense for people on non-standard layouts (including international layouts which is a lot of people).
• I think leaving the door open for people to toy with physical keys is not a bad idea, but I personally can't really see the use-case where using physical keys is preferred over logical. I'd love some thoughts on this point!
• I think that for now, things like "action maps" and so on should not be included in this RFC. This is strictly about how keyboard input is handled in terms of key codes.
• Common edge cases should be given first-class support. One big one is capital letters. For most games, A and a are not meaningfully different, and having them be different if the user is holding shift (common thing for games) is unnecessarily annoying for the developer to deal with. I think modifying is_just_pressed and friends to include something like a ignore_capitals parameter is a decent compromise, but there may be more to this than meets the eye. I'd love more thoughts on this!

[keis]

The WASD-movement scheme is a good example of where physical keys makes sense. These keys are not picked because the characters they represent but rather their position on the keyboard. On my dvorak layout that maps to the characters ",aoe".

Same goes for "grid hotkeys" in games like starcraft where they keys are selected to form a grid that maps to the order in the UI. Contrast with the regular mnemonic keybinds where the "Build" action would be triggered by the B-key.

[ivakam]

The WASD-movement scheme is a good example of where physical keys makes sense. These keys are not picked because the characters they represent but rather their position on the keyboard. On my dvorak layout that maps to the characters ",aoe".

Same goes for "grid hotkeys" in games like starcraft where they keys are selected to form a grid that maps to the order in the UI. Contrast with the regular mnemonic keybinds where the "Build" action would be triggered by the B-key.

I'll copy my response to a similar thought, thank you for replying!

I appreciate the idea of having controls be layout-agnostic by using physical locations, but there's a reason most games don't do this, right? It's very confusing as a user when you press a key and it uses the binding you set for some different seemingly unrelated key because it just happens to map that way from your layout to US QWERTY. Basically, if the key label in-game or even in the options menu says W, that action should happen when I press whatever key produces a W, right?

And if you're doing some sort of thing where you change the way the controls are shown based on the selected and/or detected layout, you could just use action mapping for that layout to begin with since you've already done the hard work of mapping one layout to another.

[keis]

I agree that's confusing and if games where to do the mapping and show the right key in the UI that would be great.

But for me it's a bigger benefit to having normal conventions just work without having to open the options menu and rebind my keys or temporarily change the layout on the operating system.

It's also important to note that key presses and character input are not always one-to-one. E.g pressing the key labeled ' might produce character input on some layouts but on some it's a dead key that awaits another press to produce a single e.g. é character. What is the expectation when game asks you to press ' in isolation?

[ivakam]

Yeah that makes sense. I think the ideal is to have the game provide a few "profiles" for common layouts. To me, that's the sanest solution (albeit one requiring a little extra work from the dev but that's just accessibility in general for you). Again, if you have the tech to show the correct key label for some given layout (so the key you're telling the player to press in the tutorial or whatever matches the key on their keyboard when using physical keys), you've already done most of the heavy lifting in terms of just making an input profile for that layout.

I am indeed aware that dead keys can cause trouble (´ is a common source of weirdness on my layout). I think that's one where I would prefer to just manually rebind any keys causing trouble, but I suppose it's up to preference.

As a takeaway I think leaving the door open for physical key mappings is fine, but my gut tells me the trade-off between "it's easy for the dev" vs. "very strange edge cases" is not worth it for projects of any reasonable size. If both are made viable in the design spec, I think it is an absolute must that the documentation is VERY clear on the trade-offs for each approach, and that working with raw key inputs without at least some degree of action-mapping in-between often leads to poor user experience (that's a bit of another topic, related as it is)

[NthTensor]

I'd like to see more justification before we devote effort to a custom keyboard layer/spec. The new winit api is based closely on the W3C spec which, though not perfect, is well specified, well documented, and has excellent cross platform support.

I also don't see the problem with using physical keys by default, so long as logical keys are an option. Most games use keys like "wasd" where the position is more important. It's good default behavior for new users.

If logical keys are hard to use, that's something I'd be interested in hearing more about. We should fix that. But currently I don't see why that would require implementing a custom keyboard layer.

[ivakam]

Yeah, that seems more or less fair enough. I do take some issue with the idea that "most games just use wasd so physical is fine" however. Yes, most games do indeed use wasd where physical is fine, but few games only use wasd with no other hotkeys Specifically, most games have a mix of hotkeys where for some the physical position is important, and for some the semantic/logical key is more important (think G for grenade or I for inventory and so on).

Seeing movement bound to wasd and then having that map to the corresponding keys on my AZERTY keyboard is fine; good, even. However, seeing a key bound to ' (or even something less weird like + for zoom) and trying to figure out what the heck I press on my non-US QWERTY keyboard is not good. It's not a good experience for the developer trying to figure out what KeyCode variant I should use to map + (or any key where the actual character is of primary concern), and it is also not a good experience for the user to press a key and have it show up as something else in the settings screen (or wherever you configure the keybinds) because it is mapped to the physical US-QWERTY key. I am aware that many very large and high-budget games do this (Dota 2 is an example I know right away), and I think it's a terrible user experience there too.

[T-R-O-L-L]

A side note: the difference between the logical and the physical mappings only exists for the international layouts with the Latin-based alphabets. Whereas the keyboards for Greek, Cyrillic, Arabic and other non-Latin scripts almost always (to the point where not having one is a rare edge-case) physically include the English-US QWERTY layout. And for the users of these layouts it only makes sense if the physical keybinds are used, because else they have to switch to the English layout. Imagine then having a chat or some other form of a text input in a game: the users of these keyboards would have to switch back and forth from the English-US layout (to use the game) to their language layout (to write the messages/other text).

Moreover: the existing tradition is that the keys are not even rendered in the target language, but are labelled according to the English-US QWERTY layout. That is, if a prompt in Russian asks to press A, the user most commonly would press the English-US QWERTY A key (that is the Ф key in the Russian ЙЦУКЕН layout), rather than the Russian ЙЦУКЕН А key (which is F in English-US QWERTY).

[ivakam]

That's good insight! That definitely makes the case for keeping physical keys much stronger. This shifts the important part to ensuring that both physical and logical keys are well-supported (with logical keys being the lacking part as of writing this, physical keys more or less just follow winit).

[maniwani]

I think it's really important to distinguish between the developer and player concerns.

The developer wants any default control schemes to be layout-agnostic. So if a game has a set of default key bindings, what's important is the way those keys are physically laid out, not the characters they emit when pressed.¹ The developer can't know all layouts the player could have but they can be sure that key positions are mostly consistent.

It wouldn't make any sense to define a set of default key bindings for every possible logical layout (that are just going to physically match the QWERTY one) when keyboards physically have almost no variation. Position just happens to be expressed as QWERTY key position because it's the most common printing. It's just a convention. (edit: More intuitive than just numbering all the keys.)

There are also cases where layouts have different amounts of keys, or where it's not clear how to map the physical positions of the keys due to variations in how the layout is staggered, the size of keys, etc.

Keyboards really don't stray far from this layout. ANSI/ISO differences are minute, and a developer really only has to account for full-size keyboards and 80% (no numpad) keyboards. Most games don't even have that many actions to bind.

The player... also wants control schemes to be layout-agnostic. Well, they want them to just work. They don't care if the app accomplishes it by using physical keys or using layout-defined keys and a default for each layout. They also want the key displayed on screen to be the one they expected to see.

I think using physical key is the more robust way to accomplish these goals since it means the player can their change keyboard layout in-game without worry. If the player wants to see the physical key, the app doesn't have to do anything. If they want to see the layout-defined character, the app can take the physical key, turn it into an OS scancode, and ask the OS for the character it currently represents (Windows, macOS, Linux).

Like that, the developer could make one default control scheme and the player wouldn't have to mentally translate. Everyone's happy. I think that might be the solution you're looking for.

I think that for now, things like "action maps" and so on should not be included in this RFC.

(edit) I don't think actions should be totally ignored. If an app's code only sees actions like Jump, then it really becomes apparent that this boils down to "developer wants to provide layout-agnostic defaults" and "player wants to see the right icons".

I think it's also worth mentioning that @alice-i-cecile has talked about wanting to upstream leafwing-input-manager as an engine plugin several times.

---

The primary way of reading keyboard input should be in terms of logical keys. That is, the "final" key code after things like layout, OS-level remapping, layers, shift/alt/... modifiers, etc. are considered. This is the only way that things end up making sense for people on non-standard layouts (including international layouts which is a lot of people).

It's not that simple. Logical key events aren't just Latin characters. A logical key event could be almost any Unicode character as well as a non-character (dead) key. I doubt we want to deal with that or case-sensitivity or Unicode normalization.

---

I think leaving the door open for people to toy with physical keys is not a bad idea, but I personally can't really see the use-case where using physical keys is preferred over logical.

I'm curious how you came to these conclusions. The broader game development ecosystem is full of examples of preferring physical keys, whether or not the game's UI displays the layout-defined character.

Unity says

The location of individual keys is agnostic to keyboard layout. This means that, for example, the A key is always the key to the right of the Caps Lock key, regardless of where the currently active keyboard layout places the key that generates an "a" character, or whether or not the layout doesn't have a key assigned to that character.

To query which (if any) character is generated by a given key, use the KeyControl's displayName property. The value of this property changes automatically when the OS changes the keyboard layout.

SDL2 says (emphasis mine)

SDL_Scancode values are used to represent the physical location of a keyboard key on the keyboard.

SDL_Keycode values are mapped to the current layout of the keyboard and correlate to an SDL_Scancode.

Which one to use is left to the application: scancodes are better suited to situations where controls are layout-dependent (e.g. the "WASD" keys as left-handed arrow keys), whereas keycodes are better suited to situations where controls are character-dependent (e.g. the "I" key for Inventory).

Steam Input says (emphasis mine)

Available bindings are all the keys on a standard keyboard. Note that the inputs here are sent by scancode, so although the display here is a standard USA English keyboard, the input that is actually sent depends on your system language/keyboard layout. So inputs like "WASD" on the configurator layout will actually map to whatever keys are in the corresponding positions in your system layout.

Godot 4.0 added physical keys to their API to fix long-standing issues with different keyboard layouts breaking in-game bindings.

So the case for preferring physical key location is very well-supported.²

Footnotes

1. This is why "jump" is A on an Xbox controller and X on a Playstation controller. The position / muscle memory is what matters. ↩

2. Both Unity and Godot now recommend using actions instead of specific inputs. Steam Input was also created specifically to achieve that. ↩

[maniwani]

Really? Is that a known winit bug / is there an issue tracking that?

I'd actually support having our key event be the full KeyEvent from winit that carries both the physical and logical key.

Also, have we already done this?

[alice-i-cecile]

We have already done that, in KeyboardInput.

Asking / looking about the text entry issues.

[rparrett]

Really? Is that a known winit bug / is there an issue tracking that?

rust-windowing/winit#3381 is the main one I experienced. The issue specifically talks about backspace, but there may be other platform inconsistencies with keys like enter, left/right arrow, delete, etc.

So if you want to do backspace in a cross-platform way, you need to use KeyboardInput.

ReceivedCharacter and KeyboardInput are two separate event streams without timing information, so if you want your backspace to arrive in the right order, you should use only KeyboardInput.

So ReceivedCharacter is pretty useless for me at the moment. There are probably standards or opinions about what non-printable characters should be included in a "received character" event or whatever too, but I don't know anything about that.

Honestly I am also still very confused about the different sources of "text content for a key event" in winit. But whatever we use for KeyboardInput::logical_key seems to work for me.

I wrote about some of this stuff pre-0.13 here: #11297

[maniwani]

Oh wow. Thanks for the context. So having KeyboardInput events carry the full info like winit's KeyEvent was a necessary move then.

Yeah, seems like we should deprecate/remove our ReceivedCharacter event then. It probably is useless now. It was in earlier versions of winit but was removed in the keyboard rework.

[NthTensor]

Mia culpa, I didn't know you could access logical keys through KeyboardInput. I think it's great having that be the main source of truth with both physical and logical key presses, and then ButtonInput as a friendly shortcut for physical buttons. That's 'first class support' enough for me and I think addresses OP's concerns pretty conclusively.

I know see what you mean about only needing the physical -> logical map. That sounds neat.