Text input rendering previous keypress on Windows

[elahn]

In this video, I clicked on the input, typed: mozilla.org[enter][enter]. Nothing appeared in the input until the 2nd key, 'o' was pressed and with each keypress, the state as of the previous keypress was rendered.

Windows 10 Home x64 with anniversary update on real hardware.

[jonathandturner]

In case anyone is trying to work on this issue, I got this far:

• components/script/textinput.rs has handle_keydown, which receives a event: &KeyboardEvent
• When you press a key, this event seems to have the correct keycode field, but the printable from the previous key (initially this is none).
• handle_keydown uses the printable field to determine what to do, so we end up always being one behind

[jonathandturner]

A bit more info:

• Issue seems to be in glutin. In ports/glutin/window.rs it assumes that Event::ReceivedCharacter will happen before Event::KeyboardInput.
• This ordering is not what I see in practice, where KeyboardInput comes first.
• The assumption happens with the self.pending_key_event_char not being set properly

After this happens, we're always one behind.

The fix seems to be to remove this assumption or ensure that glutin orders in the same way it assumes the events will come.

[jonathandturner]

cc @tomaka, who might know why the keyboard events work like this ^

[jdm]

I believe the ReceivedCharacter/KeyboardInput ordering is correct in Glutin on Mac and Linux.

[codec-abc]

I start to look at this issue too and put some println! ports/glutin/window.rs and found the following : the events in the functions handle_window_event are received in this order :

• Event::KeyboardInput (probably key down)
• Event::ReceivedCharacter
• Event::KeyboardInput (probably key up)

The events appear to be reliably ordered.

But I don't received Event::KeyboardInput for some keys but only Event::ReceivedCharacte like / for example. I am using a azerty layout on a laptop which may make things more complicated. Since the code in the ports/glutin/window.rs seems to be common for Windows, Linux and Mac, can someone please tell if they encounter odd keyboard behavior in the others platform ?

[jonathandturner]

@codec-abc - that's the order I was seeing too. Unfortunately, it looks like KeyboardInput checks for the character that ReceivedCharacter puts in pending_key_event_char. This means that we can't use it to type.

You could imagine using it for something like sending the character during a key up, but that'd different than people would expect.

[jonathandturner]

It seems like we need translate the key to a printable character ourselves, rather than waiting for ReceivedCharacter. Then we could just ignore the ReceivedCharacter event.

TBH that these are separate events are confusing to me. Why not just have a single keydown and a single keyup? Having other states in the state machine just sounds more complicated.

[jdm]

My understanding is that it's common for UI frameworks to include separate events for "printable text you might care about" and "raw keyboard input". For example, GLFW does this too.

[jonathandturner]

@jdm - yeah, I think that's where glutin gets it (since iirc, it's based on GLFW).

I did a quick look around, and Windows, SDL, macOS, qt, JS, and gtk all seem to be keydown/keyup with some also having a separate event for key holding. Granted, that was all of 5 minutes of me skimming documentation :).

Inside the key down event, things like JS allow you to get the printable char in case you want it.

[jonathandturner]

Hit send too quickly. Anywho, my point with the last line is that this would actually make the fix pretty trivial if glutin worked that way instead of having a separate event for printable char.

[jdm]

#4144 has the background on us figuring this stuff out and then never following through on modifying glutin in ways that made sense.

[codec-abc]

I have tried to understand the issue further. I agree with what @jonathandturner wrote earlier : "In ports/glutin/window.rs it assumes that Event::ReceivedCharacter will happen before Event::KeyboardInput". Reading some documentation about the Windows API it confirms that for key that can produce characters the order of events are :
Event::KeyboardInput - KeyDown
Event::ReceivedCharacter
Event::KeyboardInput - KeyUp

As a side note, I think we need to keep the 2 types of events. ReceivedCharacter are needed for input fields and KeyboardInput may be needed by javascript games for instance. More information here.

Back to the original subject. The logic of the Windows API does not seems to align with the one used by Servo. So, I think we need to see be sure in which order the events come in the other platforms (Linux & Mac) as they probably come in a different order. Then, we probably need to reorder them. It might be a good idea to look at implementation of cross platform libraries such as SDL, Qt, and more to see how they handle the problem.

EDIT : I think that they other issues in glutin and/or winit since they are keys that should create ReceivedCharacter that don't but I consider this a different issue.

[jonathandturner]

I did a quick test.

macOS gives the events in this order:

• ReceivedCharacter: 'h'
• KeyboardInput: Pressed Some('h') H
• KeyboardInput: Released Some('h') H

Windows:

• KeyboardInput: Pressed None H
• ReceivedCharacter: 'h'
• KeyboardInput: Released None H

I'm guessing it's not feasible to try to reorder the events to make them the same because we won't always know what to wait for.

Going back to earlier thoughts, it seems we're trying to mix too much together, and we're linking events together we shouldn't be linking.

Can we perhaps use just the ReceivedCharacter event when we're in an input box?

[codec-abc]

I agree with you on the last part. ReceivedCharacter may be enough for inputs field. Yet, I don't know enough to tell. But I tend to disagree with you and the reordering of events. We cannot do it on Windows since a KeyboardInput won't necessary lead to a ReceivedCharacter but the other way should work. I cannot see how we could have a ReceivedCharacter on macOS (or Linux) without a KeyboardInput just after. Moreover, the fact that the events does not came in the same order could theoretically leak to Js and it seems weird that the ordering is platform dependent. Testing this demo from this article give me the same events in the same order in macOs and Windows.

EDIT: I tested the link on macOs with Servo and it complies with the other browsers, meaning that the keydown event is triggered before the input event.

[jonathandturner]

Huh, so are you saying that we're doing a reordering for macOS and Linux somewhere else in the code?

[codec-abc]

I don't know how the Js events are created. If they are simply mapped from the keyboard ones then yes there is probably some reordering elsewhere in the code. But you can't take my word for it since I only look at the input and output and got no idea what happen in between.

[jonathandturner]

Cool demo btw, it seems like there may be other keyboard event issues potentially (though maybe related to this issue we've been tracking down)

On Firefox (on my Mac):

On Servo (on Windows):

So we're seeing one onkeydown where we should be seeing an oninput, assuming I'm understanding the test correctly.

[codec-abc]

The test is quite basic, it registers 2 events handlers that put textual description of the event below in the input field. The only particularity is that the oninput event handler remove the onkeydown event handler the first time it is triggered. On Windows since the events are in a different order we do not get the oninput event for the first character.

By the way, after looking more at the code I think I understand it a bit further. As I understand it, the whole thing happen in the function handle_window_event in the file ports\glutin\windows.rs. I think what we try to achieve in the code is to send events only for the keyboard but with an associated character if the key pressed has created one. In a way, we merge two event streams. Since on MacOs and Linux the character come before the key event we store it in pending_key_event_char and sent it with the key event that come right after. I think it is not possible to get the same behavior for Windows or a least not in a simple fashion (ie, not re-implementing an input method directly in Servo). If we do it in a reverse way (Pushing event as soon as they come but instead of storing a character we store the last virtual key code event instead) we would have something similar but with duplicate events. Typically if we do this for Windows if we type the character a we would get:

1. Keyboard event / VirtualKeyCode a / Associated Character : None
2. Keyboard event / VirtualKeyCode a / Associated Character : a

While on Linux & macOs we would only have

1. Keyboard event / VirtualKeyCode a / Associated Character : a

So I think the only solution is to not merge event streams or at least at a later point in the code. IMHO, the first thing to do is see what happen in we send events for both CharacterInput & KeyboardInput in MacOs & Linux. It would probably force us to change code elsewhere since the previous test show we get 2 events on the Js side from a single WindowEvent::KeyEvent. In fact I just wonder if the sole purpose of the pending_key_event_char field is to align macOs & Linux with Js, doing in the same time some kind of hidden reordering just like I said in a previous message.

[codec-abc]

Update: I have tested my previous idea ,ie keeping the last key pressed when receiving a keyboard KeyboardInput to send it later on when receiving a ReceivedCharacter. It does make the HTMLInputElement behave correctly on Windows. I am no more one character behind. Still, when testing the previous demo it shows that in the Js side we don't get the correct events.
At this point I see 2 solutions. The first is to send one more event so instead of having this (for Linux & MacOs) :

1. Keydown event with optionally an associated character.
2. Keyup event with no associated character.

we will get:

1. Keydown event no associated character event there is a value in pending_key_event_char.
2. Optionally Keydown event with a mandatory associated character in case the pending_key_event_char != None
3. Keyup event with no associated character.

But it seems weird to me. The other solution is to reorder the events on MacOs and Linux and not correlate the key events to the character input events.

I will stop working on this issue until it is decided how we should tackle this issue.

[jonathandturner]

@codec-abc - my priority is to get HTMLInputElement working so that we can get the nightly out there.

Getting the eventing in the right order is going to take more work. I noticed that, too.

Can you post your PR so we can review? We can talk with some of the other servo folks on the PR, but I think we should do this in stages:

• Land something that fixes the input
• Get nightly out
• File separate issue for eventing

I suspect we'll end up having to rewrite a chunk of glutin (or possibly use another eventing system), but this should at least unblock us.

[codec-abc]

@jonathandturner OK. PR is here.

[AshleyScirra]

Is this meant to be fixed in the latest Windows Servo build? If I type "https://www.scirra.com" in the address bar, I get "htt/www.scirra.co".

[codec-abc]

It seems that the PR that should fix this did not make it in the last nightly and there is another one in the way because of some error while cleaning the code. So I think the next nightly should have improvements concerning this bug.

[codec-abc]

I had very little time for a few days but I managed to do a simple test of Glutin, SDL2 and GLFW on Windows and Linux for the keyboard ordering. My test was to open a window and type Hello (with a newline at end). My findings is that SDL2 and GLFW behave the same ways on Windows and Linux while Glutin behave differently depending of the OS. Moreover, SDL2 and GLFW follows this scheme (for a simple key that will trigger a character):

1. Keydown event
2. Character input
3. Keyup event

With Glutin it is the same on Windows but on Linux (and probably Mac OS) it does the following:

1. Character input
2. Keydown event
3. Keyup event

If you want to look at the code used my tests repositories are here: Glutin, SDL2 and GLFW, but there is really nothing to look at.

At this point I think someone should open a bug on Glutin to get it fixed but that means that Servo (and the other projects that use Glutin) may have to change some code at the next upgrade.

@jdm @jonathandturner Since you followed the issue quite closely, what are you thoughts on the matter?

[jdm]

Changing the event order in Glutin on Linux and Mac is straightforward: https://github.com/servo/glutin/blob/4c31b8b16f703766d4aa8d45762dab76f1684e86/src/api/cocoa/mod.rs#L1083-L1088 and https://github.com/servo/glutin/blob/4c31b8b16f703766d4aa8d45762dab76f1684e86/src/api/x11/input.rs#L154-L168 are fully in control of the event ordering, unlike the Windows implementation. The real solution is redesigning Servo's keyboard input implementation to interact better with this model of split event kinds, though.

[codec-abc]

Thanks for the answer. Just to be sure, when you say "real" you mean difficult right? Because from my pov, there are still keyboard issues on Windows (but they are less visible thanks to the PR). Moreover, I think because Glutin aim to be a rust cross platform alternative to GLFW it should change its behavior to match GLFW. To conclude, I think both Servo and Glutin need to be updated in this regard. Anyway, I opened an issue in Glutin repository here.

[jdm]

What I meant by my comment was that changing the event order in glutin will only allow us to remove the per-platform code from the previous fix in Servo. It will not affect the fundamental limitations in the current model for processing keyboard input in Servo, which requires much more thought and effort.

[codec-abc]

Just leaving this as a note. Glutin code has been updated - for quite a while actually - and now keyboard events are ordered in the same way no matter the underlying platform.