Multi-key hotkeys are buggy.

[pixelzoom]

I have 2 hotkeys, defined like this:

keyboardDragListener.hotkeys = [

  // J+O+1 - jump to object 1
  {
    keys: [ KeyboardUtils.KEY_J, KeyboardUtils.KEY_O, KeyboardUtils.KEY_1 ],
    callback: () => { console.log( 'J+O+1' ); }
  },

  // J+O+2 - jump to object 2
  {
    keys: [ KeyboardUtils.KEY_J, KeyboardUtils.KEY_O, KeyboardUtils.KEY_2 ],
    callback: () => { console.log( 'J+O+2' ); }
  }
];

What I’m seeing is that the callback for J+O+1 is firing as soon as I type J+O, and I can’t get J+O+2 to fire at all. Have hotkeys been tested with > 2 keys?

I'm considering using hotkeys like this in Geometric Optics, where we have multiple optical Objects (O) and Images (I) that we'd like to jump to.

[pixelzoom]

@jessegreenberg confirmed in Slack that this is buggy -- it should work but does not.

[jessegreenberg]

A number of issues here... There is nothing enforcing that all hotkeys are actually down. And the method of making sure that all keys are pressed in order just looks wrong. Looks like updateHotkeys works for 1 or 2 keys, but thats about it.

[pixelzoom]

In Slack, @terracoda said:

The design pattern for jump hotkeys has been you have to hold down the J key and then press the other key while still holding down the J key. You can see the hot keys fully described using the A11y View in a sim with Hot Keys.

The key presses are not meant to be sequential if that's the issue 🙂

[pixelzoom]

Is the design pattern for jump hotkeys limited to J + 1 key? Did you consider multiple keys? For example, in Geometric Optics, we have 2 “objects” and 2 “images” that can be jumped to. We can certainly create a 2-key sequence for each. But there’s something more memorable about J+O+1 and J+O+2 to jump to “object 1” vs “object 2" respectively. (It’s also easier to describe in keyboard help, a secondary consideration.)

More generally… If I had a sim with N objects to jump to, a pattern like J+O+{number} may be the only viable pattern, since it may be impossible to come up with (or provide keyboard help for!) a unique J+{key} for each object.

As for the implementation…. The hotkeys API is general enough to support key sequences other than “jump”, with arbitrary numbers of keys. It looks like a nice API, and should serve PhET well in the future. But the implementation of the API is definitely broken, as @jessegreenberg confirmed in #1355 (comment).

[jessegreenberg]

I think we should support any number of keys for hotkeys. I had also assumed that the order keys are pressed does matter and all keys have to be held down. That is what is intended in the current implementation but its all broken for any combination of more than two keys.

[pixelzoom]

I agree, order should matter.

[zepumph]

Does order matter generally, or should it just be that order matters for the first n-1 keys as a bunch coming before the nth. I'm thinking about most key commands I use

• Ctrl + Shift + T can just as easily be Shift + Ctrl + T (reopen closed tab on Chrome)
• In webstorm I bind many keystrokes, and they are always in groups of two, for example, first keystroke is Alt + D and the next one is E. I also have some that split like Alt + D into Alt + E.

Given this, should it be implemented such that J + O + 1 (ordered) behaves the same way as O + J + 1, where the important thins are that J and O are down when 1 is pressed?

I hope I'm here improving user experience, and not just muddying the discussion-water. Feel free to disregard my thoughts outright and don't feel like we need a lengthy discussion unless you feel strongly about it.

[pixelzoom]

@terracoda said:

The design pattern for jump hotkeys has been you have to hold down the J key and then press the other key while still holding down the J key. ...

@zepumph said:

... Given this, should it be implemented such that J + O + 1 (ordered) behaves the same way as O + J + 1, where the important thins are that J and O are down when 1 is pressed? ...

I'm wondering about this requirement to hold keys down. Isn't that potentially difficult for someone who has a motor-skills issues?

I also see that KeyboardHelpSection createJumpKeyRow only takes one additional character, so the current implementation does not support something like J+O+1. See #1358

[terracoda]

@pixelzoom, if I am understanding you correctly, we do indeed have multiple destinations for an object. In BASE, you can:

• Jump a balloon to the Wall - J+W
• Jump a balloon to near the Sweater - J+S
• Jump s balloon to the Center of the room or play area - J + C

I think from an inclusive design perspective it is good to think about how many keys you are requiring a learner to press down at the same time. Two keys may be difficult for some, and 3 keys would be even harder. Not impossible in either case, but something to keep in mind.

[terracoda]

@pixelzoom, yes it is difficult, but with a screen reader enabled, individual keys are already accounted for. You need a paired set to get around screen reader commands.

Just adding... with the Jaws screen reader individual key presses are already assigned to other functions. So J+W can go to the wall, but J then W might execute 2 JAWS commands.

[pixelzoom]

@terracoda said:

@pixelzoom, if I am understanding you correctly, we do indeed have multiple destinations for an object. In BASE, you can: ...

That's not the situation in Geometric Optics. We have multiple objects, not multuple locations for an object. When a ruler is selected, a hotkey moves (jumps) the ruler to the location of one of N object. Those objects are labeled as Object 1, Object 2, Image 1, Image 2 in the UI. Hence the desire to have J+O+1, etc.

[pixelzoom]

Discussed with @arouinfar.

Here's what we currently have in Geometric Optics, from the Keyboard Help dialog. We can made do with this, but it's not at all easy to remember.

Here's what we'd like to have:

[zepumph]

I'm wondering about this requirement to hold keys down. Isn't that potentially difficult for someone who has a motor-skills issues?

It seems like we should design with this in mind. Wouldn't it be best to have hot keys that are only 2 keys instead of 3?

[terracoda]

What if there was a jump order for the ruler locations and successive jumps, J+R cycled through the locations and Shift+J+R cycled back through the locations.

I know nothing about this sim or its ruler, so this is a shot in the dark.

A learner puts focus on the ruler, grabs it, then uses the J+R short cut to jump the ruler to each location in succession. It would kind of be like using the Tab key.

[pixelzoom]

Nice idea, might work for Geometric Optics. But (in general) if you have a couple of dozen things to cycle through, that's going to be mighty frustrating.

I'm OK with whatever the designers and a11y team decide to do -- in general, and for Geometric Optics. So I'm going to add @arouinfar to assignees.

[terracoda]

It is just an idea. I don’t know the sim or how many things the ruler needs to measure. This sounds like an interesting design challenge.

[arouinfar]

I really like @terracoda's idea to cycle through the items. In Geometric Optics there will be 2-5 items visible on screen at any given time. If users are able to cycle in either direction, there would be at most 2 jumps to get to the desired location. Worst case, it would take 4 jumps if traveling in one direction. Not sure what the best key combination would be, but maybe J+I to jump between items in the play area?

I think this strategy could also generalize to more complicated situations. For example, in CCK we could use this method to jump the voltmeter probes to the vertices between circuit elements (or even jump between batteries, resistors, etc. separately). There, it wouldn't be feasible to do something like J+B+1, J+B+2, etc. to jump to a specific battery. There could be 10+ batteries in the play area and they are indexed by creation time so the number isn't going to be clear to the user. There will ultimately still be a lot of key presses to cycle through things, but for users with motor control issues, the repetitive key presses are probably far easier than precisely placing a probe with the arrow/WASD keys.

[pixelzoom]

Moving the part of this issue related to Geometric Optics to phetsims/geometric-optics#310.

Multi-key hotkeys still need to be supported in general, and are currently broken.

[terracoda]

From Slack:

@pixelzoom, our main considerations when we first thought to use hot keys in BASE was that they should be simple--easy to execute and memorable--and not conflict with screen reader commands. Our sims are generally expected to be intuitive to use, even without instructions.

Custom hot keys, however, by their very nature require that we provide instructions. We hope that one read through of the keyboard shortcuts dialog would be enough for most learners to remember the hot keys.

That said, the sims we have worked on thus far have generally been on the simpler side. As a sim increases in complexity, new ideas and new patterns can be considered, designed, and developed.

Most hot keys I have created thus are paired with the J key (it has a home-key notch on standard keyboards), and for English "jump" starts with a "J". The second letter generally has to do with the destination - again, the word is English centric.

Depending on the sim, and the necessary requirements of the interaction, other patterns can be designed. For Faraday's Law, Jon Hung designed an interesting alternative to dragging with the Arrow keys because velocity of the drag really matters in that sim.

In MAL, we used the ALT key instead of the J key since the actions were not about jumping, and I think K is a universal pause command for screen readers. I hope this provides some nice background.

Actually I think J,K,L are common keyboard commands for web-based video players, so that's likely where we got the K and L ideas.

Here's an article I came across when I was working on BASE and my Master's thesis which I found to be very enlightening:
https://sashika.medium.com/j-k-or-how-to-choose-keyboard-shortcuts-for-web-applications-a7c3b7b408ee

[terracoda]

There may be a better place for the hotkey background. Just let me know.

[pixelzoom]

Multi hotkeys are still so buggy that we bailed on them for Geometric Optics, see phetsims/geometric-optics#395. Hotkeys in general seem to be really flakey in master, see #1393. For Geometric Optics, I've easily spent more than 50% of total alt input time on 1 hotkey. So I'm adding the "blocks-publication" label to this issue.

[jessegreenberg]

This is not blocking publication for upcoming sims so Ill remove the label. But it is still important to resolve and was discussed during quarterly planning meeting to be fixed in Q3 2022.

[zepumph]

Likely this will be fixed while working on #1570.

[jessegreenberg]

This has been fixed now that KeyboardDragListener and KeyboardListener are using Hotkey, from #1570 and #1621.