Keyboard Shortcuts Dialog: how to support young learners & novice keyboard users

[terracoda]

In a recent interview with a grade student using John Travoltage, the student tried really hard to use the arrow key to move John's Arm or Foot.
However, they did not realize that they needed to use the Tab key first to move keyboard focus to either of the sliders under the hood.

Question:

• How can we better support young learners and novice keyboard users on how to use the shortcuts?
• Note for younger learner using the keyboard may actually be a nicer experience.

I was thinking a simple sentence at start of dialog might be helpful. Something like:

• “To use shortcuts, press Tab key to move to the sim object (or item)."
• "To use shortcuts, press Tab key until you see (or hear) item highlighted."

Tagging: @emily-phet, @amanda-phet , @arouinfar , @ariel-phet , anyone else?

[terracoda]

People liked: "[Tab] to get started" right under the Keyboard Shortcuts heading.

[arouinfar]

Here's a mockup of the dialog we came up with during today's design meeting.

[terracoda]

I like the mock-up!

In terms of an alternative to "General Navigation", the only thing coming to mind thus far is:

• "New to Keyboard?"

[arouinfar]

@terracoda "New to Keyboard?" could be a good alternative if "General Navigation" becomes an issue in interviews. Another option could be "Getting Started".

[terracoda]

A11y Content for new items would be:

• Use Tab key to get started
• Press buttons with Space key

Note, if folks with accessibility experience read "Press buttons with Space", they will likely say in interviews, you can also use Enter key.

[terracoda]

@arouinfar, thanks for the extra mock-ups. I agree that I think we might want to wait on changing "General Navigation".

[terracoda]

or "Basic Shortcuts" or
"Basic Navigation"

[terracoda]

@arouinfar, sorry I added "Basic Navigation" after posting "Basic Shortcuts", and perhaps you only saw saw the first post.

Do you prefer "Basic Shortcuts"?

[terracoda]

@arouinfar, I personally do not like the repeated "Getting Started" heading.
My preference in order are the following headings:

• Basic Shortcuts
• Basic Navigation
• New to Keyboard?

@arouinfar, @emily-phet, @amanda-phet, do you have preferences?

[arouinfar]

@arouinfar, sorry I added "Basic Navigation" after posting "Basic Shortcuts", and perhaps you only saw saw the first post.

@terracoda I did indeed only see the first post, but I think "Basic Navigation" works really well too! My order of preference would be: Basic Navigation, Basic Shortcuts, New to Keyboard?

[terracoda]

@arouinfar, my only hesitation with "Basic Navigation" is that we are adding in "Press buttons [space]" which is technically not really navigation.

In a slack chat @arouinfar and I considered:

• Basic Navigation
• Basic Controls
• Basic Operations
• Basic Commands

And feel that "Basic Shortcuts" is the least technical and most accurate!

So going with Basic Shortcuts.

[arouinfar]

Here's an updated mockup @terracoda

[terracoda]

@jessegreenberg, I am assigning to you for re-assignment and implementation, and un-asssigning @emily-phet and @amanda-phet.

[terracoda]

@jessegreenberg, of course this is a sample dialog for a sim that contains an obvious slider interaction.

The part that is new is the:
"[Tab] to get started" , "Basic Shortcuts", and "Press buttons [Space]".

These new features can be added to all dialogs, but the example above is an example that does not mention "group navigation". You probably already have a way to handle that sim specific difference.

[terracoda]

@jessegreenberg, I will provide the A11y text version, too, shortly, in case you need it at the same time.

[terracoda]

@jessegreenberg, in case you need it, I put the A11y version in a google doc for safe keeping:
https://docs.google.com/document/d/1QPPT2bnORUJYETbJBO7QWJ3J-ySZNsDP_Mdpfp4ju_0/edit#

[amanda-phet]

My preference is for Basic Navigation, as that seems most accurate to me. These aren't shortcuts, they are the keys used to navigate the sim.

[terracoda]

@amanda-phet, as I mentioned to @arouinfar , my only hesitation with "Basic Navigation" is that we added in "Press buttons [space]" which is technically not really navigation.

And all the operations are keyboard shortcuts of one kind or another.

@amanda-phet, would you like to get feedback from a few more folks? I can live with either, though I think I prefer "Basic Shortcuts".

I'll reassign issue to @emily-phet, @ariel-phet, and @oliver-phet to see if they want to vote on the new subheading to replace, "General Navigation".

• Basic Navigation, or
• Basic Shortcuts

@emily-phet, @ariel-phet, and @oliver-phet, please post your preference.

[zepumph]

All work is done here besides changing the name of the HelpContent from "General Navigation" to "Basic Actions"

I won't be at the next dev meeting. It is entirely possible that there is a protocol for how to handle this, and I just don't know it.

[zepumph]

BTW here is what Friction looks like as of this writing,

[samreid]

we want to change the name of a string

I presume you mean the key name? We historically avoided changing key names, though I am not 100% sure why. Perhaps if we minimize the time between committing a key name change to babel and publishing the sim with the new key name, it wouldn't be too much trouble? Perhaps @jbphet will know more.

In a recent interview with a grade student using John Travoltage, the student tried really hard to use the arrow key to move John's Arm or Foot.
However, they did not realize that they needed to use the Tab key first to move keyboard focus to either of the sliders under the hood.

What if we added code to the simulation that detected arrow key presses and automatically focuses the arm/foot if nothing else had focus? We could apply this strategy in other simulations as well: in Balloons and Static Electricity, if the user presses the arrow keys before pressing tab, then the balloon would automatically get focus. This proposed auto-focus approach would be in addition to the newly designed help dialog, not a replacement for it. Although maybe showing the focus highlight (after using arrow keys) would be enough of a cue to users that they can use tab to navigate.

[jessegreenberg]

What if we added code to the simulation that detected arrow key presses and automatically focuses the arm/foot if nothing else had focus?

This could be nice, I am curious what @emily-phet and @terracoda think of this. The complication will be that we need to make sure that this never happens when a screen reader is in use. As a general feature this may result in design decisions on a sim by sim basis when multiple items are draggable, or for the balloon which needs to be "picked up" with space bar or enter.

[samreid]

The complication will be that we need to make sure that this never happens when a screen reader is in use.

Can you specify what would go wrong if a screen user was in use, the user pressed an arrow key, and it automatically focused and moved the arm/leg?

[emily-phet]

Hey folks - the use of arrow keys to move focus in the way described here is a no-go. We're intentionally aligning with international web standards as far as how interactive elements behave and how you interact with them - as much as we possibly can. It would be incredibly non-standard to have arrow keys move focus in the way described above.
It would also mean that throughout the sim use, it would be ambiguous whether or not the user wants to move focus, or move an item that already has focus. For example, if you open John Travoltage and press an arrow key, and this arrow key press results in moving focus to the foot with subsequent arrow presses moving the location of the foot...how would we know when the user wants to move to the arm? How would the user know how to move focus again? So we'd end up with the same issue, just a little later in the interaction.
I like the idea to try to predict and make some intelligent design decisions to address possible confusions, but this one will run into too many downstream issues to pursue.

[jessegreenberg]

Can you specify what would go wrong if a screen user was in use,

When an AT is in use the arrow keys are used for other navigation purposes. If focus is on the body and the user presses "down arrow" they would expect their device to read the content of the first element in the webpage, not move focus to an interactive element. It may be that the AT will not send a keydown event to the browser in this case, so pressing an arrow key would behave for them as expected. But that will be dependent on device and user settings, we would need to verify.

[samreid]

It would be incredibly non-standard to have arrow keys move focus in the way described above.

I tried launching gmail, then pressing the down arrow and it automatically highlighted the first email. It doesn't seem like it was the same as "focus" though, something different seems to be happening. What do you make of this?

It would also mean that throughout the sim use, it would be ambiguous whether or not the user wants to move focus, or move an item that already has focus.

I was taking the perspective that the arrow key moves the leg, and the focusing is a side effect. The arrow keys cannot be used to otherwise change focus.

How would the user know how to move focus again? So we'd end up with the same issue, just a little later in the interaction.

My hope is once the user sees that something has a focus highlight, they will know they can use tab as they do in other webpages. But we would need to test that hypothesis through user testing.

If focus is on the body and the user presses "down arrow" they would expect their device to read the content of the first element in the webpage, not move focus to an interactive element. It may be that the AT will not send a keydown event to the browser in this case, so pressing an arrow key would behave for them as expected. But that will be dependent on device and user settings, we would need to verify.

I see, thanks for explaining that. I presume we have no way to know if a screen reader is on? What happens in gmail when you have a screen reader on and press down arrow?

[jessegreenberg]

I presume we have no way to know if a screen reader is on? What happens in gmail when you have a screen reader on and press down arrow?

That's right, there is no way to know. I tried out gmail with NVDA and arrow keys, it starts reading from the top, I hear "Link, skip to content", then the "Compose" button.

[emily-phet]

@samreid

I tried launching gmail, then pressing the down arrow and it automatically highlighted the first email. It doesn't seem like it was the same as "focus" though, something different seems to be happening. What do you make of this?"

In John Travoltage - there are two sliders (leg, arm) and two buttons (sound, reset all). It would be non-standard to have arrow keys navigate between sliders and buttons.

My hope is once the user sees that something has a focus highlight, they will know they can use tab as they do in other webpages. But we would need to test that hypothesis through user testing."

This will likely fail in user testing. If users know about pressing tab, they'll do that first (or second) on their own. If users don't know about pressing tab, seeing a focus highlight will likely not address that issue.

[jessegreenberg]

Discussed during developer meeting 8/2/18, instead of changing the key and value, we will need to create a new entry in the strings file with a new key and new value. This has generally been our policy for "changing" translatable strings and their values. The result will be that we have the old key value pair

  "keyboardHelpDialog.generalNavigation": {
    "value": "General Navigation"
  }, 

that will be eventually unused. Maybe in the future we will be able to remove these.

[samreid]

It would be non-standard to have arrow keys navigate between sliders and buttons.

The proposed solution is a way to help students get started using the simulation with the keyboard. It would not use arrow keys to transfer focus from sliders to buttons or vice versa.

This will likely fail in user testing. If users know about pressing tab, they'll do that first (or second) on their own.

My personal experience (N=1) is that when I am shown a web form with a text box already focused, I know I can type in it and press tab to go to the next box. When I am shown a form with nothing focused, I am less likely to try pressing tab to get the ball rolling.

Students that are used to playing web games may be accustomed to using the arrow keys to control the game. I would think that making it so you can press the arrow keys when the simulation begins makes it more accessible to users. It seems like it would have helped the person in the issue description, and other students like him/her. So that's why I'm trying to understand the technical or design constraints that would make this problematic. What if the foot has focus when the simulation starts, but the focus highlight is only shown when the keyboard is used for the first time? Would that solve the problem?

EDIT: I should have mentioned I'm interested to hear @terracoda's opinion on the matter.

[emily-phet]

Sure - happy to see what @terracoda code adds to the discussion, though I don't think we should spend too much time on this.

In general, when talking about access through assistive devices, consistency is key. Having keys do certain behaviors only some of the time breaks with international standards and is not considered good practice. To make the suggested change, we'd need a very good reason to do it - rather than seeing few reasons not to do it.

@terracoda I think it would be useful in particular to hear your thoughts on why we currently require users to press tab (or an equivalent they've chosen in their own setup for AT) before having a focus highlight appear to start with, and thoughts on having an arrow key press essentially used as an indicator someone is trying to initiate alternative input interaction.

[terracoda]

This is a great discussion.
First difference to note is our PDOM is not a custom web application meant to mimic a desktop application's user experience. Thus, using GMail with arrow keys is likely not a good comparison.

As @emily-phet mentioned we have taken a standards-based approach. Our PDOM works like an interactive web page (or interactive web document) rather than a desktop application. We use standard navigation and interaction patterns that are typical of HTML elements (interactive or non-interactive) . We have taken the approach that we only customize interactions when native interactive elements are not possible, i.e., the Book interaction in Friction, the Balloon interaction in Balloons and Static Electricity.

Thus far, this approach has worked very well to support students using AT such as the keyboard (alternative input) or screen reader software. However, for students (actually 1 student) who do not use AT, and who have perhaps never used the keyboard the instructions in the Keyboard Shortcuts dialog were not explicit enough, or not set up in the right order. All the necessary information was there, but the student didn't process the instructions about the Tab key in the John Travoltage dialog.

I think it would be interesting and prudent to see how far we can get to support visual students not familiar with using the keyboard, with the simple addition of "Press Tab to get started". The idea being that once they press tab they will see the focus highlight and see where they are in the sim. Based on where they are (on a button, a checkbox, a slider) they can learn how to interact with that HTML element with the keyboard.

[samreid]

@terracoda thanks for your thoughtful remarks. You mentioned a few times that our goal is not to provide a "desktop application's user experience". Maybe others already understand why, but I do not. It would be great to clarify on that point. Also, is a desktop application user experience diametrically opposed to a standards-based interactive webpage? Why do you think some sites like gmail provide the desktop application user experience?

What do you think about starting the application without a "Press Tab to get started" popup, but if they press any key (other than tab), then it would show the "Press Tab to get started" popup, which would disappear when they do press tab? It seems that would address the case for students like the one at the top of this issue, and we wouldn't have to break from the standards-based webpage ideology. Maybe this is what you were already thinking, but I wanted to be clear about when the message appears.

[emily-phet]

@samreid - let's keep this discussion to the topic title.

Bigger questions - like why we've taken the general approach we have - is great for real-time conversations, but very time consuming (in my opinion) to do in GitHub. @terracoda has a lot of time-sensitive work to be done, so perhaps let's table that component of the issue for some time in the future - perhaps the next time we're having a larger team meeting around long-term a11y plans.

[jbphet]

@samreid said, in reference to changing strings:

Perhaps @jbphet will know more.

If I've followed the issue correctly, it's a common code string that would be changing. Once it is changed and pushed to the common code repo, it would stop appearing to translators for all sims that used the old string until they are republished using the new string. The new string would be untranslated, so the trusted translators would need to re-do the translation for that particular string. Since it is a common code string, it would only need to be done once.

[jessegreenberg]

I added the string "Basic Actions" and started using it in the general section for keyboard navigation. Here is the dialog now, comparing it to the one in #113 (comment):

@zepumph @terracoda is there anything left for this issue?

[terracoda]

That should do it. Thanks everyone. I'll close this one.