Gracefully handle keyboard and mouse interaction

[jessegreenberg]

So far, we have been using default browser behavior with keyboard focus and mouse interaction. When using a mouse after interacting with a keyboard, some browsers blur focus, others do not. For those that do not, the sim is susceptible to multi-input issues. Scenery should manage this so that behavior is platform consistent and we have confidence that interaction with a mouse and a keyboard at the same time does not cause problems.

There are notes in the IDRC meeting on 11/10/2017 on how focus should be handled in these cases:

Focus highlight should go away, but active element should be tracked so that when keyboard interaction is resumed, the next/previous element can be focused when pressing “tab”.

[jessegreenberg]

The above commit is a stab at this. When we receive a down event, we look at the pointer's trail and determine if it has accessible content. If it does, we go back up the trail looking for the first focusable node. This node is stored, and will receive keyboard focus if tab navigation is resumed after this interaction. I gave it a test in Chrome, Safari, and FF. @zepumph would you mind reviewing? In particular, I am wondering if there is a better place for the changes made to Input.downEvent.

[jessegreenberg]

@phet-steele just notified me that in Firefox, clicking in the view when an element has focus and then pressing tab will focus the next focusable element, not the element that had focus on click.

[terracoda]

@jessegreenberg, regarding

pressing tab will focus the next focusable element, not the element that had focus on click

This is the correct default browser behaviour. The Tab key moves focus to the next focusable item.

If focus is placed virtually on an element in the PDOM when there is a input.downevent, pressing the Tab key would by default place the focus highlight on the next focusable item in the sim.

[jessegreenberg]

Just to clarify, that is also true after focus highlight disappears?

[terracoda]

@jessegreenberg, on a typical content-based website where the focus outline is always indicated, a mouse click on an interactive element would not make the focus highlight disappear. General focus (created either by a click, a pointer, a touch event, or the keyboard (e.g., Tab key or screen reader command)) would just indicate where focus actually is when on an interactive element.

Our situation is different in two ways:

1. We have a very strong pink focus highlight to follow and indicate focus changes made with the keyboard, and even sometimes to indicate different stages of keyboard interaction (i.e. grabbed/released).
2. Our actual keyboard focusable interactive items in the PDOM are off to the side, making it impossible to click on them directly with a mouse.

If mouse interactions have no visual focus style associated with them, then it seems we have by design or by accident made mouse interactions remove any previously set visual focus made with a keyboard interaction.

I'm not sure what the issue is:

• Is the issue that FF is working differently than the other browsers?
• Is the issue that switching between mouse and keyboard events is visually not intuitive?
• Or are you asking if visual focus should be placed on the element that had focus when keyboard action is resumed with a Tab key interaction (after a mouse interaction made the focus highlight disappear) rather than placing the visual focus on the next focusable item?

[terracoda]

@jessegreenberg, have we gathered any articles or resources about this issue?
I have been reading a few, so I'm just going to post a few here so we have them. I'm not entirely sure what the best step is for us.

• Jeremy Fields: https://www.viget.com/articles/managing-focus-styles-without-breaking-accessibility
  • first created track focus: https://github.com/ten1seven/track-focus
  • then created what input?: https://github.com/ten1seven/what-input

The above techniques are built on the idea of watching for

the mousedown and keydown events and saves which event happened and when it happened as data attributes. The script then watches for the focusin and focusout events and uses the last device used to add a style to the element receiving focus.

There is some code about managing focus by Rob Dobson that might be useful? Not sure, though.

• https://gist.github.com/robdodson/7aabd62083d0bf66b20c431249379981

There are several articles about not removing focus outlines in general. I'm not sure they apply to interactive sims.

[jessegreenberg]

Thanks @terracoda this is really helpful information. The question I am wondering about is the last item in your list.

Or are you asking if visual focus should be placed on the element that had focus when keyboard action is resumed with a Tab key interaction (after a mouse interaction made the focus highlight disappear) rather than placing the visual focus on the next focusable item?

Right now, when tab navigation resumes we are placing focus on the last item that was clicked with a mouse (if it was focusable), though it is not working yet in FF.

[terracoda]

@jessegreenberg, I think, in our case, it is reasonable to place the focus highlight (upon restarting keyboard navigation) on the last thing that was clicked on with a mouse (if it was focusable). In most of our cases there are not a huge number of things that are focusable, and without magnification all content is generally above the fold and in view.

It might be easier, however, to get it working consistently using the next focusable item as the restarting point.

[zepumph]

@jonathanolson and I found that navigating through items with the keyboard, and then clicking, restarted the tab order. in Area model intro

[zepumph]

@jessegreenberg I was thinking about implementing some sort of function that will keep track any time something is blurred as a result of a non interactive element being focused (like the document body when you click on a background node). Then somehow when you keyboard navigate next, that it would remember your previous placement. Do you know if this is possible? Am I on the right track with the algorithm?

[jessegreenberg]

Thanks, yes, looks like there is some weird stuff goin on with the green balloon right now. Moving to an issue in BASE.

[terracoda]

@jessegreenberg, in a previous comment #713 (comment), I said something incorrect or at least miss-leading. I said:

on a typical content-based website where the focus outline is always indicated, a mouse click on an interactive element would not make the focus highlight disappear.

Keyboard focus style would and does disappear if a user clicks on an element outside of the one that has focus. Clicking on the same interactive element, or on another interactive element would normally result in a switch to an active style. If the design is done well, that active style meshes nicely with the focus style.

If you look at the Keyboard Accessibility article on WebAIM and switch between using the Tab key and the mouse, keyboard focus highlights will disappear when clicking on the page, and pressing the Tab key after making a click moves focus to the next focusable item from where the click was made.

It sounds like @zepumph's solution would do something like that, or that your previous solution was doing something similar. However, your previous solution went to the last item that had focus, rather than the next focusable item.

Personally, I think placing focus on the next item makes more sense than the last item that had focus, at least for screen reader users. A user could click on something, hear where they are in the PDOM and then press Tab thinking they will go forward in the document.

[terracoda]

Is what @zepumph describing similar to what was described in Jeremy Field's article,

watch the body for the mousedown and keydown events and save which event happened and when it happened as data attributes [or in some PhET way]. ... [T]hen watch for the focusin and focusout events and use the last device used [keyboard or mouse?] to [decide whether to add a [focus] style to the element [in the PDOM] receiving focus [and the pink focus style on its corresponding visual node].

[Bracketed text by me]

[terracoda]

@jessegreenberg and @zepumph, I am sure you are already thinking about this, or at least @jessegreenberg is, but when we make the PDOM touch accessible, the PDOM will also be mouse click accessible, and focus highlights should become more graceful from that alignment.

Edit: changed may to should.

[terracoda]

@zepumph, what you describe in #713 (comment), is what we need. It is a way of remembering the previous placement (of the mouse), then I think it makes sense to go to the next focusable item from where the user clicked.

Do you have an idea of how long this would take?

[zepumph]

I do not! I want to sum up all of the above comments as best I can to make sure that we are on the same page.

1. It is not good to come with our own "best practice" similar to lots of things in a11y, there is a method that users expect, and we should adhere to that.
2. In general web dev:
   • when you click on a focusable item, the focus doesn't go away (nor the highlight)
   • when you click somewhere random, and then tab navigate, it goes to the next focusable item from where you clicked.

We currently aren't keeping the focus highlight when you click on a sim, but we are keeping track if you click on a focusable item, such that tab nav starts from there.

@terracoda you are recommending that we implement something where we keep track of where we click, and then go the the "next focusable item" from that pointer (if it clicked on a non focusable item), but I'm unsure how that would work. in the html it is all just one svg, so it is hard to map what focusable item is next after an x,y mouse click somewhere in the scene graph.

Further more, there is no heiristic that we could apply to this problem that couldn't be broken by manipulating the focus order. Like if we tried to say (the focusable items in the top left are first, and in the bottom right are last). For example, it could be beneficial for the last item in the focusable to be in the top of the sim, in which case how do we map a mouse click location to what to focus next?

I'm starting to think that our current algorithm is closer to the general web dev guidelines than anything we have spoken about implementing. If you have a screen reader on, and you click somewhere in the svg that isn't focusable, then it will say in document body or something like that, and the focus will start at the first focusable element in that document's body. Isn't that what you were recommending in here @terracoda?

Personally, I think placing focus on the next item makes more sense than the last item that had focus, at least for screen reader users. A user could click on something, hear where they are in the PDOM and then press Tab thinking they will go forward in the document.

[zepumph]

I will add that I think our sim's focusing is pretty similar to what I see in simple forms, when I click away and then tab, it goes to the first item (even if I click in between the email and message boxes: https://developer.mozilla.org/en-US/docs/Learn/HTML/Forms/Your_first_HTML_form#Summary

edited the link

[terracoda]

@zepumph, I'll just add to your comment

In general web dev:

• when you click on a focusable item, the focus style does go away (I mentioned I was wrong above). It usually changes to an active style.
• when you click somewhere random, and then Tab navigate, it goes to the next focusable item from where you clicked. I think that is the ideal situation.

In general, keyboard focus is not handled well by default.

Imagine you are filling out a long web form and you use the keyboard out of need - the mouse is difficult for you. You are near the end, and you relunctantly use your mouse on a datepicker because you think it will be easier. You miss and click on something unclickable. You then press the Tab key to refind your focus and it takes you to the top of the form.

Is the top of the form, the best place for you to be?

Note our sims are short (all visible when not maginified), so it is less of an issue for us, and I realize we have a strange case, a hidden PDOM. I am not sure what is possible, but I liked what you were saying in #713 (comment)

[terracoda]

I have heard surprise comments from blind users, like, "Oh, my focus went to the top." And even, "Oh cool, my focus went to the Play Area."

Both focus changes were unexpected, but had different impacts on different users. Unfortunately, I don't know what caused the change in focus. These are comments from two of my last 5 interviews.

[zepumph]

That's cool and good to know! In general it would be nice to know what was causing the behavior (that they liked or didn't), as I'm unsure about what users expect in this department. We can talk about this more at the dev meeting if it is important.

[terracoda]

@zepumph, when I click and then tab in that sample form in #713 (comment), focus goes to the next form element from where I clicked. I am using Firefox.

[zepumph]

I see that too now! On both chrome and firefox. I like the idea of striving for this, but again I think it would potentially be difficult to come up with a good solution for scenery.

[terracoda]

I agree, and I think this might resolve itself once we dive into touch access.

[emily-phet]

I'm adding the "on hold" label to this until we're able to dig more deeply into holistically handling alternating mouse/keyboard use.

[zepumph]

Unassigning myself for now.

[jessegreenberg]

We may continue this work in the issue referenced above. I reviewed comments and the thinking is in-line with the behavior we decided on in that issue. Closing this issue because to pick it up in the more recent one and to avoid a mega thread.