"Grab and Drag" interaction on mobile VoiceOver

[jessegreenberg]

When trying to click the "Grab Balloon" button with mobile VO, the balloon is released right away. After discussing with @zepumph, we realized we aren't exactly sure what the difficulty is and why the balloon seems to be picked up and released immedately after clicking the "grab balloon"button.

[jessegreenberg]

This should be looked into with high priority as part of the mobile a11y work, and since it sounds like other sims may use this patterh for accessilbe drag and drop.

[jessegreenberg]

Transfered to scenery-phet because it impacts the GrabDragInteraction implemented here.

[terracoda]

We've had at least one interview with a person using VoiceOver on an iOS device. The keyboard-accessible interaction for grabbing the Yellow Balloon in BASE was completely inaccessible to them. They were unable to figure out how to grab and drag the Yellow Balloon.

The main reason from my perspective was because the instructions on how to grab the balloon were confusing and misleading. Not surprising, as the instructions are designed for a keyboard user.

The participant double-tapped on the grab button, and then tried to swipe to move the balloon, thinking it was grabbed. However, the balloon was never fully grabbed (no dotted-grabbed focus highlight, and no alert about being grabbed).

Current keyboard-centric instructions

Grab BASE's Balloons:

Look for grab button to play. Once grabbed, press W, A,S, or D key to move up, left, down, or right. Space to release.

Grab Friction's Books:

Look for grab buttons. Once grabbed, use letter keys W, A, S, or D to move book or zoomed-in book up, left, down, or right.

Note that in the Friction sim, it is not very useful to release the book, so we don't bother explaining how to do that. Tabbing away releases it, so a release is quite natural upon trying something else.

Grab GFL's Ruler? (draft)

If needed, grab ruler to make measurements. Once grabbed, use letter keys W, A, S, or D to move ruler up, left, down, or right.

Note that the ruler interaction is similar to BASE's balloon interaction, except that the ruler won't do anything interesting upon release, except possibly animate itself to it's original home position. Also in the case of the ruler, using the ruler to measure the distance between the spheres is an optional interaction whereas grabbing and dragging the balloon is essential to succeeding with the sim.

Possible avenues to explore

1. Can we detect a mobile device and then deliver different help text?
   If we can deliver device-specific help text, here are a few phrases to start with.

BASE's Balloon

• Double-tap and hold to grab and drag balloon. Lift finger to release.
• Double-tap and hold to grab and move balloon. Lift finger to let go.

Friction's book

• Double-tap and hold to grab and move book.

GFL's Ruler (draft)

• If needed, double-tap and hold to grab and move ruler to take measurements.

2. Create wording that combines instructions for keyboard and touch?
3. Create wording that general and intuitive for all devices (I think this will be hard).
4. Try a different grab method for mobile - a different gesture? (needs investigation of custom interactions).

[terracoda]

First Steps:

1. Do not change the interaction and just change the instructions to the ones listed above.
2. Also figure out a way to update the Keyboard Shortcuts to Touch Shortcuts, or to toggle them back and forth.
3. Investigate two double taps with a hold on the second double tap for a custom drag interaction. This would require a change to instructions and interaction.

The changes to the instructions would be delivered automatically based on device detection.

[terracoda]

Keep general Grab & Drag items (e.g., visual hints) in this issue and Ruler specific items in GFL regular repo.

For example, Grab and Release rule is general and the Move or Jump grabbed ruler is not necessarily general.

[jessegreenberg]

It looks like detection of iOS was updated to support latest version in phetsims/phet-core@91bb985, so we should be good to go for next steps here. Thanks @jonathanolson!

[jessegreenberg]

Today @liquatics mentioned that it would be great to have a version of this ready for an interview on 10/23, so we will shoot to have an initial version ready by then.

[jessegreenberg]

I just tried role="application" on its own. Unfortunately, iPadOS 13 VoiceOver doesn't give us any special gestures while the virtual cursor is on it. Nor does it say anything about the "application" role. This verifies that the only way to have grab and drag is to double tap and hold on the object to initiate drag. Releasing the finger releases the object.

[zepumph]

From grab/drag interaction meeting today, here are some elements that came out of discussion:

• For now we would like to keep the keyboard help dialog the same. Perhaps adding sentence in the screen summary could be useful when on mobile devices.
• There is no reason to mess with the GrabReleaseCueNode because that isn't in the PDOM. It is only part of the focus highlight, on mobile devices we don't reliably have access to focus/blur information, so this is irrelevant.
• Help text for the grabbable/draggable Node is too often too complex and context specific to try to factor out a section for keyboard vs touch controls. So for now we will implement this in each usage. If it seems like there is something to factor out after the fact, then we can look into it.
• It would be nice for mobile to add an aria-roledescription to the "button" for touch. For GFL it may be something like: "Draggable Ruler"
• GrabDragInteraction needs a customizable accessible name. Right now it is hard coded to be (from SceneryPhetA11yStrings.js):

   	    grabPattern: {
   	      value: 'Grab {{objectToGrab}}'
   	    },
   	    movablePattern: {
   	      value: 'Movable {{objectToGrab}}'
   	    },
  

Where sim-specific, all of these should be applied to each sim that has this interaction; the current list is GFL, Friction, and BASE. I will make individual issues to design for each sim. The main priority is to update BASE help text to support #536 (comment). After that, these will be prototyped for GFL first, see phetsims/gravity-force-lab#141 for GFL implementation.

[terracoda]

Given the potential complexities and interaction nuances of grab and drag interactions in simulation, I do think it will be useful to have an option to create a label that does not contain the word "Grab".

That said, I am willing to try the label "Grab {{Measure Distance Ruler}}". It's a bit longer than the "Measure Distance" and "Measure Distance Ruler", but it might still achieve my goal of making it clear why some someone would want to grab the ruler in that sim.

I will continue working on the exact wording for the interaction and help text for GFL over in phetsims/gravity-force-lab#141

[jessegreenberg]

From meeting today with @terracoda and @zepumph:

Gesture interactions
Double Tap is the native gesture for activating buttons when VO is on
Double Tap and Hold is a custom gesture that can be used on any element when VO is on
A while ago, we were not receiving click or any other events with iOS, only point events. Now iOS gets click events AND pointer events.
This prompted a Scenery work-around that says if this element is under a PDOM element, prevent pointer events and only allow click events. (#1001)
In the case of grab-drag, we need to prevent click and activation so that the object does not get dragged on a double tap. This will prevent the focus highlight/visual cue from showing up and prevent the case where the object id grabbed but not movable.
Decisions
On iOS change the grab button to a div so that it is not clickable.
Explore the use of aria-described by to deliver the help text automatically
Watch to see how aria-roledescription=“moveable” and aria-describedby=“id-of-helptext” work together (edited)

[jessegreenberg]

I worked on this a while today and learned some new things about iOS VO.

1. aria-describedby only works on elements with interactive roles. button, img, sliderall work. Butdiv` doesn't work.
2. The black highlight VO uses doesn't surround the bounds of the element unless it is interactive. Again, only button, img, slider, etc.../
3. When using role=button on a div, the element is still clickable and so we still run into the issue where user can double tap on grabbable object and get into a dragged state without being able to move the object.

So next I am going to try keeping the balloon as a button but preventing it from turning it to a draggable in code.

[zepumph]

7a3cfa2

This looks really reasonable. Good work.

[terracoda]

@jessegreenberg, I like your idea of sticking with button, but preventing the transformation to draggable.

[jessegreenberg]

Discussion on 11/4/19:

We agree that the proposal in #536 (comment) sounds good.

Up next we want to make sure that this is working well. A few issues have been reported involving duplication of the aria-describedby text in BASE, incorrect appearances of the help text when using a desktop screen reader, and a flood of alerts. In the following comment we are going to outline the exact design specification so that we can reference it later.

We also mentioned that we want to generalize these strings that provide interaction help text:

• "Double tap and hold..." (grabbable help text)
• "Drag finger to move..." (on grab alert)

[jessegreenberg]

We expect the grab and drag interaction to behave like the following on iOS VoiceOver and Talkback:

• When in the "grabbable" state:

  • Object is a "button" in the PDOM.
  • It has an aria-roledescription of "movable".
  • It has help text associated with aria-describedby.
  • The help text is "Double tap and hold to drag {{object}}. Lift finger to release."

• When it is in the "draggable" state:

  • Alert on pickup will be something like: "Grabbed....Drag finger to move {{object}}. Lift to release." Subsequent alerts may not include the "Drag finger to..." interaction instructions like above.
  • The PDOM representation should change to a div with role="application" just like the keyboard interaction for consistency, but this should have no impact on the interaction because the device will be using pointer events.

[zepumph]

We also mentioned that we want to generalize these strings that provide interaction help text:

In the above commits, I tried to refactor usages of the gesture help text so that there are less usages of phet.joist.sim.supportsTouchA11y at the call site. Instead I converted grabbableHelpText into a few other options. I tried to make this as clear and flexible as possible. I think of BASE as the normal case, and so I think that that usage is very stream lined and nice. It is less great that we have to override gestureHelpText in ISLCRulerNode, but still I think that is better, because we are always using the same string from SceneryPhetA11yStrings.js

[zepumph]

incorrect appearances of the help text when using a desktop screen reader

This was because there was ISLCRulerNode specific code that was adding aria-describedby associations. I removed it in the above commits.

I think the only thing left for this issue is:

• Duplicated aria-describedby text, see Hear grab gesture hint 2 or 3 times balloons-and-static-electricity#457 (checked off here, but to be worked on over there)
• Reduce the number of movement alerts (BASE specific), see Reduce frequency of alerts during finger dragging balloons-and-static-electricity#455
• 7a3cfa2#commitcomment-35801446, if it is ever reproduceable we will create an issue.
• Review individual sims, checking for sim specific design as well as what was stated in "Grab and Drag" interaction on mobile VoiceOver #536 (comment)

Anything else @jessegreenberg @terracoda?

[terracoda]

@jessegreenberg and @zepumph, I thought of something new today. It may not be a good idea, but then it might be a good idea, so I just want to mention it.

What if the created grabbed object's name/label actually included the word grabbed, and then "Grabbed." in the alert would not be needed.

Code for the ruler might look like this:

<div aria-roledescription=”movable” aria-label=”Grabbed Ruler” role=”application”>Grabbed Ruler</div>

Which might sound like this:
"Grabbed Ruler, moveable." + Context Response: "In home position just below mass spheres. Centers 4 meters apart."

Code for a balloon might look like this:

<div aria-roledescription=”movable” aria-label=”Grabbed Yellow Balloon” role=”application”>Grabbed Yellow Balloon</div>

Which might sound like this:
"Grabbed Yellow Balloon, moveable." + Context Response: "In center of play area. Has no more negative charges than positive charges."

On some platforms, one get the aria-roledescription before the accessible name, like so:

• "movable, Grabbed Ruler"
• "movable, Grabbed Yellow Balloon"

Any thoughts? Do you think it might make sense to name the object as a grabbed object, rather than saying it is grabbed in a response? Do you thin it worth a quick try?

[terracoda]

@jessegreenberg and @zepumph, my previous comment may only be relevant for keyboard interaction? I am not 100% sure about a11y gesture.

[terracoda]

@jessegreenberg and @zepumph, I verified BASE's mobile a11y over in phetsims/balloons-and-static-electricity#452 (comment)

Mobile A11y gesture is working pretty god, but keyboard interaction no longer works. I will not test further until I hear from you folks.

[terracoda]

And I just took a quick pass with GFL gesture,
phetsims/gravity-force-lab#140 (comment)

then keyboard
phetsims/gravity-force-lab#140 (comment)

I won't test further until I am given the go ahead. I mostly wanted to bring myself up to speed on where we were at with Drag & Grab before our meeting.

[zepumph]

From the grab/drag meeting this morning with @terracoda and @jessegreenberg, we feel like all of the above design specifications have been implemented and much has been reviewed. From here we will look into specific sims, like GFL and BASE to review the specifics of each implmentation, and create new issues if needed to make any tweaks/bug fixes that may arise. Closing