Balloon grab/drag interaction to support mobile accessibility

[zepumph]

From phetsims/scenery-phet#536 (comment), we will likely need to do a few updates to get this sim to support mobile accessibility (like iOS and VO).

@terracoda please supply the appropriate help texts, one for keyboard (likely the same as it currently is), and one for touch, and I can add these in conditionally for mobile devices.

[terracoda]

Grab BASE's Balloons:
Keyboard Interaction

• Accessible Name (no change): Grab Yellow Balloon
• Role: use native HTML role of button
• On focus sounds like: "Grab Yellow Balloon, button"
• Help text (a small change to bring into line with Friction) Look for grab button to play. Once grabbed, use letter keys W, A,S, or D key to move up, left, down, or right. Space to release.
  • "use letter keys" replaces "press" in the current version

Touch Interaction

• Accessible Name (no change): Grab Yellow Balloon
• Role (must change to communicate a custom interaction): use aria-roledescription to override the native HTML role of button.
• option 1 aria-roledescription = "draggable"
• option 2 aria-roledescription = "draggable object"
• On focus sounds like: "Grab Yellow Balloon, draggable"
• Help text, option 1 (brief): Double tap and hold to grab then drag balloon. Lift finger to release.
• Help text, option 2 (more explicit option if needed): Double tap and hold to grab then drag balloon. Move finger to drag. Lift finger to release.

EDIT - We eventually decided on "movable" instead of "draggable" for the custom grab and drag.

[terracoda]

@zepumph, please implement the option 1's for now.
Reviewing initial alerts (i.e., context responses) now to see what could be helpful.

[terracoda]

Context Responses for Keyboard (samples):

• Initial grab:
  • Grabbed. At center of Play Area. Has no more negative charges than positive charges. Press W, A, S, or D key to move balloon. Space to release.
• Subsequent grab:
  • Grabbed. On left side of Play Area. Has no more negative charges than positive charges.

Context Responses for Touch (samples):

• Initial grab:
  • Grabbed. At center of Play Area. Has no more negative charges than positive charges. Drag finger to move balloon. Lift finger to release.
• Subsequent grab (i.e., after a successful grab and drag):
  • Grabbed. On left side of Play Area. Has no more negative charges than positive charges.

[terracoda]

@zepumph, and I'm not hearing the word "Released" in the release interaction of the current version of BASE.
Could you check to see if there is a reason for that. It might have been intentional.

[terracoda]

Tagging @jessegreenberg in case he needs to know about these descriptions.

[terracoda]

@zepumph, please re-assign if you have questions.
Also, I want to note that I am making a note in the general issue over in phetsims/scenery-phet#536 as I might have an accessible name for the ruler in GFL regular that can keep the common code design/implementation pattern you have already created.

[terracoda]

@zepumph, reposting your generalized code here from phetsims/scenery-phet#536 (comment)

		      value: 'Grab {{objectToGrab}}'
		    },
		    movablePattern: {
		      value: 'Movable {{objectToGrab}}'
		    },

I'll try to rework descriptions from #452 (comment) to remain consistent with and true to the general pattern that you have already implemented and that we likely discussed previously.

Changes coming in next post.

[terracoda]

Grab BASE's Balloons:
Keyboard Interaction

• Accessible Name (no change): Grab Yellow Balloon
• Role: use native HTML role of button
• On focus sounds like: "Grab Yellow Balloon, button"
• On activation sounds like: "Movable {{Yellow Balloon}}" (not currently implemented in published version, but is working on master)
• Help text (a small change to bring into line with Friction) Look for grab button to play. Once grabbed, use letter keys W, A,S, or D key to move up, left, down, or right. Space to release.
  • "use letter keys" replaces "press" in the current version

Touch Interaction

• Accessible Name (new): Yellow Balloon
• Role (must change to communicate a custom interaction): use aria-roledescription to override the native HTML role of button.
• aria-roledescription = "movable"
• On focus sounds like: "Yellow Balloon, movable"
• Successful activation (Double tap and hold) would trigger initial and subsequent context responses (see comment above)
• Help text, option 1 (brief): Double tap and hold to drag balloon. Lift finger to release.

[jessegreenberg]

Tagging @jessegreenberg in case he needs to know about these descriptions.

I took a quick look, looks good to me! But let me know if I can be of more help!

[zepumph]

All has been implemented except for #452 (comment). @jessegreenberg I would like to hand it over to you, because it is your sim, and I want to leave it up to you how to implement those changes. Also they seem like they are not totally necessary for fast approaching interviews (though I'll leave that decision up to you).

[terracoda]

Just noting here with keyboard and screen reader the changes sound good:
I hear "Grab Yellow, balloon" on keyboard focus, then on activation I hear "Yellow Balloon, movable" plus the grab alert.

Once I have dev link, I will test touch and screen reader on my phone.

[terracoda]

@zepumph, I tested balloons-and-static-electricity/1.5.0-dev.12 on my iPhone 7 (iOS 13.1.3)

I hear "Yellow Balloon, movable" on focus. I swipe to read the help text and get the correct help text, "Double tap and hold to drag balloon. Lift to release."
Upon activation, I usually don't hear the grab alert until I move the balloon, but I am getting alerts.

That said, the interaction is working, though there are delays and pile-ups of alerts during grabbing, dragging and releasing that we will want to look at more deeply.

Nice work @zepumph.
For immediate next steps, please (as soon as you can before next Tuesday), please implement the the context responses for touch mentioned in
#452 (comment)

With a touch-platform-specific "initial grab" response, I think the implementation will be quite usable for Jen's interviews next week (edited).

Issues to note, and maybe to move to a new issue

• with a native double tap gesture on the balloon I was able to activate the visual grabbed mode for keyboard (the 4 arrows and dashed pink highlight )
• while in the visual grabbed modem, I was not able to move the balloon with my finger,
• and if I got into that mode, I had to move the focus off the balloon to be able to try again to successfully execute the custom double-tap-and-hold interaction.

[zepumph]

Re #452 (comment), I think it would be best if jesse took it from here with his sim.

[terracoda]

@jessegreenberg, regarding my comment about the not hearing "Released" #452 (comment), I think in several cases the initial release may be getting over-written by follow-up responses (alerts).

I hear "Released." when I move the balloon far enough away from the sweater. No need to investigate. I think to make "Released" heard more often would require a design change, perhaps changing "Released." from a stand-alone-response to a parameter to be inserted at the start of the first directional response (alert).

We can investigate this in a separate issue for a maintenance release when there is time.

[jessegreenberg]

Sounds good @terracoda, I am also generally not hearing any alerts when the balloon is grabbed or released on iOS VoiceOver.

[jessegreenberg]

@terracoda do you think it would be helpful to make the on demand help text for the draggable with touch read automatically with aria-describedby?

[jessegreenberg]

In meeting today we decided to add aria-describedby.

[jessegreenberg]

@terracoda here is a version with these changes, can you please review?
https://phet-dev.colorado.edu/html/balloons-and-static-electricity/1.5.0-dev.14/phet/balloons-and-static-electricity_en_phet.html

Also, please see comments in phetsims/scenery-phet#536 (comment) as discoveries there changed the implementation a little from what we discussed this morning. FInally, #455 is still an issue. But the interaction and alert content is all there.

[terracoda]

@jessegreenberg, the aria-roledescription and the aria-describedby sound good to me.

In a complex 2-balloon situation, I heard "Press Space to release" at some point.

Could you check if there are some special instances in 2-balloon experiments where we might have some keyboard-specific strings possibly showing up?

[terracoda]

Regarding "Press space to release". I think there is a hint to release after a long drag interaction.

That is only needed fro keyboard because there is a grabbed mode. I don't think we need it for mobile interaction.

[terracoda]

@jessegreenberg, also, I am hearing the hint "Double tap and hold. Lift finger to release" 3 times upon getting focus on a balloon.

[terracoda]

Re #452 (comment), I actually hear the hint 3 times on the Green balloon and 2 times on the Yellow balloon.

[terracoda]

@jessegreenberg, I reviewed version in #452 (comment)

and made 2 new issues. Hope that's ok.

FYI, I did see pink highlights while activating other controls besides the balloons, but none of the highlights seemed to get in the way nor make me think my focus was in the wrong place.

[terracoda]

@jessegreenberg, assigning back to you as I am not sure what to do with this issue.

[terracoda]

Verifying https://phet-dev.colorado.edu/html/balloons-and-static-electricity/1.5.0-dev.14/phet/balloons-and-static-electricity_en_phet.html

Platform

• iphone 7 with iOS 13.1.3

General

• Yellow Balloon in BASE is sounding really good, and seems
• I am hearing full responses fine. I don't seem to get interrupted at like in GFL sliders.
• The utterance queue seems to be working better. I seem to be getting way fewer responses when dragging the balloon. It sounds much better.
• Actually, the responses pile up, when I do things a little more quickly

On focus I here the following (which I think is all correct):

• Yellow Balloon, movable.
• Double tap and hold to drag balloon. Lift finger to release. (two times)

Interaction

• it is easy to grab (double tap and hold) when balloon has focus.
• it is easy to drag balloon (without lifting finger). My finger does not have to be directly over balloon.

Reading the pdom content

• mostly good
• can skim through headings easily
• have some trouble reading the initial screen summary with cursor and not skipping to next interactive item (e.g. the balloon)

[terracoda]

Not sure what label is needed, but want to make sure this issue is fully wrapped up before publication with sound, assuming the goal is to publish with mobile a11y as well.

Not 100% sure that publishing with mobile-a11y is in fact a goal.

[jessegreenberg]

I read through this issue and I think what is left is #457 and #458, but it has been a long time so it would be good to verify that iOS a11y is working well. Just switching label so this doesn't appear in list of common code issues blocking all publications.

[jessegreenberg]

This work has been deferred again, and mobile accessibility will not be supported in the upcoming publication of BASE.