Mobile VO book pick up instructions given in the middle of two books

[KatieWoe]

Test device
iPad Air 2
Operating System
iPadOS 15.4.1
Browser
Safari
Problem description
For phetsims/qa#791
When reading through instructions for the sim with mobile VO, it reads out the "button" for the book, instructions for picking up and dragging the book, then the "button" zoomed in book. It is unclear which book is being referred to in the instructions, when it applies to both. In addition, I found it confusing that the books were referred to as buttons.

Visuals

[zepumph]

We have two thought-through design decisions here. @terracoda could do a better job explaining, but I'll take a first pass:

• The instructions are purposefully in between both books precisely because it applies to both books. Rather than duplicate the info (duplication in general is quite confusing to non visual access), we put it once in a spot where it can be accessed by both spaces.
• The books are buttons to communicate their interactivity. The "grab drag interaction" that @terracoda has developed is based on having a 2-step interaction to pick up the item (via a button press), and then to be able to use it. I believe that part of the reason for the confusion is that you are on mobile safari, that has some specific behavior (buttons are named "Chemistry Book" instead of "Grab Chemistry Book" and some other changes.

@terracoda does that all sound correct to you?

[terracoda]

testing...

[terracoda]

There's an interaction bug. When I grab the regular Chemistry Book, it automatically moves down. I think I already opened an issue about that a while ago.

And I can't seem to move the Zoomed-in chemistry book when I double tap and hold.

Regarding this specific issues, I do not remember why we removed "Grabbed" from the button label for mobile Voice Over. I'll have to check the design document. But @zepumph, you are correct that the word button is read out because the interaction is a button and part of a two-step drag interaction that has been shown to work well with blind learners.

The help text was intentionally placed in between the two book interactions to reduce duplication and to reinforce the idea that the two interactions are identical (except for the visual presentation). If you can see the screen, it depends on how you interpret the diagram if you get it immediately that the two interactions are the same interaction. But likley, most learners get it eventually.

The zoomed-in book is working fine now, but the regular book is not working. Some how the "Move down to rub harder hint" is not firing properly. And the book seem to try to autmomatically move down even when I move it up on grab.

[terracoda]

The related issue about the the regular book is already posted in #299.

If we fix that, things should be better.

[terracoda]

Just an FYI I am using iOS 15.5 now.

[terracoda]

I looked in the Friction design document. There are a lot of notes in there about the pros and cons of using aria-roledescription. These notes and the interaction and coding issues were worked out in this issue #152

I think we are using aria-roledescription in the Mobile grab-drag interaction whereas we might not be using it for the keyboard interaction on desktop. This difference would account for the removal of "Grab" from the button's name on mobile devices. The aria-roledescription indicates the book is movable, and the mobile help text explains how to double tap and and hold to drag the book.

I think the only issue is that the actual grab interaction is not currently working properly.

[terracoda]

@KatieWoe, could you explain what you mean by "When reading through instructions for the sim with mobile VO". I think you mean, "When reading through State Descriptions". Am I correct?

The State Description may contain explicit instructions for custom interactions like the grab-drag buttons in this sim, but generally, the State Descriptions have little to no explicit instructions for how to use an interaction. Blind learners know what to do based on the roles of the interactions (e.g., "button", "slider", "checkbox", etc.)

The "instructions" or "help text" we design should focus on why it would be a good idea to try out the interaction, rather how to activate the interaction.

I hope that helps clarify the approach.

[terracoda]

@KatieWoe, does this resolve your question about why the help text is in between the book interactions and why "button" is read out? Do you have any other questions for this issue?