Mobile VO isn't consistently reading alerts

[KatieWoe]

Test device
iPad Air 2
Operating System
iPadOS 15.4.1
Browser
Safari
Problem description
For phetsims/qa#791
The first time using Mobile VO on the phet-brand sim, when dragging the books, alerts played that described heat, particle motion, location of the book, etc. However, when trying this on the phet-io brand sim, the alerts didn't play. I went back to the phet-brand sim and wasn't able to get the alerts to play there either. I think that the alerts stop working if the sim is already loaded when you enter VO and come to the sim. If you follow a link with VO, the alerts seem to play.

[zepumph]

I will investigate this, @terracoda wanted to note that this happened at one point while switching tabs, but Apple seemed to fix it to her. I'll take a look.

[zepumph]

@emily-phet and I think that if I can't reproduce this easily, that it likely should not be investigated further.

[terracoda]

It would be good to investigate a little bit across sims just to see if there is something different about Friction.

It would be nice to verify if this is happening in a native common-code interaction, a custom common-code interaction, or a Friction-specific interaction, or all interactions. I am not sure if Friction has a Friction-specific interaction (i.e., the books). I think those grab and drag interactions in Friction are custom-common code interactions that have Friction-specific timing for context responses that describe what is happening with the atoms.

But I agree with @zepumph that if this is not easy to reproduce, it could just be something platform-specific to Apple devices.

It would be good to rule out any memory usage issue.

[emily-phet]

Insightful, as always @terracoda. I was just wondering, would QA currently know enough about the custom vs native interactions as we think of them in PhET sims to be able to recognize the need to consider these in QA testing generally? If not, this might be useful to point out to them. For example, recognizing when a bug is happening with a custom interaction, and checking if the same issue happens on similar custom interactions and related situations with native interactions, to help isolate potential root of the issue.

[KatieWoe]

We can maybe guess which are custom vs native, but if that is something you want us to test specifically, you should definitely specify in the testing issue what interactions should get special testing.

[zepumph]

Everything is native except the dragging of the book. The grabbing of the book and rest all are both native buttons. Likely this is happening within the application role (dragging the book), but it doesn't necessarily mean that this is causing that. I'll take a look

[terracoda]

I got responses fine on my iPhone 7 (iOS 15.5) the first time around, but when I returned again, VO would just read the State Descriptions - no Responsive Descriptions.

I tried Reset All, refresh, restarting VO, and could not get responses to come back.

I then shut off my phone and when I restarted my phone, I first cleared Safari's History and Website Data in Settings. Then loaded phettest, did a fresh pull all and loaded Friction from PhET Marks. The Responses worked again. I opened the phet website in a second tab, and returned to the sim. I again got responses. Then I visited a third tab and opened wikipedia. I returned to the sim and still got responses.

Maybe clearing the History and Website data is a good work around. Maybe the hard shut off (a big pain) is a good work around.

I am not sure we can address this behaviour by mobile VO which seems to be random.

However, in addition to the above, I found some issues with the descriptions. On the first grab I got hints that refered tot the A and D keys with the book, and the WASD keys with the Zoomed-in book. Also, the dynamic part of the Scene Summary went missing early in my exploration, but it eventually came back. These two issues happened twice. I'll open 2 new issues for these items. Addressing these 2 issues might help with the random behaviour of the responses, but might not. Regardless these 2 issues should be blocking issues,

[terracoda]

@KatieWoe, do you mind doing a bit more testing to see if a clearing history and website data works for you, or if both a shutdown/restart and the clearing of website data are both required?

I am just wondering if the experience is better if students know they first need to clear their history and website data.

[terracoda]

I removed blocks-sim-publication, since I was able to get responses going again. Is that ok by everyone?

[jessegreenberg]

Noting that I made a very simple HTML example for a different issue (phetsims/a11y-research#175) and iOS VoiceOver often failed to announce any alerts, indicating that this is an Apple problem.

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>TEST PAGE</title>
</head>
<body>

<p aria-live="polite" id="alerter"></p>

<article>
  <h1>My Heading</h1>
  <p>Friction is an interactive sim. It changes as you play with it. It has a Play Area and a Control Area.</p>
  <p>Chemistry book has far fewer jiggling atoms as many have broken away. Chemistry book rests lightly on top of a Physics book, and is ready to be rubbed against it. In zoomed-in view of where books meet, surface temperature is cool, and atoms jiggle a tiny bit.</p>
  <p>Reset sim to make more observations.</p>
  <p>If needed, check out keyboard shortcuts under Sim Resources.</p>
</article>

<button id="alert-button">Toggle alerting</button>

<script>
  const alertButton = document.getElementById( 'alert-button' );
  const alerter = document.getElementById( 'alerter' );

  let running = false;

  alertButton.addEventListener( 'click', event => {
    running = !running;
  } );

  let i = 0;
  window.setInterval( () => {
    if ( running ) {
      alerter.innerHTML = `This is a new alert ${i++}`;
    }
  }, 1000 );

</script>
</body>
</html>

[KatieWoe]

Clearing history for colorado edu seemed to work for me.

[terracoda]

That's great @KatieWoe. Thanks for testing.
At least we have a known work around for learners since it seems like we won't be able to address this issue before publication.

[terracoda]

@jessegreenberg, also validated HTML and nothing seems to be wrong there. Oh, this comment was meant for the issues about VO randomly skipping some of the state descriptions #292 and #308.

[terracoda]

@zepumph, I think the best approach might be to mark this as a platform bug.

[zepumph]

@terracoda, @emily-phet and I feel like since this is similar to the types of problems that screen reader users encounter, and that clearing history worked to fix this, we will close this issue.

We will work on collecting a central spot to document "tips" for when our users run into these sorts of items (for example for teachers to better support their students using description).

@terracoda will reach out to a WAI email group.