Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Screen reader seems to skip more alerts in phet-io #319

Closed
Tracked by #899
stemilymill opened this issue Jan 5, 2023 · 29 comments
Closed
Tracked by #899

Screen reader seems to skip more alerts in phet-io #319

stemilymill opened this issue Jan 5, 2023 · 29 comments

Comments

@stemilymill
Copy link

For phetsims/qa#868

Win10 chrome + JAWS

I have noticed that often alerts are skipped after moving the book. More than half the time when testing the standalone phet-io sim, there is no "jiggling more; warmer" and it skips straight to saying that the jiggling and temperature are lessening.

This does happen sometimes in phet brand, but it seems much more frequent in phet-io, both in standalone and launched from studio (even if nothing was changed)

(i am not able to upload the video for some reason)

@Nancy-Salpepi
Copy link

Nancy-Salpepi commented Jan 5, 2023

For me, with VO in general (PhET and PhET-iO), the alerts that mention the atoms breaking away are usually interrupted.

IMG_7790.mov

@zepumph
Copy link
Member

zepumph commented Jan 5, 2023

I added some code so that the cooling alerts should interrupt the break away alerts way less. I haven't cracked the "left/right" interrupting though, since it is from a different system/module. Can you please test master and see if it sounds any better?

Also I didn't notice any differences between phet-io and phet brands.

zepumph added a commit that referenced this issue Jan 5, 2023
@stemilymill
Copy link
Author

On chrome + JAWS, I think it is improved in that the alerts for jiggling more and warmer are skipped less often. When they do get skipped, it now seems like they are not being interrupted by the cooling alerts, but by the "A" and "D" when using the WASD keys to move the book. Usually there aren't alerts for left and right when moving the book, but "left" or "right" is sometimes said in between jiggling more; warmer and jiggling less; cooler. @Nancy-Salpepi is VO improved?

@Nancy-Salpepi
Copy link

It is a little better. With VO the "left" and "right" statements will still cut off the other alerts.
In general it is odd that alerts are getting cut off with VO because they usually don't--they fill the queue and go on forever (not including when assertive alerts are made). On mobile VO I still hear all the alerts.

@zepumph
Copy link
Member

zepumph commented Jan 6, 2023

So the blend between each of these interrupting each other, cooling/warming/a/d/left/right, is challenging, and imperfect. To me that is secondary to ensuring that you hear when atoms break away, the design is centered around those alerts and that is really important. I know that it is possible to hear a/d/left/right as an interrupt to atoms breaking away, but that also may be correct, since those are directly from a user input. I'm not sure we would want to ignore those, but I could potentially do that though I would want to talk to designers.

@zepumph
Copy link
Member

zepumph commented Jan 6, 2023

I tried this patch given @Nancy-Salpepi's comment, but it actually didn't help much. What I'm finding with NVDA is that while you have keys pressed, most alerts are not spoken. That makes it challenging to hear the complete picture. I still may commit this though. Next steps for me are to have a bit of a discussion with @jessegreenberg.

Subject: [PATCH] formatting, https://github.com/phetsims/projectile-motion/issues/306
---
Index: js/friction/view/BreakAwayAlerter.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/friction/view/BreakAwayAlerter.js b/js/friction/view/BreakAwayAlerter.js
--- a/js/friction/view/BreakAwayAlerter.js	(revision 2573038475e1eb29f633a285f123fa9d535d4d73)
+++ b/js/friction/view/BreakAwayAlerter.js	(date 1673027813031)
@@ -15,6 +15,7 @@
 import friction from '../../friction.js';
 import FrictionStrings from '../../FrictionStrings.js';
 import FrictionModel from '../model/FrictionModel.js';
+import AriaLiveAnnouncer from '../../../../utterance-queue/js/AriaLiveAnnouncer.js';
 
 // constants
 const capitalizedVeryHotString = FrictionStrings.a11y.temperature.capitalizedVeryHot;
@@ -57,7 +58,10 @@
     // @private
     this.utterance = new Utterance( {
       alert: new ResponsePacket(),
-      priority: Utterance.HIGH_PRIORITY
+      priority: Utterance.HIGH_PRIORITY,
+      announcerOptions: {
+        ariaLivePriority: AriaLiveAnnouncer.AriaLive.ASSERTIVE
+      }
     } );
 
     // @private

@Nancy-Salpepi
Copy link

I had mentioned a while ago that I was missing A LOT with NVDA because it doesn't speak any alerts until you stop pressing a key. Definitely easy to miss things. #296

@zepumph
Copy link
Member

zepumph commented Jan 6, 2023

Right! Thanks for that. We closed that likely because there is nothing to be done. It is good to note #296 (comment) over here though. Thanks @Nancy-Salpepi

@zepumph
Copy link
Member

zepumph commented Jan 6, 2023

Screen reader seems to skip more alerts in phet-io

@jessegreenberg mentioned his best guess here would be about performance. I tested with ?profiler in https://phet-dev.colorado.edu/html/friction/1.6.0-dev.28/phet-io/friction_all_phet-io.html?postMessageOnError&phetioStandalone&profiler and it was running around 40FPS, then I turned on NVDA and it immediately went to 15 FPS. Weird! I will need to do some more research.

UPDATE: Ahhh, false alarm, it was because I turned on the NVDA highlighting. Turning that off fixed things.

@jessegreenberg and I really like ffe597d.

In general we feel like we have always felt: we wish we could do more to control the ordering of aria-live alerts. All things considered though, the two commits in this issue really help. I'll take a look at the PhET-iO stuff.

@zepumph
Copy link
Member

zepumph commented Jan 6, 2023

It has taken me far too long to realize this, but over in #321 we fixed the fact that the book dragging had become REALLY slow. @stemilymill and @Nancy-Salpepi could you please spend a bit of time retesting the cases that brought you to comment here and see if things got a lot better with faster dragging. The main piece with this is that you have to change direction much less frequently, so "a/d/left/right" interrupt important alerts much less frequently.

@Nancy-Salpepi
Copy link

I think things got a bit worse with the faster dragging using VO.

fasterbook.mov

@Nancy-Salpepi Nancy-Salpepi removed their assignment Jan 9, 2023
@Nancy-Salpepi
Copy link

@stemilymill how does it look on your end?

@zepumph
Copy link
Member

zepumph commented Jan 10, 2023

Thanks @Nancy-Salpepi. Can you please also compare your experience to the published version? Nothing should have changed too much in the design of description since that, but it is quite possible that timing issues have snuck in. If it is just as bad, @emily-phet and I think that we may want to punt.

@Nancy-Salpepi
Copy link

@zepumph
Something has definitely changed with VO. The alerts used to pile up and now they get interrupted too much!
In the published version I heard the "more atoms break away" statement 2/17 times. 🏈

@terracoda
Copy link
Contributor

terracoda commented Jan 11, 2023

@zepumph, re #319 (comment), I am not sure what you need to crack, but if the left-right responses are getting in the way, they are of the least importance. Their main function is to let blind keyboard users know that they are moving but not creating friction. Ideally, they are only heard when there is air space between the books, and even then, that space may not last long, so if they are interrupted that is completely fine.

@jessegreenberg
Copy link
Contributor

In #319 @Nancy-Salpepi reported the problem happens in Chrome as well, so that indicates in an issue with VoiceOver and not Safari.

@Nancy-Salpepi
Copy link

In phetsims/qa#886, I tried to use short key strokes with VoiceOver. 50% of the time I still didn't hear the entire statement "Very Hot. More atoms break away." It was either interrupted or entirely skipped over.

@zepumph
Copy link
Member

zepumph commented Feb 1, 2023

I believe that this is actually the same issue as #329. We had places in the code where we were clearing out alerts before they were being heard. @Nancy-Salpepi can you please double check this issue and comment on what else there is to do and we will got ahead and cherry pick the commits above to the RC branches.

@Nancy-Salpepi
Copy link

Nancy-Salpepi commented Feb 1, 2023

Sorry @zepumph. The break away alerts still sometimes get interrupted with VoiceOver.

I noticed that if an entire row breaks off, I hear the entire alert. If only some atoms break away, the alert gets interrupted. Here is a video showing both cases:

IMG_7848.MOV

@zepumph
Copy link
Member

zepumph commented Feb 1, 2023

@jessegreenberg, looking at the above video, should we get rid of the left/right alerts? Or perhaps some of them?

zepumph added a commit that referenced this issue Feb 1, 2023
@zepumph
Copy link
Member

zepumph commented Feb 1, 2023

@Nancy-Salpepi how about that on master? Now the left/right shouldn't interrupt other more important alerts as much. But since it is description, we don't have a ton of control on our side before it is sent to aria-live.

@jessegreenberg
Copy link
Contributor

The video isn't loading for me for some reason. But I don't know if we can do too much about this considering phetsims/a11y-research#182.

@Nancy-Salpepi
Copy link

When left/right alerts occur seems very random to me. Sometimes I hear them too much and sometimes rarely. They can
'interrupt jiggling more/less warmer/cooler" statements at times, but as you said @zepumph, not all the time.

@zepumph
Copy link
Member

zepumph commented Feb 1, 2023

I don't really know how much more effort to put here, I also feel like our changes are going to continue to step more and more on the original design decisions, and potentially cause other regressions.

My general sentiment about this issue is, "its a bummer that we don't hear some items, and sometimes it is important items." Is that an appropriate summary about the voice over experience?

I wonder if we should just proceed as is.

@jessegreenberg
Copy link
Contributor

That is my recommendation and preference. If Apple is able to fix phetsims/a11y-research#182 in the future we might have more confidence about making changes to improve this.

@zepumph
Copy link
Member

zepumph commented Feb 2, 2023

Excellent. Thanks everyone for diving in here and taking it on. Marking as ready for some cherry picking.

@jessegreenberg
Copy link
Contributor

I am a little confused because this issue refers to dev.28, but maybe some fixes here happened after the rc.1 was built.

Commits referenced in this issue are:
2a93dfd
ffe597d
945a1fb

The first two are in the history of friction-1.6. So the last one is the only one that needs to be cherry-picked.

@jessegreenberg
Copy link
Contributor

Done, ready to verify.

@Nancy-Salpepi
Copy link

In rc.3, description is definitely better than in was in the beginning in testing. I can hear getting warmer and break away alerts now, although they are still interrupted at times.
Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants