iOS VoiceOver doesn't support "cursor tracking"

[jessegreenberg]

Apple calls the ability for the virtual cursor to be in sync with DOM focus "Cursor Tracking". This is on by default for VO on macOS and can be turned off with a setting, but is permanently disabled on iOS.

This means that when reading with mobile VO, we never get blur and focus events. So we never see PhET's pink focus highlight and any scripted behavior that we write to change things on focus and blur (like aria-valuetext in phetsims/scenery#951 or other alerts on focus) generally won't be heard as the user moves around the simulation.

I submitted a feature request to Apple to support "Cursor Tracking" on iOS on 4/16/19, we will see what happens.

[jessegreenberg]

@terracoda assigning you so you are aware of this and to see if you have any thoughts.

[terracoda]

The loss of pink highlight may not be a huge loss on touch devices. We don't see them mouse interaction, so not sure they would be needed for touch/gesture interaction.

Does VoiceOver on iOS have it's own built-in highlighting?

I am also curious what happens when someone is using a keyboard with a touch device, which I know is possible.

[terracoda]

Is this helpful?
https://www.bbc.co.uk/guidelines/futuremedia/accessibility/mobile/design/visible-focus

Apple and the BBC say something about managing focus in the View. Is that an option for us?

[jessegreenberg]

Does VoiceOver on iOS have it's own built-in highlighting?

Thankfully it does! It uses the same black focus highlight as macOS VoiceOver.

I read that article @terracoda but I couldn't find anything about managing focus. Recommendations were about visually displaying the focused item. The problem is that when using mobile VO there is no focus at all. The virtual cursor just moves around the DOM.

My biggest concern is that focus event doesn't trigger when using mobile VoiceOver. So any time we send an alert to the utteranceQueue on focus or any time we change aria-valuetext on focus or do anything else, it will never happen on mobile VO.

[terracoda]

Oh, I see.
Is the mobile A11y version of Ohm Law on phet test? I'd like to try it out.

[jessegreenberg]

Yes, master on phet test supports mobile VO. Also this one:
https://phet-dev.colorado.edu/html/ohms-law/1.4.0-dev.29/phet/ohms-law_en_phet.html

[jessegreenberg]

Apple responded with:

There are no plans to address this based on the following:
We're not going to be doing this with VoiceOver anymore. Traditionally screen-readers have moved KB focus as they navigate, but since there's generally no keyboard involved on IOS it causes more problems than it solves and it leaks privacy data.
We are now closing this report.

[terracoda]

@jessegreenberg, does that mean we will never have access to blur and focus events on iOS?

[jessegreenberg]

I think so @terracoda. We may receive them when attached to a bluetooth keyboard, but otherwise we will not.

[jessegreenberg]

Meeting notes from 4/24/19:

• We would like to reach out to the community about this issue. Has anyone else encountered the limitation that mobile VO doesn't synchronize the cursor with keyboard focus, and if so are they aware of any clever workarounds?
• We should be aware that for the foreseeable future focus and blur events are not likely to happen on mobile VO. This may generally be OK. It isn't the end of the world if some information is lost on focus and blur. Current usages of changing content on aria-valuetext require that the user initiated the change through interaction. But critical information should generally be conveyed with more traditional methods.
• We are going to investigate on the design side how dynamic notifications that are currently embedded in aria-valuetext can be simplified. There has been other need for more structure around design of aria-valuetext.
• We are not going to investigate technical workarounds right now. Nothing good comes to mind, and the only solutions that might work are sim specific or fragile.
• If necessary, we could do sim-specific workarounds in gravity-force-lab-basics as we don't want to redesign the description/alert structure for that sim as it nears republication. But we are going to look at moving around some of the descriptions first in case the focus/blur information in the aria-valuetext is compartmentalized and easy to move around.
• We are going to challenge the response from Apple, as it is a decision that will impact the broader accessibility comunity. Their argument only applies to reading of the most basic web documents.

Assigning to myself to respond to Apple.

[jessegreenberg]

I pushed back in the issue thread and we will see what comes of it. @terracoda I believe that the remaining action items are reaching out to the community and investigating how this can be handled design-side. Reassigning to you for now, but let me know if I can help.

[jessegreenberg]

I heard back again from Apple, they responded by saying something to the effect of "Thanks, but leaving the report closed." They suggested that since this is a feature request we could follow up at http://www.apple.com/feedback/ or: http://www.apple.com/contact/

I will go ahead with that.

[jessegreenberg]

I sent an email through http://www.apple.com/contact/

[terracoda]

@emily-phet, @jessegreenberg and @zepumph I sent an email to the WAI Interest group and copied you all on the email. We'll see of we get any responses.

[terracoda]

@jessegreenberg, @zepumph I got 2 responses that might be helpful as we move forward for mobile a11y.

Suggested resources From Ashraf Aleem:

If I am not wrong you are talking about issues with "Touch and State(:focus, :hover, :active)". Please refer the following:
https://developers.google.com/web/fundamentals/design-and-ux/input/touch/?hl=en#stateful-elements-respond-to-touch
https://www.youtube.com/watch?v=13Gw1zskb9I (this is a part of free course on Udacity https://www.udacity.com/course/building-high-conversion-web-forms--ud890)

Historical perspective from Patrick H. Lauke

iOS used to fire focus on when VO was running and a control was swiped/touch-to-explored to, but you're right this changed at some point (have to admit I lost track of this quirk - the last time I tested and documented this - https://patrickhlauke.github.io/touch/tests/results/ - was iOS7.1, so quite a while back, and that had the focus event). Note that this (not having focus/blur events fire) has been the way Android with TalkBack behaved pretty much forever, so this will also be a problem for you on that platform.
The rationale from Apple for removing the focus/blur events is likely similar to the rationale brought up when I (many moons ago) wondered out loud why Firefox on Android didn't do this... (long story short: because when actual inputs receive focus, they bring up the on-screen keyboard, which would be disruptive to AT users) https://bugzilla.mozilla.org/show_bug.cgi?id=1000082
I am, unfortunately, not aware of any kind of workaround here off the top of my head...

@jessegreenberg, have you tried to incorporate the events described in the first article? There seem to be quite a few different ones, and they vary between browsers. If not, I'm wondering if @pixelzoom's suggestions (in some other issue) about a "universal event listener" might soon be needed. Does this suggested article help with that https://developers.google.com/web/fundamentals/design-and-ux/input/touch/?hl=en#stateful-elements-respond-to-touch?

[terracoda]

@emily-phet, @zepumph, I have reviewed the "verbose" aria-valuetext option in Gravity Force Lab Basics, and have assigned two logic/implementation questions to @zepumph from the design doc.

The table for changing position cognitive walk-through is now kind of messy, but it might be possible to re-arrange the progress indicators without too much impact on the design. I'll know more once I discuss with @zepumph what kind of impact it is on the implementation side and if the design changes are possible.

[jessegreenberg]

I heard back from Apple, who said that the request was better done through the bug reporter. So the recommendation to use the contact link was likely automated. This is probably the end of the line for communication with Apple, and since we know that this is the behavior for Talkback we would probably make more progress by taking this up with the standards committees.

[jessegreenberg]

Thanks for the resources as well @terracoda, that is helpful.

have you tried to incorporate the events described in the first article?

I played with the article and its examples this morning. The examples demonstrate usage of CSS state selectors for hover, focus, and active. It also suggests that to get the active state to work on iOS you have to add an event listener to the window like

window.onload = function() {
  if(/iP(hone|ad)/.test(window.navigator.userAgent)) {
    document.body.addEventListener('touchstart', function() {}, false);
  }
};

I tried the example in the article with and without VoiceOver and the focus selector does not work on iOS.

Interestingly, the buttons in the example do receive the mouseover event when VoiceOver is enabled, but only after they are activated.

So I still don't see a way to receive any focus or blur events on mobile VoiceOver.

Have we done everything in #138 (comment) and can this be closed? Or are there next steps we should take?

[terracoda]

@jessegreenberg, I think we are done with this issue. @zepumph and I are working on some design changes for GFL Basics, and in the future, I'll be more careful about how I design content for aria-valuetext and aria-live alerts.

[emily-phet]

From @jessegreenberg comment above:

we would probably make more progress by taking this up with the standards committees.

Do we want to reach out to the standards committee on this?

[terracoda]

@jessegreenberg, do we know which committee? I don't think there is a committee specific to mobile accessibility see: https://www.w3.org/WAI/standards-guidelines/mobile/

[terracoda]

@emily-phet and @jessegreenberg, it might be worth talking to people who are working on the Accessible Object Model.
https://github.com/WICG/aom

For reference, here's an article about the AOM
https://tink.uk/playing-with-the-accessibility-object-model-aom/

[emily-phet]

@terracoda We have spoken a developer at Mozilla before, previously with the IDRC. We could reach out to him for a potential connection. Or we could just reach out to the devs of the aom directly. Thoughts?

[jessegreenberg]

This issue may be more general than just specific to mobile. I think the behavior we want is the one that Patrick Lauke commented in https://bugzilla.mozilla.org/show_bug.cgi?id=1000082 (@terracoda linked above).

At a very high level, the request here is probably "cursor/focus should follow accessibility focus (with slight exceptions/special-casing for touch devices where a 'real' focus on a text entry field or similar triggers special behavior/interactions like on-screen keyboards)" (this would not only apply to Android/TalkBack/Firefox, but possibly more fundamentally on desktop etc as well)

So I think that the ARIA group may be the best bet? I am not sure that the AOM group is related to this.

[terracoda]

@jessegreenberg, the report is submitted, is it safe to close this issue?

[jessegreenberg]

As it turns out, this has been changed in iOS 14! VoiceOver now supports cursor tracking. It is the only setting and cannot be turned off (like in macOS). But this means we do get focus events on that platform. Closing.