Candidate review: Toggle Button (Apple)

[jscholes]

Candidate review scope

• Vendor: Apple
• Test plan: Toggle Button
• Published test report: https://aria-at.w3.org/reports/1504
• Aim: gather any feedback and/or questions on the state of the test plan, including testing task scope, instructions, included commands, applicable assertions, and anything else believed to be a relevant concern.

Note: we are currently designing and implementing specific workflows and tools within the ARIA-AT Application to assist AT vendors with future reviews. In the meantime, leaving feedback on this issue directly is encouraged, or alternatively new issues can be raised using the existing facilities in the app.

Overview

The recommended starting point for the review is the published test report, available at the URL listed earlier in this issue. The report page includes high-level results for all AT/browser combinations that were tested, allowing the full data set to be explored and comparisons to be made between support levels.

Specifically, each combination is represented by a table, with one row per testing task. Each row conveys the following:

• the name of the individual testing task;
• the total number of required assertions, plus how many were recorded as passing;
• the total number of optional assertions, plus how many were recorded as passing; and
• the number of additional, unexpected/undesirable behaviours recorded by testers (such as excess screen reader verbosity).

Every test name acts as a link to view more detailed results, as does the "View Complete Results" link before each table. For each test, the details page includes:

• A "Raise an Issue" link, to file specific feedback on GitHub. Again, feedback may be recorded in this issue instead if preferred.
• An "Open Test" page, to view the full testing form that was completed by our testers. This form also includes the "Open Test Page" button; use this to view the final version of the APG example used as the test case.
• A table of expanded results for each testing command, including:
  • command key(s);
  • support level;
  • specific screen reader output; and
  • enumerations of all passing assertions, failing assertions, and unexpected/undesirable behaviours.
• A "Run History" disclosure button, facilitating access to AT and browser information and testing dates.

Note that the original APG example can also be accessed from the high-level report page, but this version may have been modified slightly to better support the purposes of the ARIA-AT project and its testers. Such changes are not substantive, and do not have any impact on results. The version under test can always be accessed from an individual test page as already described.

Finally, it may be helpful to reference the project's Working Mode, which describes the testing phases in detail.

CC @cookiecrook

[cookiecrook]

Published test report: https://aria-at.w3.org/reports/1504

This appears to load a nearly blank page.

[jscholes]

@cookiecrook Thanks for the quick response here! There indeed are some performance problems that are currently causing slow ARIA-AT App page loads, including the case of the page initially appearing to be blank. Your patience is appreciated as work continues to address this situation, and in the meantime I recommend reloading the page and waiting a couple of minutes for the content to render.

[cookiecrook]

Okay, it looks like the site has serious performance problems. Each page, including the test case, takes minutes to load, so this is not an exhaustive review.

However, I did find an incorrect assumption that seems to account for most of the failures I saw on the couple pages that loaded. Since it's a toggle button that does not speak "pressed", that means it's not pressed. The test assumptions seems to insist VoiceOver would redundantly speak "not pressed" but that state is inferred by the lack of "pressed."

Once those errors have been corrected in the test case and the site performance problems have been resolved, I'll review in more detail.

Thanks for working on these test cases!

[richnoah]

@cookiecrook we have implemented some performance improvements this morning. If you would not mind please try to access the pages again.

[jscholes]

@cookiecrook Thanks for the feedback on this plan so far. The following was discussed during a recent ARIA-AT Community Group meeting:

However, I did find an incorrect assumption that seems to account for most of the failures I saw on the couple pages that loaded. Since it's a toggle button that does not speak "pressed", that means it's not pressed. The test assumptions seems to insist VoiceOver would redundantly speak "not pressed" but that state is inferred by the lack of "pressed."

We appreciate having this additional context. We'd like to discuss with you the possibility of Apple accepting the test as-written. The group feels that there would be significant advantages if this test were to be recognised industry-wide as an appropriate statement of expected default behaviour for all screen readers, with Apple making plans to modify VoiceOver's default handling accordingly. Explicitly conveying the "not pressed" state in macOS Safari would:

1. Make VoiceOver's toggle button rendering consistent with its treatment of similar elements. In the same browser, VoiceOver explicitly conveys the "not checked" and "off" states of checkboxes and switches, for instance.
   • Note: some group members suggested a potential VoiceOver enhancement, whereby non-default verbosity levels would result in states like "not pressed", "not checked" and "off" being omitted in a consistent fashion across similar control types.
2. Make VoiceOver's rendering of toggle buttons consistent with their treatment in iOS Safari, where VoiceOver communicates both states.
3. Automatically generate confirmatory speech when toggling a "pressed" toggle button to the "not pressed" state. Currently, VoiceOver in macOS Safari is silent when carrying out this action, and the Community Group is concerned about the usability impact this could be having.
4. Improve usability in additional ways, especially for novice users. For those who are not familiar with toggle button semantics and how VoiceOver communicates them, concerns were raised about increased cognitive load due to the lack of explicit rendering of the "Not Pressed" state. Specifically:
   • Users may not know the acceptable states for a toggle button; and
   • There is a perceived requirement to activate the control multiple times to verify that a "not pressed" toggle button has the user's intended state.
5. Make VoiceOver's toggle button rendering consistent with most other screen readers, including NVDA, TalkBack and Narrator. Note that JAWS is also an outlier in this regard.

We welcome further input and discussion about this proposal and how Apple made this design decision for macOS VoiceOver, including any input from end-users if relevant. If you would prefer to conduct such a conversation privately and/or synchronously, rather than via this issue thread, please send me an email.

Thank you.

[cookiecrook]

@richnoah wrote:

we have implemented some performance improvements this morning. If you would not mind please try to access the pages again.

These problems are still preventing review. See #827 for one of the consistently reproducible issues.

[cookiecrook]

@jscholes wrote:

We'd like to discuss with you the possibility of Apple accepting the test as-written.

I thought the purpose of this test was to ensure that the control is understood to be togglable, and to convey the state of that toggle. It does, and so IMO the tests should pass. It's not clear to me why the verbiage "pressed" is considered critical here, but a request to change VoiceOver's interface seems like a separate issue. This test "failure" is based on incorrect test assumptions.

As I understood it, the intention of ARIA-AT was to ensure sufficient understandability/usability of interfaces, not to make all screen readers align on the same interface and language. @mcking65 Do you have thoughts on this?

[cookiecrook]

@jscholes wrote:

Make VoiceOver's toggle button rendering consistent with its treatment of similar elements. In the same browser, VoiceOver explicitly conveys the "not checked" and "off" states of checkboxes and switches, for instance.

This rendering ("selected toggle button") in web content is consistent with Mac VoiceOver's rendering of toggle buttons in native controls, and pre-dates ARIA. The rendering of switches and checkboxes is also consistent with similar native controls on Mac.

Note: some group members suggested a potential VoiceOver enhancement, whereby non-default verbosity levels would result in states like "not pressed", "not checked" and "off" being omitted in a consistent fashion across similar control types.

VoiceOver has detailed verbosity controls in the VoiceOver Utility app, but as mentioned above, I don't believe this is a verbosity issue. At default verbosity, "selected toggle button" conveys the same information as "pressed toggle button"… Likewise, "toggle button" conveys the same as "not pressed toggle button" but without redundancy.

Make VoiceOver's rendering of toggle buttons consistent with their treatment in iOS Safari, where VoiceOver communicates both states.

This is a reasonable request that I will discuss with the Mac and iOS teams. Thanks. (However, I don't believe the test result should be marked as failing in the meantime.)

Automatically generate confirmatory speech when toggling a "pressed" toggle button to the "not pressed" state. Currently, VoiceOver in macOS Safari is silent when carrying out this action, and the Community Group is concerned about the usability impact this could be having.

Thanks for the feedback. I will pass this on as well. One correction though:

"VoiceOver in macOS Safari is silent when carrying out this action"

VO does speak "selected" when toggling the control on, and plays a press sound icon (or "earcon") when the state changes to either selected or unselected.

Please be aware that there are a wide range of opinions on how action confirmation or navigation confirmation should be conveyed. A simple example of differing opinions is a Windows-style SR default speaking of the word "Tab" when the user presses the "Tab" key. VoiceOver doesn't speak this because it's redundant: the user pressed that key, so in most cases, verbalizing "Tab" is unnecessary, especially for common, frequent navigation or activation commands.

Improve usability in additional ways, especially for novice users. For those who are not familiar with toggle button semantics and how VoiceOver communicates them, concerns were raised about increased cognitive load due to the lack of explicit rendering of the "Not Pressed" state.

If I'm understanding this request correctly, it seems to consider the needs of web authors (who are novice VO users) over the needs of everyday users of VoiceOver. If so, this request is not in line with priority of constituencies. (e.g. consider users over authors)

Make VoiceOver's toggle button rendering consistent with most other screen readers, including NVDA, TalkBack and Narrator. Note that JAWS is also an outlier in this regard.

I don't believe screen reader homogeneity is a design goal for ARIA-AT, but perhaps @mcking65 has some thoughts on the matter.

We welcome further input and discussion about this proposal and how Apple made this design decision for macOS VoiceOver, including any input from end-users if relevant.

Many VoiceOver users are employed by Apple and are involved in every stage of product development: design, engineering, QA, and iteration.

[cookiecrook]

Make VoiceOver's rendering of toggle buttons consistent with their treatment in iOS Safari, where VoiceOver communicates both states.

This is a reasonable request that I will discuss with the Mac and iOS teams. Thanks.

For closure, the Apple-internal tracking numbers are rdar://100389758 (for Mac) and rdar://100389766 (for iOS). Those teams will discuss potentially converging on either the Mac or iOS behavior, or some variant in between. I am not able to predict a timeline for that discussion.

In the meantime, I believe the test expectations should be changed to match the existing VoiceOver behavior. Thanks.