Ability to provide additional guidance for individual assertions

[jscholes]

Different screen readers can convey the same concept, such as the editable nature of a combobox or the presence of a pop-up menu, in very different ways. Sometimes these differences may be significant enough to confuse a human tester into providing the wrong result, particularly if they believe the wording of assertions to imply exact speech output. Examples:

• For role="button" with aria-haspopup="true", NVDA announces "menu button" which matches the role in our assertions. But JAWS says "button menu", and VoiceOver says "menu pop-up button".
• For grids, some screen readers still announce them as tables. I think this should represent an outright fail, but I can see how some testers may think differently if the rest of the grid semantics are respected.
• Some roles are silent either all or some of the time, like "option" or "menuitem".

Would it be helpful to have a small disclosure button next to the assertion text to show/hide an additional note, but have the note be optional? If yes, what sort of situations would warrant a note, and how should they be worded?

[mcking65]

Minutes of discussion of this issue are at:
https://www.w3.org/2021/01/21-aria-at-minutes.html

[robfentress]

Perhaps we could mention what is announced for the matching native system widgets in the test instructions. Then, a pass would be if the screen reader announces the same thing when encountering that kind of widget in a web page, the theory being that it would not violate user's expectations if that was what was presented. Or do we want to fail it either way, because the native system widget gives a theoretically confusing cue?

[robfentress]

Perhaps we could mention what is announced for the matching native system widgets in the test instructions.

A downside of this approach is that it may make the tests more brittle, because the way a native widget is presented by a screen reader may change as the product evolves in response to user feedback.