Impact of hint text/help messages on test results

[jscholes]

This came up on an ARIA-AT CG call recently. For some controls, JAWS and macOS VoiceOver speak some associated help/hint text, which in VoiceOver's case is often quite substantial. Help text announcements are enabled by default, so will be heard by human testers. The question is, should these messages contribute to the passing or failing of assertions in any way?

A practical example: on the Mac, the main announcement for an editable combobox doesn't convey editable state, but the hint text does. If we provide testers with an assertion that the editable state was conveyed, should it pass or fail? Note that AFAIK, the help text isn't repeated when querying information about the control e.g. with VO+F3, so the assertion will fail in at least some circumstances.

[jscholes]

Update following Feburary 4th community group meeting: we ideally need more examples of where hint text from a screen reader conveys information considered critical to an assertion passing. In the case of macOS behaviour for combo boxes:

• For an editable combo box (Create updated tests for APG design pattern example: Editable Combobox With Both List and Inline Autocomplete #340), as noted above VoiceOver only indicates that text can be typed in the control as part of the hint text.
• Question: is the role of combo box on the Mac sufficient to convey editable state on its own? I.e. should all controls that are identified by VO as combo boxes be considered to be editable?
• Currently, no. The select-only combo box (Create tests for APG design pattern example: Select-Only Combobox #331) is identified as a combo box in Safari, but a pop-up button in Chrome.
• The Chrome behaviour matches that of native select-style widgets.
• We should file an issue against the Core Accessibility API Mappings, which currently include a mapping of "AXComboBox" for anything with a combobox role, regardless of whether it is editable or not. If a combo box is select-only, is it reasonable for macOS user agents to present a pop-up button instead?

[jscholes]

Comment from @robfentress:

One issue I had when I was doing testing before was that I wasn't sure what to include when recording the output for VoiceOver. In particular, in the default configuration, VO announces hint text about the item in the VO cursor after a brief delay. I don't remember if we ever decided how to handle this. Would it simplify things if we changed the configuration instructions to instruct testers to turn off hints in the VO Configuration Utlity before conducting testing? Is any information we are looking for in our assertions communicated exclusively through hint text? If hints are to remain enabled in our configuration recommendations, then I think we should probably add something to our instructions telling users to make sure to wait long enough for them to surface.