Develop assertion model for screen reader testing

[mcking65]

Objective: Determine how gramular to make assertions when testing screen reader support for an element or widget pattern.

[mcking65]

Proposing a less gramular model where:

1. There are two kinds of tests: Reading and Operation.
2. Each test may be performed with one of two pre-conditions: 1) SR in reading mode or 2) SR in Interaction mode
3. Each test is associated with an element or element type, e.g., checkbox, group, menubar, menuitem.
4. Each test has an importance (Must Have | Should Have | Nice to Have)
5. There is one assertion for a given test type, pre-condition, element, and importance. This assertion may cover multiple ARIA attributes. The test may be executed in multiple ways.

Attaching a spreadsheet with with examples. So far, only includes checkbox testing.
TestSimplificationExploration.xlsx

[mcking65]

Started testing menubar and experimented with some other ideas on presentation. See attached workbook with new tab for navigation menubar.
TestSimplificationExploration.xlsx

[mcking65]

Made more progress refining naming tests and defining assertions. Also made more friendly column (less technical) column names in spread sheet for menubar. Added more columns for notes about results of tests. See menubar tab in attached workbook.
TestSimplificationExploration.xlsx

[mfairchild365]

This is looking good @mcking65 - some comments:

• I like the names of the columns and agree that they are more friendly
• I like the addition of the "Success Notes" column. That should help future testers identify past behavior and compare their new results. It also helps whoever is reading the report understand how the screen reader behaves.
• It might be good to consider renaming the "tested attributes" column to something like "tested attributes and elements". I'm thinking of a case where the ARIA example uses an HTML element for its implicit roles and attributes. If I remember correctly, the simple two-state checkbox example included a <ul> element.
• It's not clear at a glance which attributes are failing within an assertion. For example, row 18 "Menubar menu open" is listed as partial support and has a couple of attributes. It's not immediately clear if both attributes have partial support, or if one of the attributes is supported but the other is not, etc.
• Do all commands need to yield full support for an assertion to be listed as full support? If one command fails, but the rest pass, does that yield partial support? If the goal is to simplify testing, there might be an opportunity to reduce the number of commands that we test. In my opinion, there needs to be at least one command available that yields a passing result for an assertion to be considered as being minimally supported. This might be worth discussing more.

[mfairchild365]

One further thought: It might be possible to use this simplified data format for collection purposes, then convert the data to fit a more granular data storage, which would include detailed information such as exactly which attributes and commands failed, etc. This detailed information could then power a more granular and flexible frontend. However, this would require another manual step after initial data collection, and would depend on the quality of the notes taken.

[mfairchild365]

A couple of thoughts. I recently put together a test for a11ysupport.io and tested 10 different AT/browser combinations. The whole process probably took 3 hours. 1 hour to set up the test (write expectations and an HTML example), and 2 hours to test against all of the at/browser combinations. This was with very granular expectations for each feature and support details including output and notes tied to multiple commands for each expectation.

I thought this was fairly quick. Some of the speed is probably due to the fact that I'm familiar with the system and screen readers that I tested. However, I didn't attempt to test every possible command provided by the screen reader. My goal was to identify basic support instead of comprehensive support. If I tested every command or even the majority of possible commands, I'm sure it would have taken a lot longer.

All of this is to say, I think it might be worthwhile to think about how many commands we want to test, and how we can address efficiencies at the UI layer (copy repeated output, etc) and still provide granular information in the data model.

[ghost]

For a future reference, I'm attaching assertions for menubar and checkbox as follows.
Two templates are different based on discussions made in each stage.

Experimental Assertions

Navigation Menubar (Aug. 5th)
Tested Screen Reader: JAWS, NVDA
190805_TestSimplificationExploration.xlsx

Checkbox (May 29th)
Tested Screen Reader: JAWS, NVDA, VoiceOver
19_0529_Checkbox_assertion_role.xlsx

[jfhector]

@mcking65 and I had a useful conversation about this on the call today. The minutes are here: https://www.w3.org/2019/08/28-aria-at-minutes.html

There is a mistake in my notes:

On the interface/table that a tester users, there might be: • at least one row for each key command; • Test result; • Anything needed to make sure that testing that key command in that circumstance is a repeatable thing (e.g. mode) • Something about result and output (JF: I didn't get that)

I believe that Matt wasn't talking about the "interface/table that a tester uses", but more about the data model.

Later in the notes, it becomes clear that we'd like interface/table that testers use to be much simpler, and only progressively disclose extra fields based on their answers. This is described later in the notes.

[zcorpan]

I think this is mostly covered in the working mode doc, which is tracked in issue #41 . Closing this issue -- please reopen or comment if something here is not yet addressed.