Test Format Definition V2

Proposal for New Test Format Definition

NOTE: This page currently describes a proposal to be utilized in the resolution of ARIA-AT issue 974. Once finalized, it will replace the current Test Format Definition.

Overview of test files

Each test plan is defined by a set of files in its own directory inside of the "tests" directory. The name of the test plan subdirectory is named to reflect the test case, e.g, "alert" or "horizontal-slider". In the below list of files, the directory for each plan is represented by the string PLAN_DIRECTORY.

The test builder extracts and translates content from the following files:

• tests/PLAN_DIRECTORY/data/tests.csv
• tests/PLAN_DIRECTORY/data/assertions.csv
• One file for each covered AT with a file name format of tests/PLAN_DIRECTORY/data/AT_KEY-commands.csv
• tests/PLAN_DIRECTORY/data/references.csv
• tests/resources/keys.mjs
• tests/support.json

tests.csv

This file specifies information about the tests included in a test plan, including their names, the order in which they are presented, which assertions apply to each test, and more. It is located in the directory tests/PLAN_DIRECTORY/data, and the columns contain the following information.

testId

Integer that identifies the test. Tests are number sequentially starting with 1. This id controls the order of presentation of tests in the test runner and reports.

title

String that names the test. This string appears in headings that title test pages and report sections and in the row headers of report tables.

The title string should:

• Be written with sentence capitalization, i.e., the first word is capitalized.
• describe the user task the test is evaluating in straightforward and brief terms that enable people reading reports to understand what users of this AT can and cannot do when they encounter an element represented by the test case.

Examples:

• Navigate forwards to a slider
• Navigate backwards to a slider
• Request information about a slider
• Increment a slider by one step

setupScript

The name (not including .js extension) of a JavaScript file in the tests/PLAN_DIRECTORY/data/js directory. For example, a value of "focusOnFirstLink" in this column of the tests.csv file for command-button refers to file tests/command-button/data/js/focusOnFirstLink.js.

This is the script that will run when the "Setup" button is pressed on the page that presents the test case, e.g., the window that displays the example command button that is used to test command button pattern assertions.

setupScriptDescription

A short description of what the setup script does. This is presented in the test plan review page.

Examples:

• "sets focus on the 'Font' menu item"
• "sets focus on a link just before the menubar"

refs

Space separated list of the names of ARIA roles, properties, and states covered by the assertions in this test.

instructions

This is a short description of the user task that is accomplished when the test is run. It is presented as one or more list items in the list of instructions provided to people who are running a test.

• It includes a verb that is synonomous with the verb used in the test name.
• It may start with a prepositional phrase that provides information about the starting point of the task being performed.
• If the description includes multiple statements that should be presented as separate list items in the test instructions, the | character is used to separate statements.
• Note: If a test will be performed with both reading cursor commands and application commands, be careful to avoid wording that infers a particular cursor or screen reader mode.

Examples:

• Starting at the 'Navigate forwards from here' link, navigate to the 'Print Page' button
• With focus on the 'Red' slider, increment the value 10 steps to a value of 138.

assertions

a comma-separated list of assertionId values that specify which assertions to include in this test. The assertionIds are specified in assertions.csv.

Example: 1,2,3 specifies that assertions 1, 2, and 3 should be included in this test.

It is possible to override the assertion priority that is specified in assertions.csv with the syntax:

new_assertion_priority_integer|assertionId

For example, 1,2,2|3 specifies that assertions 1, 2, and 3 should be included in the test and that assertion 3 should have a priority of 2 (SHOULD).

Generally speaking, the priority of an assertion should be the same for all tests, so overriding assertion priority in this file should be unusual. It is not common that an assertion should have a given priority in one test and a different priority in another test.

More details about assertion priority syntax are provided below in the description of assertions.csv.

assertions.csv

This file provides wording and priority for the assertions included in the test plan. It is located in the directory tests/PLAN_DIRECTORY/data, and the columns contain the following information.

assertionId

Integer that identifies the assertion.

assertion

This column specifies the assertion priority and wording.

Each assertion is a human-readable string that describes a single expected behavior of the assistive technology. The complete set of assertions included in a given test answers the question: "What information should be communicated to users by the assistive technology as a result of this interaction?"

NOTE: The assertion may be written with tokens that are replaced with AT-specific values during the test build process. The method of token definition and substitution is described below.

Assertions should be written such that:

• Each assertion can fail or pass independently of other assertions.
• It provides precise expectations in terms of the example test case. For example, if asserting the name of a checkbox is conveyed, and the accessible name of the checkbox is "Lettuce", then the assertion should be written as: "Name 'Lettuce' is conveyed".

Assertions can have three priorities specified with the integer 1, 2, or 3.

Priority ID	Priority Name
1	MUST-have behavior
2	SHOULD-have behavior
3	MAY-have behavior

To specify a priority, start the assertion string with the priority level followed by a vertical bar, AKA pipe, character ('|'). For example, "2|The position of the item and number of items in the menubar are conveyed". If a priority is not specified, it defaults to 1.

Example assertions:

• "The role 'menubar' is conveyed"
• "The menubar name 'Text Formatting' is conveyed"
• "The menu item name "Font" is conveyed"
• "Availability of a submenu is conveyed"
• "2|The position '1' of the item in the menu is conveyed"
• "2|The menu contains a total of '3' items is conveyed"

Note about the precedence of assertion priority specifications:

• Assertion priorities specified in assertions.csv are the default for all tests run with any covered AT using any command specified for a test.
• If the assertion priority 'P' is specified for testId 'T' in tests.csv, 'P' takes precedence over the assertion priority value specified in assertions.csv for all AT only when running and reporting on 'T'.
• If assertion priority 'P' is specified in AT_KEY-commands.csv for testId 'T' and command 'C' of AT 'A', 'P' takes precedence over assertion priority values specified in both tests.csvandassertions.csv` only when running and reporting on performance of 'T' with 'C' of AT 'A'.

It is possible to write assertions that are rendered with AT-specific language. This reduces how often people reading reports need to learn ARIA-AT terms that are used to describe AT features in a generic way, e.g., "reading cursor" or "application focus". It also helps avoid misunderstandings when communicating with AT developers.

For example, to test for mode switching in Windows screen readers, a generic assertion might be: "The screen reader switched from reading mode to interaction mode". Reports would be more clear if a report about JAWS listed the assertion as "JAWS changed from virtual cursor active to PC cursor active, and a report about NVDA listed the assertion as "NVDA switched from browse mode to focus mode.

This can be done by writing the assertion using tokens that enable the build script to pull AT-specific values from tests/support.json. The format of the assertion is "generic-assertion-wording|tokenized-assertion-wording". The generic wording allows for a fall-back presentation of the assertion in the event that an AT doesn't have the necessary values specified in tests/support.json.

Token names are specified in curly braces. For example, the above assertion could be written as:

The screen reader switched from reading mode to interaction mode|{at} switched from {READING_MODE} to {INTERACTION_MODE}

The token names and their screen text equivalence are defined in an array named assertionTokens that is defined for specific ATs in the ats array in the file tests/support.json. The build script looks up the AT using the AT key from the name of the at_key-commands.csv file. In the object for that AT, it looks for an array named assertionTokens.

The assertionTokens array contains objects with properties name and screenText. The build script substitutes the token in the assertion with the string in the screenText property. The build script can render the above example assertion for JAWS and NVDA as follows:

JAWS switched from virtual cursor active to PC cursor active

NVDA switched from browse mode to focus mode

To get that result, support.json would need to contain the following:

"ats": [
    {
      "name": "JAWS",
      "key": "jaws",
      "assertionTokens": [
        {"name": "AT", "screenText": "JAWS"},
        {"name": "READING_MODE", "screenText": "virtual cursor active"},
        {"name": "INTERACTION_MODE", "screenText": "PC cursor active"}
      ]
    },
    {
      "name": "NVDA",
      "key": "nvda",
      "assertionTokens": [
        {"name": "AT", "screenText": "NVDA"},
        {"name": "READING_MODE", "screenText": "browse mode"},
        {"name": "INTERACTION_MODE", "screenText": "focus mode"}
      ]
    }
  ]

If the build script cannot find the necessary entries in support.json for a specific AT, it will use the fallback generic wording of the assertion.

AT_KEY-commands.csv (Example: voiceover_macos-commands.csv)

These files specify what commands are to be performed for each test in the test plan. There is one file of commands for each AT covered by a plan. The name of the file starts with the key string defined for the AT in tests/support.json. For example, a test plan that covers JAWS, NVDA, and VoiceOver for macOS contains the following files in its data directory:

• `jaws-commands.csv
• nvda-commands.csv
• voiceover_macos-commands.csv.

These files are located in the directory tests/PLAN_DIRECTORY/data, and the columns contain the following information.

testId

An integer that refers to a value in the testId column in the tests.csv file.

If a screen reader provides multiple commands for executing the task represented by a test, this file will have multiple rows with the same testId value -- one for each command.

settings

A comma-separated list of setting names.

A value is specified in this column if it is necessary for the tester to ensure the screen reader is in a specific state before executing the command. Note, all tests assume a screen reader is in its default state.

Settings specified in this column should be settings that screen reader users are expected to frequently change on the fly as they work. In some cases, they are changed automatically by the command being tested. The motivating use case for this parameter is to describe whether or not a screen reader reading cursor must be active before the command is executed.

The setting names must match the value of the name property of one of the objects in the settings array contained in the relevant object in the ats array in tests/support.json. For example, if a setting of "VIRTUAL_CURSOR" is specified in a file named jaws-commands.csv, then tests/support.json must contain:

{
      "name": "JAWS",
      "key": "jaws",
      "settings": [
        "name": "VIRTUAL_CURSOR",
        "screenText": "virtual cursor active",
        "instructions": ["Press Alt+Delete to determine which cursor is active.", "If the PC cursor is active, press Escape to activate the virtual cursor."]
      ]
    }

command

This column contains the command the tester executes in order to perform the interaction being tested. If the command requires a sequence of key strokes, such as two consecutive presses of an arrow key, each key in the sequence is separated by a comma.

The strings that represent keys are defined as constants in resources/keys.mjs.

assertionExceptions

A comma-separated list of assertion priority exceptions in the form NEW_PRIORITY|assertionId. An assertion can be removed from a command by setting its priority to 0.

For example, to specify that the priority of assertion 1 should be set to 2 (SHOULD) and that assertion 7 should not be evaluated for a command, the value in this column for that command would be:

2|1,0|7

This is particularly useful for testing automatic switching from reading to interaction for JAWS and NVDA. They are expected to automatically switch settings when certain navigation and interaction commands are executed and not switch settings when other commands are used. Similarly, VoiceOver does not automatically change settings while navigating, so such assertions do not apply to voiceOver commands.

If assertion 7 stated that the screen reader switched mode from reading to interaction, this column would need to include the value 0|7 in the rows for tests that include that assertion with commands that would not be expected to cause the change in screen reader state.

references.csv

This file provides values for the references specified in the refs column in the tests.csv file. The refs column contains names of ARIA roles, states, and properties covered by a test.

This file is located in the directory tests/PLAN_DIRECTORY/data, and the columns contain the following information.

refId

The name of the reference information. This name is used to label the information when it is presented in a test plan.

The following is a list of refIds that are often included:

• author
• authorEmail
• designPattern
• example (note: this is a required refId)
• reference (note: this is a required refId)
• Names of ARIA roles, states, and properties, e.g., button or aria-pressed

value

The value for the corresponding refId.

Following are the values that should be specified for the commonly used refIds listed above.

refId	Value Description	Example Value
author	Name of the author of the test plan	John Doe
authorEmail	Email address for the test plan author	john.doe@example.com
designPattern	URI of the APG design pattern related to the test plan example	https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/
example	URI of APG page containing the specific example implementation covered by the test plan https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/examples/checkbox/
reference	URI of the test case html contained in the reference subdirectory of the test plan directory. Skeleton HTML for this reference can be generated from the example using the update-reference script. The reference HTML file should be placed in a subdirectory of the reference directory that is named with the date of creation using format YYYY-MM-DD_HHmmss. When anything in this reference file needs to change in a way that could effect test results (i.e not just fixing a typo), a new reference subdirectory should be created and this file should be updated.	reference/2020-11-23_175030/checkbox-1/checkbox-1.html
ARIA role, state, or propertyname, e.g., checkbox URI of the ARIA specification for the role, state, or property	https://www.w3.org/TR/wai-aria-1.2/#checkbox