Test Format Definition V2

Proposal for New Test Format Definition

NOTE: This page currently describes a proposal to be utilized in the resolution of ARIA-AT issue 974. Once finalized, it will replace the current Test Format Definition.

Overview of test files

Each test plan is defined by a set of files in its own directory inside of the "tests" directory. The name of the test plan subdirectory is named to reflect the test case, e.g, "alert" or "horizontal-slider". In the below list of files, the directory for each plan is represented by the string PLAN_DIRECTORY.

The test builder extracts and translates content from the following files:

• tests/PLAN_DIRECTORY/data/tests.csv
• One file for each covered AT with a file name format of tests/PLAN_DIRECTORY/data/AT_KEY-commands.csv
• tests/PLAN_DIRECTORY/data/references.csv
• tests/resources/keys.mjs
• tests/resources/command-settings.mjs

tests.csv

The columns in this CSV file contain the following information.

testId

Integer that identifies the test. Tests are number sequentially starting with 1. This id controls the order of presentation of tests in the test runner and reports.

title

String that names the test. This string appears in headings that title test pages and report sections and in the row headers of report tables.

The title string should:

• Be written with sentence capitalization, i.e., the first word is capitalized.
• describe the user task the test is evaluating in straightforward and brief terms that enable people reading reports to understand what users of this AT can and cannot do when they encounter an element represented by the test case.

setupScript

The name (not including .js extension) of a JavaScript file in the tests/PLAN_DIRECTORY/data/js directory. For example, a value of "focusOnFirstLink" in this column of the tests.csv file for command-button refers to file tests/command-button/data/js/focusOnFirstLink.js.

This is the script that will run when the "Setup" button is pressed on the page that presents the test case, e.g., the window that displays the example command button that is used to test command button pattern assertions.

setupScriptDescription

A short description of what the setup script does. This is presented in the test plan review page.

Examples:

• "sets focus on the 'Font' menu item"
• "sets focus on a link just before the menubar"

refs

Space separated list of ARIA roles, properties, and states covered by the assertions in this test.

instructions

This is a short description of the user task that is accomplished when the test is run. It is presented as one or more list items in the list of instructions provided to people who are running a test.

• It includes a verb that is synonomous with the verb used in the test name.
• It may start with a prepositional phrase that provides information about the starting point of the task being performed.
• If the description includes multiple statements that should be presented as separate list items in the test instructions, the | character is used to separate statements.

Examples:

• Starting at the 'Navigate forwards from here' link, navigate to the 'Print Page' button
• With focus on the 'Red' slider, increment the value 10 steps to a value of 138.

assertion1 through assertion26

Each assertion is in a separate column. There may be up to 26 assertions.

Each assertion is a human-readable string that describes a single expected behavior of the assistive technology. The complete set of assertions answers the question: "What information should be communicated to users by the assistive technology as a result of this interaction?"

Assertions should be written such that:

• Each assertion can fail or pass independently of other assertions.
• It provides precise expectations in terms of the example test case. For example, if asserting the name of a checkbox is conveyed, and the accessible name of the checkbox is "Lettuce", then the assertion should be written as: "Name 'Lettuce' is conveyed".

Assertions can have three priorities specified with the integer 1, 2, or 3:

• 1: MUST-have behavior
• 2: SHOULD-have behavior
• 3: MAY-have behavior

To specify a priority, start the assertion string with the priority level followed by a colon character, e.g., "2: The position of the item and number of items in the menubar are conveyed". If a priority is not specified, it defaults to 1.

Example assertions:

• "The role 'menubar' is conveyed"
• "The menubar name 'Text Formatting' is conveyed"
• "The menu item name "Font" is conveyed"
• "Availability of a submenu is conveyed"
• "2: The position '1' of the item in the menu is conveyed"
• "2: The menu contains a total of '3' items is conveyed"

AT_KEY-commands.csv (Example: voiceover_macos-commands.csv)

These files are contained in the directory tests/PLAN_DIRECTORY/data, e.g., tests/alert/data.

There is one file of commands for each AT covered by a plan. The name of the file starts with the key string defined for the AT in tests/support.json. For example, a test plan that covers JAWS, NVDA, and VoiceOver for macOS contains the following files in its data directory:

• `jaws-commands.csv
• nvda-commands.csv
• voiceover_macos-commands.csv.

The columns in this CSV file contain the following information.

testId

An integer that refers to a value in the testId column in the tests.csv file.

If a screen reader provides multiple commands for executing the task represented by a test, this file will have multiple rows with the same testId value -- one for each command.

title

a copy of the value of the title cell from the referenced row in the tests.csv file. This column is present only to make the data in this file mor human readable and to facilitate error checking during the test build process.

commandSettings

A comma-separated list of strings. Each string is a constant defined in tests/resources/command-settings.mjs.

Each constant describes a setting that must be active when the command is executed. These are settings that users are expected to change frequently while using the AT. In some cases, they are changed automatically by the command being tested. The motivating use case for this parameter is to describe whether or not a screen reader reading cursor must be active before the command is executed.

command

This column contains the command the tester executes in order to perform the interaction being tested. If the command requires a sequence of key strokes, such as two consecutive presses of an arrow key, each key in the sequence is separated by a comma.

The strings that represent keys are defined as constants in resources/keys.mjs.

invalidAssertions

A comma-separated list of assertion numbers that are not applicable to this command for this AT.

By default, all assertions defined for a given testId apply to each command listed in this file. However, if the design of an AT prevents some assertions from being applicable to a specific command, those assertions can be listed in this cell.

For example, if a navigation test includes an assertion stating that the active AT cursor or mode switched from reading to interaction, that assertion will only apply to some of the navigation commands for some of the screen readers. So, this cell would contain the number of the active cursor switching assertion in the rows that represent navigation commands that should not trigger an active cursor change.

references.csv

This file provides values for the references specified in the refs column in the tests.csv file. The refs column contains names of ARIA roles, states, and properties covered by a test.

The columns in this CSV file contain the following information.

refId

The name of the reference information. This name is used to label the information when it is presented in a test plan.

The following is a list of refIds that are often included:

• author
• authorEmail
• designPattern - The value contains the link to the design pattern found in ARIA Practices, e.g., https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/
• example - required - The value contains the link to original example html from aria practices, e.g., https://www.w3.org/WAI/ARIA/apg/patterns/checkbox/examples/checkbox/
• reference - required - The value contains a link to the reference html file for the example the user will be using doing the test, like reference/2020-11-23_175030/checkbox-1/checkbox-1.html. A skeleton for this reference can be generated from the example using the update-reference script. The reference should be placed in the reference directory for the test in a subdirectory named with the date of creation format YYYY-MM-DD_HHmmss. Anything this reference needs to change in a way that might effect test results (i.e not just fixing a typo) a whole new reference subdirectory should be created and this file should be updated.
• states, properties, roles - The key should be names of relevant roles, properties, and states for the example used in the tests.csv. The corresponding value should be the url with more information about this reference.

value

The value for the corresponding refId.