Working Mode Scenario Support Analysis
This is an analysis of how community group (CG) members execute the ARIA-AT business process as defined by the Working Mode. The analysis is based on a list of scenarios that CG members encounter while executing that process. For example, one common senario is the CG needs to modify a candidate test plan as a result of discussions with AT developers.
A significant portion of CG ability to execute the working mode depends on data and test plan management capabilities provided by the aria-at.w3.org app. The goal of this analysis is to identify critical gaps in the ARIA-AT app functionality.
Scenario numbering is based on the working mode phases and steps as of March 25, 2023, (Working Mode V2). Scenario 0 is the "happy path" where a test plan does not require any rework or iteration. The numbering of subsequent scenarios is based on the step where the scenario deviates from the happy path. For example, scenario 2.2 is a variation of the happy path scenario that is identical to scenario 0 until the test plan reaches the second step of the second phase of the working mode.
Some of the following are implied by the working mode and some may need to be either added to the working mode or glossary.
- The version of a test plan is equivalent to the date-time stamp of the commit where it was merged to the main branch of the aria-at repository.
- The initial working mode phase of a new version of a test plan is "Draft".
- A given test plan can have only one version in any given phase of the working mode:
- Creating a new draft version replaces the current draft version if one exists.
- Advancing a draft version to candidate replaces the existing candidate version if one exists.
- Advancing a candidate version to recommended replaces the existing recommended version if one exists.
- Test plan changes may be either substantive or editorial:
- Substantive changes are modifications that affect the meaning or scope of reports generated from the plan.
- Editorial changes are modifications that affect content that does not appear in reports (e.g., instructions) or modifications to correct errors that do not affect the meaning or scope of reports generated from the plan.
- If a new version of an existing test plan is created and the changes in the plan are all editorial, the test plan administrator may advance the new version to the phase of the prior version without re-generating any report data.
Conditions:
- Test plan name: Alert Example
- In-scope AT: JAWS, NVDA, VoiceOver
- Browsers used during test run execution: Chrome on Windows and Safari on macOS
- The draft plan does not change as it moves through the steps of the working mode. There are no issues raised during the process.
NOTE: dates in scenario descriptions are fake. Dates are included to add clarity where dates are used for versioning.
- January: Steps 1 through 6 are executed in the aria-at repo.
- Feb 1: Admin executes step 7. When a test developer indicates a draft test plan is ready for community group review, merge the pull request into the main branch of the aria-at repo, and add runs for the draft test plan to the ARIA-AT Test Queue.
- Purpose: Enable CG members to review the "Alert Example" test plan for test writing errors and ommissions by running the tests for each in-scope AT (JAWS, NVDA, and VoiceOver).
- Actions:
- Admin creates the Feb 1 version of the Alert Example test plan by merging a PR that includes it into the main branch of the aria-at repo. As a new plan, it's status is "Draft". NOTE: This event marks the "Alert Example" test plan transition from working mode phase "Test Plan Research and Development" to phase "Draft Test Plan Review".
- Admin uses the "Add Test Plans to the Test Queue" experience on the "Test Management" page to add 3 runs of the Feb 1 version of "Alert Example" test plan to the test queue -- one run each for JAWS/Chrome, NVDA/Chrome, and VoiceOver/Safari.
- Outcome: On the "Test Queue" page, in the tables for "JAWS and Chrome", "NVDA and Chrome", and "VoiceOver and Safari", there is a row representing test plan runs of the "Alert Example" test plan.
- App issues:
- p0 Bug Issue 580: On the Test Management Page, 19 test plans are missing from the status summary table
- p0 bug Issue 579: When adding a test plan to the test queue, the plan version dropdown shows commits that do not include a change in the chosen test plan
- Issue 581: When adding test plans to the Test queue: the test plan dropdown includes two tests that are not in the tests directory of aria-at
-
Issue 582: When adding test plans to the Test Queue, The design creates inefficiencies and opportunities for data integrity errors by combining add and update capabilities:
- The design should distinguish between adding new runs of a test plan to the queue from updating the version of a test plan being used by an existing run. A new plan must be reviewed for all in-scope AT. The add experience forces the admin to repeat the same process for each in-scope AT, and it does not surface information about which AT are in scope nor about which already have test runs in the test queue.
- The design does not provide clear confirmation of action.
Note: For the "Alert Example" test plan, this phase of the working mode began on Feb 1 when the test admin merged the test plan into the main branch of the aria-at repo. The actual work of reviewing the correctness of the tests in the plan can start at any time after the phase start and happens when a tester starts executing one of the test runs.
- February: Admin and testers execute Step 1. Review and run the tests in the draft test plan.
- Purpose: Proof the "Alert Example" test plan for JAWS, NVDA, and VoiceOver to find test writing errors, e.g., incorrect assertions or missing commands.
- Process requirements:
- The test scripts, instructions, commands, and assertions for each in-scope AT are proven complete, accurate, and repeatable by having multiple CG members produce results that are consistent with one another.
- CG members raise an issue for each error they find in the test plan.
- Actions:
- On the "Test Queue" page, admin uses "Assign testers" function to assign 2 people to each of the 3 runs of the Feb 1 verssion of the Alert Example test plan.
- The six assigned testers use "Start Testing" function to run tests and record results for all tests.
- Outcome: On the "Test Queue" page, in the tables for "JAWS and Chrome", "NVDA and Chrome", and "VoiceOver and Safari", the "Report Status" column of the rows for "Alert Example" contain text "Draft" and a button named "Mark as In Review".
- Steps 2 through 4 do not apply to this scenario because testers do not find any errors in the test plan.
- March 1: Admin executes step 5. Ensure each test plan run is fully executed by at least two testers.
- Purpose: To ensure the quality of draft test plan reviews, the CG expects at least 2 CG members to review all test for each in-scope AT.
- Process requirement: For each run, all reviewers need to produce equivalent results, i.e., 0 conflicts.
- Actions: On the test queue page, in the three table rows that represent each of the 3 runs of the Feb 1 version of the Alert Example test plan, the admin:
- In "Testers" column, Ensure both assigned testers have completed all tests.
- In report status column, Ensure 0 conflicts arose during testing, i.e., the status text is "Draft".
- Step 6 does not apply to this scenario because there are no conflicts in results.
- Admin executes step 7. For each test plan run where there are no conflicts and where some tests fail, facilitate Community Group conversation about the failures to ensure Community Group consensus that the test is appropriate and that the failure indicates the presence of a browser, accessibility API, or AT bug.
- Purpose: Before asking Apple, NVAccess, and Vispero to review the "Alert Example" test plan, ensure CG members are confident the failures represent problems that Apple, NVAccess, or Vispero should address. (Exception: failure is due to browser bug.)
- Actions:
- Admin locates failures that require discussion, possibly by paging through the test runner for each of the three runs to hunt for failing tests.
- After discussions are complete, admin marks test runs as fully executed. On the test queue page, in the three table rows that represent each of the 3 runs of the Feb 1 version of the Alert Example test plan, the admin activates the "Mark as In Review" button in the "Report Status" column.
- Outcome: On the "Test Queue" page, in the tables for "JAWS and Chrome", "NVDA and Chrome", and "VoiceOver and Safari", the "Report Status" column of the rows for "Alert Example" contain text "In Review" and a button named "Mark as Candidate".
- App issues:
- Test Queue: Test plan run results summaries are not available. Information about numbers of passing and failing tests is not visible during the test running process. Identifying which tests have failures requires paging through the entire plan.
- Test Queue Report Status: The status "Draft" is not correct for a fully executed test plan run. Suggest: "N complete result sets".
- Test Queue Report Status: The button label "Mark as In Review" does not accurately represent the function the admin needs to perform when test plan run results satisfy process requirements. Suggest: "Mark Results Complete".
- Test Queue Report Status: The status "In Review" is not correct for a fully executed and finalized test plan run. Suggest that marking a test plan run complete should remove it from the test queue. Links to reports from "Draft" test plans could be made available from the "Test Management" page.
- April 1: Admin executes step 1. promote draft test plan to candidate test plan.
- Purpose: Indicate the "Alert Example" test plan is ready for review by representatives of Apple, NVAccess, and Vispero.
- Actions and outcomes:
- Action: On the test queue page, in the "JAWS and Chrome" table, in the row for the "Alert Example" test plan, the admin activates the "Mark as Candidate" button in the "Report Status" column.
- Outcome: On the "Test Queue" page, the "Alert Example" row is removed from the "JAWS and Chrome" table. On the "Candidate Test" page, A row for "Alert Example" is added to the tables for JAWS and Status Summary. Note that 1) the "Candidate Test" page status Summary shows "none" for NVDA and VoiceOver, and 2) the "Reports" page is also updated but not described here.
- Action: On the test queue page, in the "NVDA and Chrome" table, in the row for the "Alert Example" test plan, the admin activates the "Mark as Candidate" button in the "Report Status" column.
- Outcome: On the "Test Queue" page, the "Alert Example" row is removed from the "NVDA and Chrome" table. On the "Candidate Test" page, a row for "Alert Example" is added to the NVDA table, and the NVDA column of the status summary table is updated to show the "Alert Example" is "Ready for Review". Note that the "Reports" page is also updated but not described here.
- Action: On the test queue page, in the "VoiceOver and Safari" table, in the row for the "Alert Example" test plan, the admin activates the "Mark as Candidate" button in the "Report Status" column.
- Outcome: On the "Test Queue" page, the "Alert Example" row is removed from the "VoiceOver and Safari" table. On the "Candidate Test" page, a row for "Alert Example" is added to the VoiceOver table, and the VoiceOver column of the status summary table is updated to show the "Alert Example" is "Ready for Review". Note that the "Reports" page is also updated but not described here.
- App issue: Moving a test plan from "Draft" to "Candidate" should be a single action that can only be performed when all in-scope AT have at least one complete test plan run.
- April 1: Admin executes step 2. Notify developers of the in-scope assistive technologies that the candidate test report is available for their review and of the due date for initial feedback.
*Purpose: Inform Apple, NVAccess, and Vispero that the "Alert Example" test plan is ready for review and of the CG timeline for the review process.
- Action: Communicate via email and/or GitHub issue, per individual preferences.
- April through July: Apple, NVAccess, and Vispero reps execute step 3. AT if the plan is not satisfactory, raise issues in the aria-at repo.
- Purpose: Build and document industry consensus around desktop screen reader behaviors for the "Alert Example".
- Actions:
- On "Candidate Test" page, in the VoiceOver table, Apple representative opens the "Alert Example" review experience, reviews information for each test, and submits approving review.
- On "Candidate Test" page, in the JAWS table, Vispero representative opens the "Alert Example" review experience, reviews information for each test, and submits approving review.
- On "Candidate Test" page, in the NVDA table, NVAccess representative opens the "Alert Example" review experience, reviews information for each test, and submits approving review.
- Steps 4 and 5 do not require any action in this scenario because all 3 AT developers viewed the plan as satisfactory and submitted approving reviews.
Analysis not yet documented
- (Test Admin) Promote candidate test plan to recommended test plan when all the following conditions are met:
- The test plan has been in candidate phase for at least 120 days.
- There are no open test plan issues.
- (Test Admin) If a recommended test plan does not have a report for all in-scope AT and browser combinations, add the missing test plan runs to the test queue.
- (Tester) Execute all test plan runs in the test queue for the recommended test plan.
- (Test Admin) Publish the reports for any test plan runs where there are equivalent test results from at least two testers using a comparable AT/Browser combination.
- (Test Admin or AT Developer) For all commands where test results indicate an assertion is either not supported or incorrectly supported, file an interoperability bug against the AT.
- (AT developer) Resolve interoperability bugs; optionally integrate the aria-at test into AT build process.
- (Test Admin) When a new version of an in-scope AT or browser is released:
- Update lists of product versions for which testing is required.
- Update test queue to show that the AT/Browser combination is missing test results for current versions.
- (Tester) For any recommended test plan that does not have a published report for a current assistive technology and browser combination, execute a test plan run for that combination.
- (Tester) If a report for a new AT or browser version generates regressions, i.e., the assertions passed with the prior version and fail with the new version, raise an issue against the recommended test report.
- (At Large Web Developer or Interested Party) If testing a different but valid implementation of an accessibility semantic generates results that differ from the recommended test report, raise an issue against the report.
Analysis not yet documented
- (Test Admin) When an issue is raised against a recommended test report, within 30 days, complete one of the following:
- If the issue is a regression and the root cause is a change in AT output that appears to still satisfy assertions, validate this conclusion with community group and revise the report accordingly, i.e., the newly failing assertion becomes a passing assertion.
- If root cause is a new AT bug, file an AT interoperability bug with the AT developer.
- If root cause is a browser or an OS-level accessibility API bug, raise a bug with the owner of that product.
- If root cause is a problem with the test or test infrastructure, discuss corrective action options with community group and disposition report accordingly, e.g., revert the plan to candidate status and follow the candidate phase process.
A variation of scenario 0 where a CG member discovers during draft test plan review that the alert example test plan is missing a test.
To-do: Scenario analysis.
A variation of scenario 0 where a CG member discovers during draft test plan review that the alert example test plan includes an editorial error in the test instructions.
To-do: Scenario analysis.
A variation of scenario 0 where a CG member raises an issue for a test, suspecting an assertion may be inappropriate. The test developer determines that the assertion needs to be removed because the semantic is blocked.
To-do: Scenario analysis.
A variation of scenario 0 where the admin needs to support the conflict resolution process during test plan review.
To-do: Scenario analysis.
Scenario 3.3.1: Candidate test plan includes an assertion where consensus to change priority to optional is reached
A variation of scenario 0 where a screen reader developer asks to change an assertion priority to optional and consensus to make the change is reached.
To-do: Scenario analysis.
Scenario 3.3.1: Candidate test plan includes an assertion where consensus to change priority to optional is reached
A variation of scenario 0 where a screen reader developer asks to change an assertion priority to optional and consensus to make the change is reached.
To-do: Scenario analysis.
A variation of scenario 0 where a screen reader developer observes that a test is missing a command.
To-do: Scenario analysis.
A variation of scenario 0 where a screen reader developer asks to completely remove a test from a test plan and consensus to make the change is reached.
To-do: Scenario analysis.
A variation of scenario 0 where the plan is recommended and has not yet been run with JAWS or NVDA when using Firefox.
To-do: Scenario analysis.
A variation of scenario 0 where a new version of Chrome is released after the test plan becomes recommended.
To-do: Scenario analysis.
A variation of scenario 0 where the behavior of a new version of NVDA is different from the prior version when a recommended test plan is run.
To-do: Scenario analysis.