APG Regression Testing Framework Project Description
The center-piece of ARIA Authoring Practices 1.1 is a set of design patterns for web-based GUI components. Each pattern includes an “Examples” section that includes one or more canonical reference implementations of the pattern. The primary purpose of the examples is to provide assistive technology, web, and browser engineers with stable, functional references for testing their support and use of ARIA. It is thus essential that modifications to the example implementations do not unintentionally break any aspect of the ARIA implementation.
The objective of this project is to integrate a test framework and test cases into the APG build process that prevents regressions in behaviors, accessibility properties, or accessibility features prescribed by the documentation that accompanies each APG example.
Test Framework: Open source test framework integrated into the aria-practice GitHub repository and Travis CI configuration that runs the below described test suites.
Test suites: Files associated with each example that define the set of assertions that encompass the below defined test coverage.
Documentation supporting framework implementation and utilization.
Validation Data: Data that validates the efficacy, veracity, and completeness of the framework configuration and test suite code.
Work will be managed by a project manager, APG Task Force chair (Matt King), and reviewed by the W3C Authoring Practices Task Force.
Implementation and development will be carried out in the w3c/aria-practices GitHub repository.
The project will be divided into phases documented by GitHub milestones.
The milestones, work breakdown, and review points for each phase will be defined by the contractor in collaboration with the project manager after a brief orientation and exploration phase.
Test suites for a subset of the examples, approximately six, will be coded by various members of the Authoring Practices Task Force and reviewed by the contractor to ensure task force members adequately understand how to utilize the framework.
Contributor Friendly: For people who develop or modify code for the APG, the framework is:
a. Platform agnostic, e.g., is equally easy to use on Windows, Mac, or Linux.
b. Easy initial setup, e.g., NPM package or installs by executing a script.
c. Generates easily digestible output.
a. The framework is integrated into Travis CI.
b. Performance is sufficient to run for every build. Or, if a specific class of tests require long run time, they are executed with a separate script that is run manually while the remainder is run for every build.
a. Developing test suites for new examples relies on skills and knowledge common among front-end web engineers.
b. The architecture utilizes packages that have a proven history of reliability and stability.
Extensibility: While adding and modifying test suites and assertions should be as simple as possible, any other form of extensibility or reusability is much less important that minimizing complexity. The goal is to focus on developing the most straightforward and maintainable framework for achieving the singular objective of preventing errors in the APG.
Types of Tests
Role, state, property, named and description: Each APG example page includes one or more tables that describes the implementation of ARIA roles, states, and properties for the examples or examples on that page. The test suite for an example will include assertions that validate claims made within those descriptions. This may include test after page load as well as after exercising elements of the component that change states or properties.
Keyboard focus: Each APG example page includes a table or tables documenting the keyboard interface for the example or examples on that page. The test suite for an example will include assertions that validate the behaviors described in those tables.
Validation of Framework Implementation and Test Suites
The project includes developing and executing a methodology for generating data that validates:
Efficacy: Does it capture a wide variety of injected faults and make the source of the fault easy to locate?
Veracity: Is the output accurate?
Completeness: Do the tests cover all documented requirements for each example tested?
The validation data will be included in the repository and surfaced in the documentation wiki.
The project includes providing documentation in a section of the aria-practices wiki that describes:
How APG contributors install and use the framework
How APG contributors code test suites for new or changed examples
Framework selection decision process
Framework Implementation and configuration details
Framework validation process
Sizing by Project Phase
Activity and estimated Hours:
- Explore: 4
- Experiment: 8
- Evaluate Options: 4
- Design/Configure: 8
- Coding: 300
- Documentation: 6
- Validation: 8
- Task Force Reviews and issue responses: 12
- Total: 340
Test Suite Development Breakdown
- Examples: 59
- Approximate keyboard command requirements to test: 407
- Approximate Role, State, and property requirements to test: 488
Calculation of coding hours assumes:
- 1000 requirements to test across 59 examples.
- Average of 5 hours per example.
- Total hours for test suite coding: 295