QASP

Tested Code

Standard(s): Code delivered under the order must have substantial test code coverage and a clean code base.

AQL: Minimum of 90% test coverage of all relevant code

Method of Surveillance: Combination of manual review and automated testing, using agreed-upon publicly-available SaaS products

Backend

Backend code will support a robust test suite using the pytest framework. Code will be written with a goal of maintainability and with scope sufficient to allow proper unit testing. Tests will be automatically run upon every commit using the selected continuous integration platform. The main branch of the back end github repository will be marked as a "protected" branch, with the requirements that any pull requests into this branch meet the following criteria before merging will be permitted:

• All tests pass as reported by the CI platform
• The test coverage percentage of source code is at minimum 90%
• Code has been reviewed by a team member (aka "vendor code review")

Other requirements can be added to this list, such as static analysis status.

Front End

Frontend code will be tested using a Javascript test runner relevant to the frontend framework used for the web application, as well as pa11y, an automated accessibility testing tool. Tests will be automatically run upon every commit using the selected continuous integration platform. Pull requests will be merged if:

• All tests pass as reported by the CI platform
• The test coverage percentage of source code is at minimum 90%
• Code has been reviewed by a team member (aka "vendor code review")

Static code analysis will be tested during local development by developers using the Airbnb Javascript Style Guide.

Cross-browser compatibility testing will be performed based on client device requirements, including browser type, browser version and operating system.

Accessible

Standard(s): Client-side rendering must conform with section 508 standards.

AQL: 0 errors reported for 508 Standards using an automated scanner and 0 errors reported in manual testing.

Method of Surveillance: http://squizlabs.github.io/HTML_CodeSniffer/ or https://github.com/pa11y/pa11y

In order to maintain a section 508 accessible web application, our team will use multiple tools and procedures. These include complying with WCAG 2.0 A & AA standards by performing automated and manual testing using pa11y and WAVE and resolving all errors that these tools report. Additionally, accessibility will considered during the production of mockups to ensure compliant color contrast, and consistency with any relevant Department of State style guides.

Deployed

Standard(s): Code must successfully build and deploy into staging environment at the Department of State.

AQL: Successful build with a single command

Method of Surveillance: Combination of manual review and automatic testing

Code will be automatically deployed to a staging server hosted on AWS after any pull request has been merged into the relevant branches via a continuous deployment program such as Travis using AWS CodeDeploy.

Production deployment will be supported by a single command script which will step through all setup steps automatically, taking a system from 0 to deployed with a single execution.

Documented

Standard(s): All dependencies (and licenses for dependencies) are listed and all major functions are documented.

AQL: All dependencies are listed and the licenses are documented. Software/source code is documented. System diagram is provided.

Method of Surveillance: Combination of manual review and automatic testing

To facilitate adequate and useful documentation, the creation of documentation shall be included in the estimation of backlog stories. A story does not meet the "definition of done" unless all new functionality has appropriate documentation where applicable and the source code has appropriate in-line documentation as determined by vendor code review. By incorporating documentation firmly into the development and agile process, we will ensure high quality and useful documentation is maintained throughout development.

Secure

Standard(s): Code must be free of medium- and high-level static and dynamic security vulnerabilities.

AQL: Clean tests from a static testing SaaS, such as Gemnasium, and from OWASP ZAP, and/or documentation explaining any false positives.

Method of Surveillance: Combination of manual review and automated testing following the guidelines in the 18F Before You Ship guide.

Throughout the development lifecycle, both static and dynamic security scans will be incorporated into the CI/CD pipeline to certify that each deployment is secure. To help ensure vulnerabilities are addressed in a timely manner, each development-related story will not meet the "definition of done" unless it is free of medium and high vulnerabilities. Therefore making any deployment that introduces new medium or high vulnerabilities into the solution, to be considered not complete. Before the end of each sprint and for any potential false positives, a manual review of the code will take place to certify the results of the scans and catch anything that may have been missed.