Testing

The code for cms.VA.gov undergoes numerous tests before merging, and tests are run before deployment and release.

The automated test suite for cms.VA.gov is defined in the tests.yml file and is run using the Yaml-Tests tool, allowing the same command to be used local development, in CMS -CI and for production releases.

The Yaml Tests Composer plugin is required by the main va.gov-cms composer.json file.

Goals

To adopt a strong test driven culture, the testing tools must:

Run the same tests in multiple environments with minimal configuration and a single command.
Allow developers to define tests to include in the suite and to write the tests.
Provide feedback to developers as quickly as possible and make test output as readable and accessible as possible.

Scope

To avoid entanglement of tests, tests should adhere, when possible, to their own area of concern. Practice separation of concerns as much as possible. There are three areas of concern.

CMS - This is the functioning of being able to login, edit and publish content. It's boundary of concern ends at the GraphQL endpoints.
Front-end - This is the Metalsmith build that creates the html front-end from the content accessed at the GraphQL endpoints of the CMS.
Content - This is the realm of making sure menu links and other links in content work. 508 testing is also part of content testing.

Entanglement should be avoided because it causes people from the non-relevant team to spend time solving issues that are not in their area of concern. Example: Developers chasing down a mis-entered content link is not a good use of time. End to End tests should be achieved when possible, by each area of concern providing coverage for their particular area.

The Yaml Tests tool was designed with these goals in mind.

VA.gov CMS Test Suite

Always refer to the file tests.yml for the canonical list of required tests that are included in the automated testing system, and are required to pass before merge or deployment,

There are 4 main types of tests:

Static Tests: Static tests are run as a Git Commit hook: Developers cannot commit code if any of these tests fail. Static tests only require the source code. No site is needed.

See the hooks/pre-commit file for the exact command run before git commit.
1. va/tests/phpcs - "PHP CodeSniffer" tests ensure coding standards are met.
2. va/tests/phplint - Ensures no syntax errors are present.
Deployment Tests: These commands are run during a production deployment. By treating them as tests, developers can identify failures in the deployment process before they go to production.
1. va/deploy/0-composer - Composer Install.
2. va/deploy/1-cache - Cache Rebuild.
3. va/deploy/2-update - Database Update.
4. va/deploy/3-config - Configuration Import.
NOTE: The tests are run in the order listed in tests.yml. The numbers here are to keep them in order when listed on GitHub.
WEB Integration Tests
1. va/web/build - Build the front-end from the current site. (Alias for composer va:web:build).
2. va/web/unit - Run the front-end unit tests. (Not yet merged. See PR547)
The long term goal is to run all of the WEB project's tests in our test suite, but more work is needed in the WEB codebase to make that possible.
Functional Tests
1. va/tests/phpunit - The CMS PHPUnit Tests include a number of tests, including Creating Media, testing GraphQL, Performance tests , Security, and more. See the tests/phpunit folder to see all the PHPUnit tests.
Utilizing the DrupalTestTraits library with PHPUnit gives developers the ability to bootstrap Drupal and write tests in PHP without an abstraction layer provided by Gherkin. PHPUnit is the preferred tool to write tests due to its speed of execution.
```
 Run a specific PHPUnit test with the "path" argument:

 ```
 lando phpunit {Path-to-test}
 ```

 Run a group of PHPUnit tests:

 ```sh
 lando phpunit . --group security
 ```
```
1. va/tests/behat - The Behat test suite includes:
  1. Content-Edit-Web-Rebuild test:
    
    This test is critical: it ensures the CMS does not break the WEB build.
    
    See tests/behat/features/content.feature
  2. Permissons Test:
    
    See tests/behat/features/perms.feature
  3. Drupal Spec Tests: The DST tool enforces the desired structure of the Drupal site by generating Gherkin Feature files. See tests/behat/drupal-spec-tool folder for all of the tests and more information on managing the Drupal Spec Tool.
  Run a specific behat test with the --name or --tags options:
```
lando behat --tags=dst
```

Running Tests

The main way to run Yaml-tests is the composer yaml-tests command.

Run composer yaml-tests --help for more information.

Composer Command: `composer yaml-tests`

All composer commands can be shortened to any unique string, so composer y is an alias for composer yaml-tests.

Run composer y --help to see more options.

Bin dir executable: `./bin/yaml-tests`

There is a bin/yaml-tests file provided with this package. You can run it from the project root.

NOTE: The bin directory is automatically included in the $PATH for all Composer commands, including yaml-tests itself.

See Composer Paths for more information on Composer and $PATH.

Local Testing with Lando: `lando test`

This project is configured to work with Lando out of the box.

Lando commands are listed in .lando.yml. There are some helper commands that map to Composer Yaml-test commands.

Lando Command	Composer Command
lando test	composer yaml-tests
lando test va/deploy	composer yaml-tests va/deploy
lando web-build	composer va:web:build
lando phpunit	composer yaml-tests va/tests/phpunit
lando web-build	composer va:web:build
lando behat	cd /app/tests/behat && /app/bin/behat

NOTES:

Any arguments passed to the lando command are passed through to the composer command.
Any Composer command can be run inside a Lando container after you call lando ssh.

@TODO: Standardize this mapping on Yaml-tests. It will continue to improve with features like timing, profiling, output logging, etc.

Limit tests to run

You can add an argument to filter the tests to run:

# Run the entire test suite.
composer yaml-tests

# Run `va/tests/phpunit` only
composer yaml-tests phpunit

# Run all `va/deploy/*` tests.
composer yaml-tests va/deploy

GitHub Integration

The Yaml-Tests tool also integrates with GitHub, providing pass/fail commit status for each test listed in tests.yml, and posting errors as comments on the commit's page on GitHub.com.

Branch Enforcement Rules

All of the tests in tests.yml are required to pass before a Pull Request can be merged. This is enforced by GitHub.com and is configurable: See the Branches section of the repository's Settings.

If an individual test fails, the Yaml-test tool creates a comment on the commit with the failed test results. The test results are also logged in DevShop.

GitHub Statuses API

The API used by Yaml Tests and GitHub for testing code is called the "Statuses API": https://developer.github.com/v3/repos/statuses/

It stores test results attached to the commit, based on SHA.

Yaml-tests reads the SHA of git repository, runs the test, and sends the state to GitHub Status API, which sends it along to the users.

What you end up seeing is something like this:

NOTE: The GitHub API stores this information attached to the Commit, not to the PR.

This means if you open a second PR with the same commits, the commit status AND the commit comments will show in both pull requests.

Composer, `config.bin-path`, and $PATH

Composer automatically loads the directory bin into the PATH of any composer command or script. More accurately, it includes the directory set in the config.bin-dir section of composer.json.

This means you only have to include the script name when referring to them in composer.json or in tests.yml.

For example, if you wanted to create a composer special-tests command as an alias for yaml-tests but with a different file and with a filter, add this to composer.json:

{
  "scripts": {
    "special-tests": [
      "which yaml-tests",
      "yaml-tests myuniquetests --file=custom.yml"
    ]
  }
}

Or, if you want to run drush or npm (or any other script in the bin dir) as a test, just call the script name:

# tests.yml example that runs commands from the project's ./bin directory.
example/drush/status: drush status
example/drush/version: drush --version
example/npm/which: which npm
example/npm/version: npm --version

The which npm command helps you find out which file is actually being run.

In this project's case, which npm would print /path/to/va.gov-cms/bin/npm.

Fortify security scans

Fortify scans are run manually.

About Drupal Security Team Coverage When a module is covered by the Drupal Security Team it means that the team will receive reports of vulnerabilities from the Drupal community and the general public and will work with the maintainer to fix and coordinate the module and advisory release.

Symfony and other non-Drupal.org hosted libraries are all out of scope for the Drupal Security Team, though the security team will occasionally work with these projects security teams to coordinate releases or help test etc. Symfony has an active security team and process/advisories (see https://symfony.com/blog/category/security-advisories).

Composer libraries don't have any defined process nor advisories, therefore this scan offers of additional scrutiny.

Excluded directories Drupal 8 core and contributed modules covered by the Drupal Security Team were not included in the scan.

  ./docroot/core/**/*"
  ./docroot/includes/**/*"
  ./docroot/modules/contrib/**/*"
  ./docroot/themes/contrib/**/*"
  ./docroot/profiles/**/*"
  ./docroot/scripts/**/*"

Included Vendor Libraries Vender libraries are third party open source packages included by Drupal core and modules to add functionality. For example Drupal 8 includes the Symfony open source project which in turn may include libraries from other open source projects. Symfony has an active security team monitoring security and posting process/advisories (see https://symfony.com/blog/category/security-advisories).

Whether these third party libraries are secure involves multiple factors (and has no definitive answer) project lifetime, maintenance status, frequency/size of major changes, number of maintainers, skills of maintainers in security topics, security of the projects own dependencies, security surface area (does the project deal with user actions, data, sessions, external systems etc), security architecture and threat model, code quality, documentation etc.

Nightwatch accessibility testing

Nightwatch is not currently included in the CMS test suite.

@TODO: Add Nightwatch tests using composer npm-asset and add to tests.yml

Manual visual regression

@TODO: Document what this means.

Test Coverage Map

(edit)

Table of Contents

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testing.md

testing.md

Testing

Goals

Scope

VA.gov CMS Test Suite

Running Tests

Composer Command: `composer yaml-tests`

Bin dir executable: `./bin/yaml-tests`

Local Testing with Lando: `lando test`

Limit tests to run

GitHub Integration

Branch Enforcement Rules

GitHub Statuses API

Composer, `config.bin-path`, and $PATH

Fortify security scans

Nightwatch accessibility testing

Manual visual regression

Test Coverage Map

Files

testing.md

Latest commit

History

testing.md

File metadata and controls

Testing

Goals

Scope

VA.gov CMS Test Suite

Running Tests

Composer Command: composer yaml-tests

Bin dir executable: ./bin/yaml-tests

Local Testing with Lando: lando test

Limit tests to run

GitHub Integration

Branch Enforcement Rules

GitHub Statuses API

Composer, config.bin-path, and $PATH

Fortify security scans

Nightwatch accessibility testing

Manual visual regression

Test Coverage Map

Composer Command: `composer yaml-tests`

Bin dir executable: `./bin/yaml-tests`

Local Testing with Lando: `lando test`

Composer, `config.bin-path`, and $PATH