Skip to content

Commit

Permalink
Infrastructure: Run regression tests in parallel in TravisCI and impr…
Browse files Browse the repository at this point in the history
…ove test selection logic (pull #1465)

Fixes issues #1449 and 1450 by running regression tests in parallel and improving logic that determines which tests to run.

To run in parallel:
1. Updates `.travis.yml` to use the [Ava feature](https://github.com/avajs/ava#parallel-runs-in-ci) that allows tests in parallel across three VMs in the CI
2. Updates Ava version to include [this bug fix](avajs/ava#2461) -- the fix prevents a false error when Ava tries to parallelize less than three test files (it parallelizes by files, so one test file is not parallelized).
3. Updates the script that runs the regression tests in the CI to only the test files for the example or test file you are editing to take advantage of the parallelization option. Before, it filtered tests by using a regex on the name of the test. Now, it explicitly lists all the test FILES that should be run.

When running `bash regression-tests.sh` locally from a branch that is not master, the script will now run as if the current branch is a pull request against master.
You can now test locally that file changes in your branch will trigger the correct tests.
The logic is:
1. If `package.json` has been edited, or if test utility files have been edited, or if example utility files have been edited, then all regression tests will be run in the CI.
2. Otherwise, if an example is edited, then the regression tests for all of the examples contained in the same example directory are run. For example: if you edit one of the two checkbox examples, then both the checkbox-1 and checkbox-2 example regression tests will be run in the CI.
3. All test files that have been edited will be run in the CI.
  • Loading branch information
spectranaut committed Aug 4, 2020
1 parent 962481a commit 32f0774
Show file tree
Hide file tree
Showing 4 changed files with 260 additions and 181 deletions.
39 changes: 31 additions & 8 deletions .travis.yml
Expand Up @@ -7,6 +7,7 @@ git:
depth: 3

stages:
- Lint
- Test
- Deploy

Expand All @@ -16,30 +17,52 @@ jobs:
- env: ALLOW_FAILURE=true

include:
- stage: Test
- stage: Lint
name: CSS Linting
script: npm run lint:css
- stage: Test
- stage: Lint
name: JS Linting
script: npm run lint:es
- stage: Test
- stage: Lint
name: HTML Linting
script: npm run vnu-jar
- stage: Test
- stage: Lint
name: Spellcheck
script: npm run lint:spelling
env: ALLOW_FAILURE=true
- stage: Lint
name: Regression Tests Coverage Report
script: node test/util/report.js
env: ALLOW_FAILURE=true

- stage: Test
name: AVA Regression Tests
addons:
firefox: latest
script: scripts/regression-tests.sh
env: TEST_WAIT_TIME=1000
env:
- TEST_WAIT_TIME=1000
- CI_NODE_TOTAL=3
- CI_NODE_INDEX=0
- stage: Test
name: Regression Tests Coverage Report
script: node test/util/report.js
env: ALLOW_FAILURE=true
name: AVA Regression Tests
addons:
firefox: latest
script: scripts/regression-tests.sh
env:
- TEST_WAIT_TIME=1000
- CI_NODE_TOTAL=3
- CI_NODE_INDEX=1
- stage: Test
name: AVA Regression Tests
addons:
firefox: latest
script: scripts/regression-tests.sh
env:
- TEST_WAIT_TIME=1000
- CI_NODE_TOTAL=3
- CI_NODE_INDEX=2

- stage: Deploy
if: branch = master AND type != pull_request
script: skip
Expand Down

0 comments on commit 32f0774

Please sign in to comment.