Add basic visual regression testing (#158)

* Add basic visual regression testing * Add inline comments explaining RGBA Co-Authored-By: Zach Margolis <zbmargolis@gmail.com> * Readable-ize assertion error message * Use remote host for sitemap * Split CircleCI jobs to isolate visual regression tests * Add CircleCI workflows configuration Required when not using singular "build" job * Upgrade CircleCI configuration to 2.1 Necessary for "commands" support See: https://discuss.circleci.com/t/circleci-2-1-config-overview/26057 * Rename bundle-yarn-install to bundle-npm-install Style guide not (currently) using Yarn * Add README documentation for visual regression tests * Remove acknowledged commit logic New workflow will be to bypass failed result of visual regression test status check as implied acknowledgment of changes Co-authored-by: Zach Margolis <zbmargolis@gmail.com>
18F · Oct 14, 2020 · b196ccf · b196ccf
1 parent 086d240
commit b196ccf
Show file tree

Hide file tree

Showing 6 changed files with 484 additions and 31 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -1,13 +1,8 @@
-version: 2
-
-jobs:
-  build:
-    working_directory: ~/identity-style-guide
-    docker:
-      - image: circleci/ruby:2.5.3-node-browsers
+version: 2.1
 
+commands:
+  bundle-npm-install:
     steps:
-      - checkout
       - restore_cache:
           key: dependency-cache-v1-{{ checksum "package-lock.json" }}-{{ checksum "Gemfile.lock" }}
       - run:
@@ -21,15 +16,62 @@ jobs:
           paths:
             - ~/.npm
             - ~/.bundler
-      - run:
-          name: Lint JavaScript and Sass
-          command: npm run lint
+  build:
+    steps:
       - run:
           name: Build assets and site
           command: npm run build
+
+jobs:
+  lints:
+    working_directory: ~/identity-style-guide
+    docker:
+      - image: circleci/ruby:2.5.3-node-browsers
+    steps:
+      - checkout
+      - bundle-npm-install
+      - run:
+          name: Lint JavaScript and Sass
+          command: npm run lint
+  integration:
+    working_directory: ~/identity-style-guide
+    docker:
+      - image: circleci/ruby:2.5.3-node-browsers
+    environment:
+      SKIP_VISUAL_REGRESSION_TEST: true
+    steps:
+      - checkout
+      - bundle-npm-install
+      - build
       - run:
           name: Run jest integration test
           command: npm run test-jest
+  visual-regression:
+    working_directory: ~/identity-style-guide
+    docker:
+      - image: circleci/ruby:2.5.3-node-browsers
+    environment:
+      ONLY_VISUAL_REGRESSION_TEST: true
+    steps:
+      - checkout
+      - bundle-npm-install
+      - build
+      - run:
+          name: Run visual regression test
+          command: npm run test-jest
+      - store_artifacts:
+          path: tmp/results
+          destination: results
+      - store_test_results:
+          path: tmp/results
+  accessibility:
+    working_directory: ~/identity-style-guide
+    docker:
+      - image: circleci/ruby:2.5.3-node-browsers
+    steps:
+      - checkout
+      - bundle-npm-install
+      - build
       - run:
           name: Run pa11y accessibility test
           command: npm run test-pa11y
@@ -38,3 +80,12 @@ jobs:
           destination: results
       - store_test_results:
           path: tmp/results
+
+workflows:
+  version: 2
+  test:
+    jobs:
+      - lints
+      - integration
+      - visual-regression
+      - accessibility
diff --git a/README.md b/README.md
@@ -31,6 +31,12 @@ npm start
 npm run lint
 ```
 
+## Visual regression tests
+
+When a pull request is submitted, a visual regression test will be automatically run to check for any visual changes between the working copy of the branch and the live documentation site. These will be reported as the `ci/circleci: visual-regression` GitHub status check.
+
+A failure of this status check only indicates that a visual change was detected. Depending on the types of changes being proposed, this may be expected. Anyone with access to the CircleCI dashboard can review the specific changes by following the status check "Details" link and comparing the set of screenshots under the "Artifacts" tab. If the visual changes are acceptable, the pull request can be merged, even if the status check is reported as a failure.
+
 ## Deploying documentation updates
 
 Documentation deploys are performed automatically upon merging to `master` by [Federalist](https://federalist.18f.gov/). Federalist performs the following steps:
@@ -44,7 +50,7 @@ More information can be found in Federalist’s [How Builds Work](https://federa
 
 ## Publishing a release to `npm`
 
-When you're ready to release a new version of the `identity-style-guide` package there are just a few steps to take. 
+When you're ready to release a new version of the `identity-style-guide` package there are just a few steps to take.
 
 1️⃣ Make sure all the changes indended for release are merged into the `master` branch.
 
@@ -54,7 +60,7 @@ When you're ready to release a new version of the `identity-style-guide` package
 
 `npm version patch -m "Upgrade to %s for reasons"`
 
-And a new version will be created. 
+And a new version will be created.
 
 4️⃣ Once you’re satisfied with any updates, do a trial publish to `npm` by running:
 
@@ -70,4 +76,4 @@ No need to run any special build steps — the publish script will lint the sour
 npm publish
 ```
 
-6️⃣ Document the release on Github. After you've pushed the release changes back up to `master`, [create a new release](https://github.com/18F/identity-style-guide/releases) with a target of `master`. The release version should match the version you just sent off to `npm` (like `v2.1.5`) and the title can be the same. Use the release notes to link to any important issues or pull requests that were addressed in the release. 
+6️⃣ Document the release on Github. After you've pushed the release changes back up to `master`, [create a new release](https://github.com/18F/identity-style-guide/releases) with a target of `master`. The release version should match the version you just sent off to `npm` (like `v2.1.5`) and the title can be the same. Use the release notes to link to any important issues or pull requests that were addressed in the release.
diff --git a/jest.config.js b/jest.config.js
@@ -5,5 +5,13 @@ module.exports = {
   errorOnDeprecated: true,
   notify: true,
   preset: 'jest-puppeteer',
-  testMatch: ['**/test/**/*.test.js'],
+  testMatch: [
+    'ONLY_VISUAL_REGRESSION_TEST' in process.env
+      ? '**/test/screenshot.test.js'
+      : '**/test/**/*.test.js',
+  ],
+  testPathIgnorePatterns: [
+    '/node_modules/',
+    'SKIP_VISUAL_REGRESSION_TEST' in process.env && 'screenshot.test.js',
+  ].filter(Boolean),
 };