"Sanity Checks With BackstopJS" talk & demo
Switch branches/tags
Nothing to show
Clone or download

README.md

Background

Visual Regression Testing allows us to make visual comparisons between the correct (baseline) versions of our site and versions in development or just about to be deployed (new) via screenshots diffs on targeted parts of your UI.

Why should you test?

  • We’re already doing it manually.
  • It’s super time consuming.
  • We don’t catch everything.

When should you test?

  • the pull request stage, via CI tools like Travis-CI or Jenkins
  • manually — good for a basic sanity check

‘When should I test?’ is actually a pretty hot topic in vis. reg. testing, so realistically you should find out what works for you and your team.

Choosing a tool

The more I’ve learned about visual regression testing, the further it has pulled me into end-to-end testing suites, running nightly builds, firing up VMs for testing environments, etc. when realistically, I just want to know if something has changed in my UI. I just want a sanity check. Has the code I’ve written/refactored messed anything up?

Theres a ton of visual regression testing tools. I looked at:

Criteria

When I’m looking at new tech and deciding whether or not to use it, I always think about these things:

  • Does its benefit outweigh the cost of having to learn it?
  • What does its future look like? (community support, funding, key players using it, etc)
  • How easy is it to implement into my current workflow?

Chosen Tool: BackstopJS

  • simple
  • large, incredibly active community
  • If you use Gulp, you can plug it right in. If not, you can run it easily via NPM.

What’s in BackstopJS?

  1. PhantomJS - headless browser = a browser without a GUI. Super useful for doing browser-y things like rendering a DOM, styling elements, doing AJAX requests, etc, all via the command line.

  2. CasperJS - scripting tool that allows us to interact with the page rendered by PhantomJS just like a user would: moving a mouse, performing clicks, entering text into input fields, and any other user actions you define on it.

  3. ResembleJS - the actual “diffing” tool itself. It compares two images and can determine pixel differences between them.

Installation

The following instructions is using the Gulp flavor of BackstopJS. If you’re interested in keeping it agnostic, roll over to BackstopJS for the official documentation.

1. Install BackstopJS

npm install backstopjs

2. Move to BackstopJS' node_module folder

Backstop makes you go into node_modules/backstopjs and to run its scripts currently. This is the one strange thing I've found about BackstopJS, but upon speaking to one of its core maintainers, it is currently being worked on to be phased out. In the meantime, run this command:

cd node_modules/backstopjs

3. Create a config file

while still inside of node_modules/backstopjs, run:

npm run genConfig.

NOTE: If you wish to use the example provided, you DO NOT need to run this command, as I’ve created a simple backstop.json file for you.

This command produces a backstop.json file, which gives you a file of options for you to configure: viewports to target, selectors to target for tests, names/paths for test/reference screenshot results, etc. This is where you write your tests, and if you have any CasperJS scripts, they go in backstop_data/casper_scripts

NOTE: in the example backstop.json file, I have set up the Baseline screenshots as:

"referenceUrl": "https://mathesond2.github.io/visual-regression-testing/"

the would-be development version is listed as:

"url": "file:///Users/davidmatheson/Desktop/visual-regression-lecture/index.html"

for the above url value, include your own path to the project.

4. Take reference screenshots

to create reference screenshots, run this command:

npm run reference

Reference screenshots are put in a generated folder: backstop_data/bitmaps_reference

5. Test new screenshots against baseline screenshots

To create test screenshots to diff against the reference screenshots you've already created, run this:

npm run test

Test screenshots are put in a generated folder: backstop_data/bitmaps_test/*time-stamped-folder-name

Running npm run test also generates a test results summary page that automatically pops up in your bowser to show you what tests have passed/failed, reference,test, and diff screenshots of each test, in addition to the config options for each test.

As long as your baseline screenshots are taken, you may run gulp test and test to your heart’s content. 💕

NOTE: This is for manually testing at the moment, but this repo will be updated with Travis-CI integration in the short future.

Quick Installation

1. npm install backstopjs
2. cd node_modules/backstopjs
3. npm run genConfig (NOTE: only use this if you DON'T want to use the created backstop.json config file & want your own)
4. npm run reference (to create reference screenshots)
5. npm run test (to create screenshots that test against your reference screenshots)

Further Reading

Challenges

Interference from dynamic content

Any data that updates/changes (ex: newsfeeds) will cause a false positive. How do you get around this problem?

  1. Identify the dynamic content and target it.
  2. Prior to every visual reg. test, use CasperJS to evaluate some JS that hides the dynamic content.
  3. Set a fixed height to each parent element, so adding/hiding a piece of content with each container wont cause tests to fail due to container expansion/shrinkage. This leaves the UI with just fixed outlines of the containers themselves.

Best Practices

  • Name your screenshots. Initially, screenshots are named screenshot_0.png which isn’t too descriptive, so include a name so you know what you’re actually looking at.

  • Don’t try to test everything. Try to maximize coverage with fewer tests. This is difficult because you want to know everything that is going on but the more granular your tests are, the easier it is to track down the actual bugs causing the problem.

  • Full page screenshots are a bad idea: keep your tests modular or it will get annoying. Also, bigger screenshots mean longer processing time.

  • Scaling visual regression testing: If you’re using git to store your baseline images (the initial, ‘correct’ screenshots) and your project becomes increasingly larger, you’ll be better served to use a tool like git annex or git media to prevent your repo growing way too huge.

  • Got a pattern library? make those elements/components your reference screenshots. If your team has an established pattern library, writing tests that map to your individual components can make your life much, much easier.

Sources

Visual regression testing
BackstopJS
phantomcss
gulp-phantomcss
Selenium
git storage tools