Visual Regression Testing allows us to make visual comparisons between the correct (baseline) versions of our site and versions in development or just about to be deployed (new) via screenshots diffs on targeted parts of your UI.
Why should you test?
- We’re already doing it manually.
- It’s super time consuming.
- We don’t catch everything.
When should you test?
- the pull request stage, via CI tools like Travis-CI or Jenkins
- manually — good for a basic sanity check
‘When should I test?’ is actually a pretty hot topic in vis. reg. testing, so realistically you should find out what works for you and your team.
Choosing a tool
The more I’ve learned about visual regression testing, the further it has pulled me into end-to-end testing suites, running nightly builds, firing up VMs for testing environments, etc. when realistically, I just want to know if something has changed in my UI. I just want a sanity check. Has the code I’ve written/refactored messed anything up?
Theres a ton of visual regression testing tools. I looked at:
- Selenium + other tools (webdriver, phantomcss)
- Gemini + browserstack (or sauce labs)
When I’m looking at new tech and deciding whether or not to use it, I always think about these things:
- Does its benefit outweigh the cost of having to learn it?
- What does its future look like? (community support, funding, key players using it, etc)
- How easy is it to implement into my current workflow?
- large, incredibly active community
- If you use Gulp, you can plug it right in. If not, you can run it easily via NPM.
What’s in BackstopJS?
PhantomJS - headless browser = a browser without a GUI. Super useful for doing browser-y things like rendering a DOM, styling elements, doing AJAX requests, etc, all via the command line.
CasperJS - scripting tool that allows us to interact with the page rendered by PhantomJS just like a user would: moving a mouse, performing clicks, entering text into input fields, and any other user actions you define on it.
ResembleJS - the actual “diffing” tool itself. It compares two images and can determine pixel differences between them.
The following instructions is using the Gulp flavor of BackstopJS. If you’re interested in keeping it agnostic, roll over to BackstopJS for the official documentation.
1. Install BackstopJS
npm install backstopjs
2. Move to BackstopJS' node_module folder
Backstop makes you go into
node_modules/backstopjs and to run its scripts currently. This is the one strange thing I've found about BackstopJS, but upon speaking to one of its core maintainers, it is currently being worked on to be phased out. In the meantime, run this command:
3. Create a config file
while still inside of
npm run genConfig.
NOTE: If you wish to use the example provided, you DO NOT need to run this command, as I’ve created a simple
backstop.json file for you.
This command produces a
backstop.json file, which gives you a file of options for you to configure: viewports to target, selectors to target for tests, names/paths for test/reference screenshot results, etc. This is where you write your tests, and if you have any CasperJS scripts, they go in
NOTE: in the example backstop.json file, I have set up the Baseline screenshots as:
the would-be development version is listed as:
for the above
url value, include your own path to the project.
4. Take reference screenshots
to create reference screenshots, run this command:
npm run reference
Reference screenshots are put in a generated folder:
5. Test new screenshots against baseline screenshots
To create test screenshots to diff against the reference screenshots you've already created, run this:
npm run test
Test screenshots are put in a generated folder:
npm run test also generates a test results summary page that automatically pops up in your bowser to show you what tests have passed/failed, reference,test, and diff screenshots of each test, in addition to the config options for each test.
As long as your baseline screenshots are taken, you may run
gulp test and test to your heart’s content.
NOTE: This is for manually testing at the moment, but this repo will be updated with Travis-CI integration in the short future.
npm install backstopjs
npm run genConfig (NOTE: only use this if you DON'T want to use the created backstop.json config file & want your own)
npm run reference (to create reference screenshots)
npm run test (to create screenshots that test against your reference screenshots)
Interference from dynamic content
Any data that updates/changes (ex: newsfeeds) will cause a false positive. How do you get around this problem?
- Identify the dynamic content and target it.
- Prior to every visual reg. test, use CasperJS to evaluate some JS that hides the dynamic content.
- Set a fixed height to each parent element, so adding/hiding a piece of content with each container wont cause tests to fail due to container expansion/shrinkage. This leaves the UI with just fixed outlines of the containers themselves.
Name your screenshots. Initially, screenshots are named
screenshot_0.pngwhich isn’t too descriptive, so include a name so you know what you’re actually looking at.
Don’t try to test everything. Try to maximize coverage with fewer tests. This is difficult because you want to know everything that is going on but the more granular your tests are, the easier it is to track down the actual bugs causing the problem.
Full page screenshots are a bad idea: keep your tests modular or it will get annoying. Also, bigger screenshots mean longer processing time.
Scaling visual regression testing: If you’re using git to store your baseline images (the initial, ‘correct’ screenshots) and your project becomes increasingly larger, you’ll be better served to use a tool like git annex or git media to prevent your repo growing way too huge.
Got a pattern library? make those elements/components your reference screenshots. If your team has an established pattern library, writing tests that map to your individual components can make your life much, much easier.
Visual regression testing