Proposal: Visual Regression / Screenshot Diffing

[RandallKent]

We probably will not do this as a built-in feature in Cypress just yet. Instead please consider using open source plugins or commercial systems that built on top of Cypress test runner.

Here is our documentation guide: https://on.cypress.io/visual-testing

Find image diffing plugins in https://on.cypress.io/plugins#visual-testing

[ezeedijk]

+1 on this feature request! We're now using blink-diff to compare screenshots generated by Cypress for visual regression testing, but having this integrated would be much better.

[rmharrison]

@ezeedijk: These guys do continuous visual integration (pixel perfect), but not functional BDD/TDD: https://percy.io/

Pro: Have out of the box plugins for popular MVC frameworks, e.g. ember-percy
Con: No free tier. Starts at $150/month after 2-week trial.

Seems like visual regression would go along great with Cypress, esp. given the dashboard. Functional test + visual diff all in one dashboard.

[rndmerle]

@ezeedijk , if you don't mind I've some questions.
Are you using blink-diff with an assertion right into a Cypress test?
How are you handling the validation? (updating the reference screenshot)

[leonardoanalista]

Image diff would be a great feature and make cypress a solid alternative to phantomCSS/Casper for css regression testing.
Please ensure we can compare portions of the screen and exclude page sections too.

[ezeedijk]

@rndmerle No we're using blink-diff in our CI after we've run our Cypress tests. We've simply created three folders:

• 'screenshots' - new Cypress output
• 'screenshots_base' - set of correct screens for blink-diff to compare against
• 'screenshots_diff' - where blink-diff outputs the files in case of error

Updating the base set of screenshots in case of changes is a manual step. We use it very limited just to discover changes in parts of our application that are very difficult to assert other than visually, in our case mainly graphs. We do still have stability issues in some tests due to the a-synchronise nature of the cy.screenshots. So it would be nice to have visual assertions directly into Cypress.

[leonardoanalista]

@ezeedijk can you take a screenshot from part of the screen like phantomCSS? Example, from body or #contents.

[brian-mann]

Yes. We control the DOM. We can basically do anything we want prior to taking the screenshot. It would be trivial to "blackout" all the other areas, or even calculate the box model for the thing you want to screenshot and then resize the image accordingly.

[eskitek]

We currently use PhantomCSS for visual regression testing. We use it in combination with PhantomFlow, which provides very nice visual diff presentation and the ability to replace baseline images easily.

Usage: We take targeted screenshots at various points during our functional test runs. We don't use screenshots to replace functional assertions per se, but we do take them within functional test flows to verify the visual state of sections of the page under various conditions. Or goal is to screenshot every screen element once under each of its states. Some functional tests may have several screenshots taken within them, others may have none.

Overall, we like the functionality of PhantomCSS a lot, but we are looking for a replacement that doesn't require PhantomJS. We are excited to see what Cypress will do in this space. Some things we'd like to see:

• Ability to screenshot different parts of the screen by selector (xpath, ID, etc)
• Ability to take screenshots during a test run at whatever point we choose to
• Ability to easily visualize which pixels don't match between 2 images
• Ability to replace the baseline image easily (Example: PhantomFlow provides a "rebase" button. See https://github.com/Huddle/PhantomFlow#rebasing-visual-tests)
• Ability to set threshold for percentage mismatch that will trigger a failure

[garris]

Hi guys, I am the maintainer of BackstopJS. I wonder if there is some kind of interesting integration that could happen here? Happy to chat if there is some interest.

[bahmutov]

@ezeedijk we have a proof of concept using blink-diff and that is very nice workflow - and with some UX love could be game changer. Stay tuned!

[zth]

@bahmutov I was very intrigued by your talk of your concept - any update on that? I'm looking to do the same thing basically.

[rsudarson]

It would be really nice if cypress supports this feature sooner.

[maxime1992]

Me when I receive plenty of +1 notifications in 2018

.	.

This is not helping guys, just disturbing for the whole discussion.
Use reactions...

... add something usefull
... or make a PR 😅

I mean, if the topic was dying I might understand, trying to get attention.

But with Gleb saying:

@ezeedijk we have a proof of concept using blink-diff and that is very nice workflow - and with some UX love could be game changer. Stay tuned!

It does not seem that we're being abandoned 😉

[julianburr]

hey @bahmutov, was just wondering since you seem to be actively working on this feature but the help wanted label is still on, if there is anything we could help with? 😊

Fwiw I quickly threw something together (https://github.com/julianburr/cypress-match-screenshot) cause I needed to do some screen diffing for a project, but that’s obv just a temporary solution. Really looking forward to proper official support for matching screenshots.

Btw, thanks so much for your work in general. Been working with Cypress for the past couple of months and its been amazing 🎉🙏

[bahmutov]

@julianburr it looks good, we will discuss this as soon as I get a little bit of time - we made the same thing, the pixel comparison is easy, the harder part is coming up with the entire workflow. Like where do you store images, how do you approve changes, etc.

[egucciar]

@bahmutov if you release an API for it let us know. I need to adjust what @julianburr did to work in CI and not sure if we have the time for it.

[maple-leaf]

we recently try doing visual regression with cypress + cypress-image-snapshot, But there are one main issue seems hard to resolve: the screenshot persist is not guaranteed caused by browser size not in control.

1. if you run test with cy:open, then the window size may be resized by you, resized by opening devtool, or run test on different screen you are using.
2. let's say you try your best to control window size persist every time you run test, which obviously not possible and not convenient and worst for team work, you generate your base visual snapshots with cy:open, then you run it with cy:run on CI, the test will fail caused by different browser size between cy:run and cy:open.

Is anyone has solution for this?
So far, We are trying to use puppeteer to do screenshot do instead of cypress itself. The puppeteer will be launched by cy.task, and do its jobs and return result. The benefits of this way:

1. enough control of size and behavior of browser
2. cypress app and puppeteer is isolated, they will not have chance to affect each others.

[anatoliyarkhipov]

@maple-leaf Why the size is different? Did you try to define it explicitly via cy.viewport()? 🤔

[maple-leaf]

@anatoliyarkhipov Not the viewport problem, the viewport is being set at cypress.json and nerver changed.
when you take snapshot, the real window size matter.

First we should stick to one concept, when take screenshot, we only take the real part we care, which is red area in image below.

And this area's size will affected by whole window's size, devtool position and size which will produce diffs.
Check the diff image below, the diff is caused by content overflow when browser's size is not enough while the actual rendered content are not different at all.

[maple-leaf]

Find solution, you can control chrome or electron window size from plugin/index.js

    on('before:browser:launch', (browser = {}, args) => {
        if (browser.name === 'chrome') {
            args.push('--window-size=200,800');

            // whatever you return here becomes the new args
            return args
        }

        if (browser.name === 'electron') {
            args['width'] = 1600
            args['height'] = 900
            args['resizable'] = false;

            // whatever you return here becomes the new args
            return args
        }
    })

[san-slysz]

Is there a way (like an early release, or a branch to clone) to test this ?
Even without the slack & dashboard part it could be nice to try it and do an early qa/feedback. I think a lot of us would be interested in helping this way :). If that's not enough baked yet for a alpha/beta of this sort, no worries, I'll wait.

[callumacrae]

@dimitriy-k have you read the issue or just the title?

[bahmutov]

Official Workaround

For now we do not plan to offer built-in visual regression feature in Cypress, but we did evaluate 2 commercial 3rd party services (both have nice free monthly plans) and played with Do-It-Yourself visual plugins. We have documented examples, conducted live webinars and posted our slides on the new documentation https://on.cypress.io/visual-testing

[proCrichard]

@bahmutov I think documentation will need an update with this issue closed.
https://docs.cypress.io/guides/guides/screenshots-and-videos.html#Visual-Regression-Test-Screenshot-Diffing

[avkonst]

Hi guys, I am the maintainer of BackstopJS. I wonder if there is some kind of interesting integration that could happen here? Happy to chat if there is some interest.

Hey, has any integration actually happened? How could I use the backstopjs engine to do screenshot testing form me?

[jennifer-shehane]

@proCrichard Thanks. Will be addressed in this PR. cypress-io/cypress-documentation#1757

[justinlevi]

Hi guys, I am the maintainer of BackstopJS. I wonder if there is some kind of interesting integration that could happen here? Happy to chat if there is some interest.

Hey, has any integration actually happened? How could I use the backstopjs engine to do screenshot testing form me?

@avkonst I've been a user of Backstopjs for a while and would LOVE to see a collaboration here.

[garris]

@justinlevi @avkonst Just an update. I have just completed ember-backstop, a test-helper addon for ember apps. The Ember stack uses qunit -- but this same exact technique would work perfectly for cypress as a script helper...

Example here -- the addon...
https://github.com/garris/ember-backstop

and here for a tutorial...
https://github.com/garris/ember-backstop-tutorial

Someone should be able to take this script (which is based on Percy.io approach)...
https://github.com/garris/ember-backstop/blob/master/addon-test-support/backstop.js
...and convert it to a helper script for cypress -- translating the various DOM grabbing and Ember related conventions.

I probably don't have time to do this myself -- but I am happy to provide tips and feedback to anyone who wants to take this on. It should be pretty straightforward and it will likely address many of the issues experienced with earlier attempts.

That's all I got. Sorry for the long silence. Been heads-down on developments closer to home for the past two years 😄

Best wishes everyone.

[mstruensee]

+1 for backstopjs

[adamchenwei]

backstopJs is pretty rigid and hard for any sort of debugging and not very actively maintained, highly not recommended.

[mstruensee]

meh I got it working with the maintainers suggestion above... but I feel these 2 concepts should be kept separate... let backstop do backstop and Cypress do Cypress ... I didn't start using Cypress for visual testing and I agree with their choice not to include one

backstopjs is one of the best visual comparison tools ... maintained by original creator or not ... it's open source... contribute 😉

[dimitriy-k]

I use cypress-image-snapshot in combination with Cypress, works perfect

[cdeutsch]

Just finished trying out BackstopJs.

Wasn't a super fan of all the JSON based config (already switched to using TypeScript).

Also driving your tests by configuring them seemed limiting at first, (instead of using puppeteer directly), but haven't found something I can't do yet.

The real win with BackstopJS is it can easily be run in Docker, which makes it the best, OSS/Free solution I've found for my needs.

I can get rock solid comparisons to run every time on Linux and Mac hosts, unlike every other library I've tried.
(one big reason comparisons are flakey is that running them on macOS directly takes into consideration my retina vs non-retina display, forcing me to chose when and where I can run the tests)

I'm going to talk to my boss about publishing the API mocking code I wrote, which makes it easy to turn dynamic data into static.

Thanks for the recommendation!!

...sorry for the off topic, but I've been trying for over a year to get reliable visual diffing tests, and finally found a solution.❤️

UPDATE: here's the Puppeteer API mocking library I mentioned:
https://github.com/axiomhq/puppeteer-request-intercepter

[jdubbski]

@ezeedijk: These guys do continuous visual integration (pixel perfect), but not functional BDD/TDD: https://percy.io/

Pro: Have out of the box plugins for popular MVC frameworks, e.g. ember-percy
Con: No free tier. Starts at $150/month after 2-week trial.

Seems like visual regression would go along great with Cypress, esp. given the dashboard. Functional test + visual diff all in one dashboard.

looks like now there's a free version (10 users + 5k snapshots). giving it a try now.