Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
This is mainly JFYI because it came up on twitter: matplotlib has a similar system in place to do unittesting on their images. It is also used in downstream packages like seaborn. The system is based on comparing raster images and compares the rasterized output of svg, tiff and ps backends to a baseline png which is included in the repo. rasterization is done with ghostscript. I suspect that the rasterize step is there because svgs can produce the same visual but have different internal representations (e.g. when plotting a point and a line, AFAIK the xml can contain point -> line and line -> point).
The workflow is:
From my experience with this:
A test looks like this:
-> tests all three images formats (no
The main part is here: https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/testing/compare.py#L268 (mpl is
CC: @hrbrmstr because twitter... :-)
Thanks for your insights Jan.
My main goal with the initial release of vdiffr is to offer a convenient UI for writing visual tests with
I chose to compare SVG files mainly for convenience. As good as svglite is, it does not offer a completely accurate rendition of R plots. But in most cases, complete accuracy is not necessary for the purpose of testing regressions. I wrote vdiffr with ggplot2 extensions in mind, which are more oriented towards data exploration than creating graphics for publication. The advantage of SVG is that I don't have to deal with tolerance.
It's certainly possible to add backends though. I like how you apply different testing strategies in one go.
@janschulz Winston's vtest uses ImageMagik compare of raster images with a tolerence threshold, seems to be more what you had in mind. See https://github.com/hadley/ggplot2/tree/master/visual_test for usage in ggplot2.
This is an old issue, but since it's still open I'll add my two cents: I have found the comparison of svg's extremely valuable. The one thing I can do with svg's that I can't do with raster images is diff the new image against the old and hunt down exactly what has changed. I do this regularly, in particular when I don't see a difference visually but vdiffr tells me the images aren't the same. I find it helpful to understand why vdiffr thinks the images are different and what in the code changed to cause those differences. With raster images, you're mostly flying blind.
Example: this is a case where the visual tests failed because changes in the calculation of axis tick locations resulted in slightly different locations for the ticks and labels.