Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Varying results depending on freetype version #8796
This is probably not really a bug, but more a huge annoyance caused by the freetype version used by matplotlib.
For a visualization of the issue, you can take a look at this commit:
freetype 2.7.0 changed the way fonts are rendered, which is most likely the root cause of the issue.
The difference is big enough to fail our ImageComparison tests. To make matters worse, even if we disable rendering text for our tests, the freetype version still affects the overall layout of the graph enough to fail a lot of our tests.
There seems to be no good way to detect the used freetype version, specially as matplotlib from pip can come with its own bundled version that's independent from the system one.
Installed via pip install --no-binary :all:
The way we handle this is there is a 'blessed' version we use on travis (and can optionally build as part of the Matplotlib install process) which lets us use 0-tolerance tests. You can put tolerances on comparing images, but picking a global threshold is basically impossible (we used to have a non-zero threshold and the tests kept passing despite completely wrong text coming out in some cases, a 1px shift generates about the same number of 'wrong' pixel as swapping out a glyph).
You might see if switching to svg, pdf, or ps for your test images helps? Store the vector version under version control and rasterize the baseline images as-needed for the tests.
I'm already using that, and it has no effect on the issue. Even though the text is not rendered anymore, the minor layout changes still persist, and break the tests.
Nice, that's what I was looking for. So we can at least automatically detect which set of reference images to use.
So we can just output some vector format, and compare that? That sounds like the best solution, but I'm not sure of pytest-mpl supports that. It looks pretty hard-coded for png images.
referenced this issue
Jun 26, 2017
I'm not sure if pytest-mpl does this, but the matplotlib tests convert pdf and svg files to png at comparison time. The pdf conversion is just a call to Ghostscript, while the svg conversion does something interactive with Inkscape:
We would prefer not, because this is an ever moving target (freetype rendering changes even between patchlevel changes) and regenerating all the images further would increase the size of the already fairly heavy repo by ~15Mb (IIRC) every time.
I'm going to close the issue but feel free to suggest better solutions.