Use a specific version of Freetype for testing #5306

mdboom · 2015-10-23T16:31:21Z

This should theoretically allow us to turn down the tolerance on image comparison tests.

If all is going well, this should pass Travis-CI, which is now using a locally-built freetype.

jenshnielsen · 2015-10-24T07:53:24Z

setupext.py

+            urlretrieve(
+                'http://download.savannah.gnu.org/releases/freetype/{0}'.format(tarball),
+                tarball_path)
+


Could we do a hash check here to make sure that we get an uncorrupted file?

jenshnielsen · 2015-10-24T07:53:40Z

👍

QuLogic · 2015-10-24T21:39:56Z

lib/matplotlib/__init__.py

+    from matplotlib import ft2font
+    if (ft2font.__freetype_version__ != LOCAL_FREETYPE_VERSION or
+        ft2font.__freetype_build_type__ != 'local'):
+        raise ImportError(


Does that mean I can't run tests without this specific freetype installed?

Is it feasible to allow running the tests, just giving a big warning and marking all image comparisons as known-fail? That would still let people run the other tests, or look at the results of single image-comparison tests manually, while not leading them to think the test results are canonical.

@QuLogic: You don't need a specific version of freetype installed. You need to build matplotlib with the special "local_freetype" option, which will download and build against a particular version of freetype for you.

@jkseppan: Maybe a warning is the right thing to do -- but knownfail doesn't always communicate "the test run you just ran should give you very little confidence that anything is working" which is really the situation we would have here. What about a warning + real failures?

Warning + failures makes sense to me.

mdboom · 2015-10-26T20:40:23Z

The experiment in #5307 shows that there are now no text differences, even on Linux vs. Mac. There are other sorts of differences that happen when you turn the tolerance all the way down to 0, however, that I'm now trying to get to the bottom of. Some of it appears to just be random number effects that we've never noticed before with the higher tolerances.

QuLogic · 2015-10-26T22:24:35Z

BTW, which tests were different, out of curiosity? Is it mostly Linux vs Mac or even between Linux distros?

mdboom · 2015-10-26T22:51:12Z

They mostly seem to be differences from one run to the next (all other things being equal). Some things just need an explicit random seed. The others are caused by a very deep and long standing bug in the test runner -- it's possible to write a test returning multiple figures all using the same output filename. Race conditions abound! I hope to have a test for that soon.

QuLogic · 2015-10-26T22:56:51Z

Sorry, I meant which tests prompted this PR specifically?

mdboom · 2015-10-26T23:03:48Z

Oh -- different versions of Freetype have always been a problem since we've had our test suite. It's the whole reason we do "fuzzy" rather than exact matching. But the goal here is to nail down the version of Freetype such that we can do exact matching (and catch more bugs that way).

QuLogic · 2015-10-27T00:39:34Z

OK, I'd just like to know whether FreeType eventually stabilized and we could drop this once all the older distros were EOL, or whether it's a more cross-platform issue.

In testing Cartopy, I've never run into anything caused by Freetype and not changes in matplotlib, so perhaps we are all running new (or close) enough versions that it does not trigger anything.

mdboom · 2015-10-27T00:44:52Z

Freetype has stabilized somewhat in recent years, but we ran into this as recently as 2014 the last time. The cross-platform and cross-distro issues still exist though -- it can be compiled with different hinting settings etc. and this is an attempt to normalize that.

mdboom · 2015-10-27T11:27:38Z

@QuLogic: Just as an example that this is still a live problem, see #5305. @zLbZ generated a new set of images on Debian testing, and the version of Ubuntu on Travis (I think it's 12.04, maybe 14.04), with freetype 2.6.0 and 2.4.8 respectively, mismatched the images by as much as 20% (notably, these are text-heavy images).

mdboom · 2015-10-27T16:07:16Z

#5307 is now passing (except for some PEP8 stuff), so I'm convinced this approach to use a fixed version of freetype is sound. This is ready for final review.

QuLogic · 2015-10-28T00:27:48Z

Ah, I had forgotten about all the configurable hinting settings in FreeType; those are quite likely to cause a sizeable difference (even if mostly imperceptible).

QuLogic · 2015-10-28T01:23:24Z

doc/devel/testing.rst

@@ -33,6 +33,22 @@ Optionally you can install:

  - `pep8 <http://pep8.readthedocs.org/en/latest>`_ to test coding standards

+Build matplotlib for image comparison tests


The tense here is different from most of the rest of the file (though it's not all totally the same.)

jenshnielsen · 2015-10-28T09:27:09Z

Is there any particular reason to use Freetype '2.5.2' According to http://www.freetype.org/ several security fixes has gone in since. It seems better to me to use 2.5.5 or 2.6.1?

mdboom · 2015-10-28T12:37:38Z

@jenshnielsen: I started with the version that is currently on Travis, and the version many (but not all) of the current images were created with, in an attempt to avoid updating too many of the images. Alternatively, we could just move to the latest and start fresh, but that will add more stuff to the repository.

jenshnielsen · 2015-10-28T12:44:16Z

AFAIK the version on Travis is 2.4.8

from https://travis-ci.org/matplotlib/matplotlib/jobs/87692893

 freetype: yes [version 2.4.8]

We could upgrade Travis from the existing 12.04 image to the "new" beta 14.04 images to get 2.5.2

This is of cause slightly more complicated because the versions in ubuntu have backported some of the security fixes from 2.5.x

mdboom · 2015-10-28T12:51:00Z

@jenshnielsen: Ah -- that was my bad -- I just just for what 14.04 had rather than what Travis was actually using. Maybe it would be best to put this at the latest now just for longevity -- the idea should be to rarely, if ever, update this again, except for zero-visible-changes security updates.

mdboom · 2015-10-28T13:05:06Z

@QuLogic: I've addressed all of your minor points in the latest commits here.

mdboom · 2015-10-28T15:32:57Z

So, I think before merging, we should have a discussion about which version of Freetype to use here. I could be persuaded either way, I think.

mdboom · 2015-11-05T16:42:33Z

Rebased.

WeatherGod · 2015-11-05T17:14:05Z

Things look good to me. Just waiting for Travis to go green.

Use a specific version of Freetype for testing

jenshnielsen · 2015-11-05T17:33:34Z

The plan is to backport this to 2.0.x right? In that case feel free to merge #5413

mdboom · 2015-11-05T17:43:54Z

Yes. This should be merged to 2.0.x. I'll do that.

Use a specific version of Freetype for testing

mdboom · 2015-11-05T17:45:44Z

Cherry-picked to 2.0.x as d730481

mdboom added the status: needs review label Oct 23, 2015

mdboom force-pushed the local-freetype branch from f507823 to cd12a12 Compare October 23, 2015 18:58

mdboom mentioned this pull request Oct 23, 2015

TST: Lower tolerance of mathtext tests, update baseline images to master #5305

Closed

jenshnielsen reviewed Oct 24, 2015
View reviewed changes

QuLogic reviewed Oct 24, 2015
View reviewed changes

mdboom mentioned this pull request Oct 27, 2015

Lower test tolerance #5307

Merged

QuLogic reviewed Oct 28, 2015
View reviewed changes

mdboom force-pushed the local-freetype branch from 30f9c79 to e6e3d35 Compare October 28, 2015 13:04

mdboom added 16 commits November 5, 2015 11:42

Python 2 fix

733cda9

Fix setup.cfg in .travis.yml

56c686e

Generate setup.cfg in a different way

924950e

Use -fPIC

bab2d6a

Don't use special freetype in docs build

7935fa1

Verify hash of freetype tarball

ca61716

Change error to warning

662ae7d

Fix memory management revealed by using static freetype

d102d39

Amend, rather than override, CFLAGS

c2af9c6

freetype -> FreeType

5c6173a

Fix tense

6a9f3c4

Fix name of config parameter in docs

85a6e78

Make setup.py develop work

660adc6

Don't use shell to call tar

13be4e7

Use freetype 2.6.1

420cd8e

Fix capitalization

31cb9b3

mdboom force-pushed the local-freetype branch from 4e91598 to 31cb9b3 Compare November 5, 2015 16:42

jenshnielsen added a commit that referenced this pull request Nov 5, 2015

Merge pull request #5306 from mdboom/local-freetype

0a7a41d

Use a specific version of Freetype for testing

jenshnielsen merged commit 0a7a41d into matplotlib:master Nov 5, 2015

tacaswell removed the status: needs review label Nov 5, 2015

jenshnielsen added a commit to jenshnielsen/matplotlib that referenced this pull request Nov 5, 2015

Merge pull request matplotlib#5306 from mdboom/local-freetype

54b5339

Use a specific version of Freetype for testing

jenshnielsen added a commit that referenced this pull request Nov 5, 2015

Merge pull request #5306 from mdboom/local-freetype

d730481

Use a specific version of Freetype for testing

mdboom deleted the local-freetype branch November 10, 2015 02:40

mdboom mentioned this pull request Dec 2, 2015

Fix errorbar extension arrows #5586

Merged

QuLogic added the topic: testing label Sep 28, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use a specific version of Freetype for testing #5306

Use a specific version of Freetype for testing #5306

mdboom commented Oct 23, 2015

jenshnielsen Oct 24, 2015

mdboom Oct 24, 2015

jenshnielsen commented Oct 24, 2015

QuLogic Oct 24, 2015

jkseppan Oct 25, 2015

mdboom Oct 26, 2015

jkseppan Oct 26, 2015

mdboom commented Oct 26, 2015

QuLogic commented Oct 26, 2015

mdboom commented Oct 26, 2015

QuLogic commented Oct 26, 2015

mdboom commented Oct 26, 2015

QuLogic commented Oct 27, 2015

mdboom commented Oct 27, 2015

mdboom commented Oct 27, 2015

mdboom commented Oct 27, 2015

QuLogic commented Oct 28, 2015

QuLogic Oct 28, 2015

jenshnielsen commented Oct 28, 2015

mdboom commented Oct 28, 2015

jenshnielsen commented Oct 28, 2015

mdboom commented Oct 28, 2015

mdboom commented Oct 28, 2015

mdboom commented Oct 28, 2015

mdboom commented Nov 5, 2015

WeatherGod commented Nov 5, 2015

jenshnielsen commented Nov 5, 2015

mdboom commented Nov 5, 2015

mdboom commented Nov 5, 2015

		@@ -33,6 +33,22 @@ Optionally you can install:

		- `pep8 <http://pep8.readthedocs.org/en/latest>`_ to test coding standards

		Build matplotlib for image comparison tests

Use a specific version of Freetype for testing #5306

Use a specific version of Freetype for testing #5306

Conversation

mdboom commented Oct 23, 2015

jenshnielsen Oct 24, 2015

Choose a reason for hiding this comment

mdboom Oct 24, 2015

Choose a reason for hiding this comment

jenshnielsen commented Oct 24, 2015

QuLogic Oct 24, 2015

Choose a reason for hiding this comment

jkseppan Oct 25, 2015

Choose a reason for hiding this comment

mdboom Oct 26, 2015

Choose a reason for hiding this comment

jkseppan Oct 26, 2015

Choose a reason for hiding this comment

mdboom commented Oct 26, 2015

QuLogic commented Oct 26, 2015

mdboom commented Oct 26, 2015

QuLogic commented Oct 26, 2015

mdboom commented Oct 26, 2015

QuLogic commented Oct 27, 2015

mdboom commented Oct 27, 2015

mdboom commented Oct 27, 2015

mdboom commented Oct 27, 2015

QuLogic commented Oct 28, 2015

QuLogic Oct 28, 2015

Choose a reason for hiding this comment

jenshnielsen commented Oct 28, 2015

mdboom commented Oct 28, 2015

jenshnielsen commented Oct 28, 2015

mdboom commented Oct 28, 2015

mdboom commented Oct 28, 2015

mdboom commented Oct 28, 2015

mdboom commented Nov 5, 2015

WeatherGod commented Nov 5, 2015

jenshnielsen commented Nov 5, 2015

mdboom commented Nov 5, 2015

mdboom commented Nov 5, 2015