Fixes for Windows test failures on appveyor #5922

janschulz · Jan 26, 2016

Fixes the windows test errors as seen on appveyor and makes appveyor test failures into build failures.

Closes: #5748

Probably interesting, but I think this file is not actually related to this PR?

That's handy. There's also tools/test_triage.py that does something similar in PyQt4 in case you're not aware of it.

yep, at least in should not belong to that commit... Also needed a fix for the image comparison decorator test images ...

I left that in for the moment, should I add it or remove it? if "keep": in a different PR? I would run this script on failure on appveyor so that the html file is included in the result_images.zip, which is IMO nice for a quick overview...

janschulz · Jan 26, 2016

The image comparison tests for some the failures are IMO to sensitive, at least I can't see much of a difference in that images.

The only "real" difference is the font_styles, which is simply empty (-> all text missing)?

janschulz · Jan 26, 2016

Current output in my local py35 test run:

[matplotlib_build] λ python tests.py
......KKK.K.KK...SS.SS.......SS...SS..SS.SS.SS.....SS.SS.SS.SS...SS..........................SS.SS.SS.SS...SS.SS.SS.SS.SS.SS.SS.SS.SS...SSC:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\dateutil\rrule.py:833: DeprecationWarning: fractions.gcd() is deprecated. Use math.gcd() instead.
  i_gcd = gcd(self._interval, base)
..SS.SS.SS.SS.SS.SS.....SS.SS.SS..SS.SS.SS.SS.SS.SS.SS.SS.SS......SS........SS.SS.SS.SS......SS...SS.SS.SS.SS.SS.SS..SS.SS.SS...SS...........SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.....SS...SS..SS.SS.SS.SS.............SS.SS.SS.SS.SS.SS.SS.SS.SS....SS.......................................SS.......................SS....SSS.Der Befehl "gs" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
Der Befehl "gs" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
Der Befehl "gs" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
Der Befehl "gs" ist entweder falsch geschrieben oder
konnte nicht gefunden werden.
SSKKK....c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1314: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to Helvetica
  (prop.get_family(), self.defaultFamily[fontext]))
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=Helvetica:style=normal:variant=normal:weight=400:stretch=normal:size=medium. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\pdfcorefonts\Times-BoldItalic.afm
  UserWarning)
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=Helvetica:style=normal:variant=normal:weight=400:stretch=normal:size=large. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\pdfcorefonts\Times-BoldItalic.afm
  UserWarning)
.............KKKKKKKKKKKKSSS.SS......K..SS.SS.SS.SS..........................SS..SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS..............c:\data\external\pydata\matplotlib\lib\matplotlib\collections.py:364: UserWarning: Collection picker None could not be converted to float
  % self._picker)
....SS........................................c:\data\external\pydata\matplotlib\lib\matplotlib\colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 4
  colors = np.asarray(colors)[igood]
c:\data\external\pydata\matplotlib\lib\matplotlib\colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3
  colors = np.asarray(colors)[igood]
.........................C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\dateutil\rrule.py:833: DeprecationWarning: fractions.gcd() is deprecated. Use math.gcd() instead.
  i_gcd = gcd(self._interval, base)
C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\dateutil\rrule.py:719: DeprecationWarning: fractions.gcd() is deprecated. Use math.gcd() instead.
  for j in range(rep_rate // gcd(interval, rep_rate)):
C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\dateutil\rrule.py:751: DeprecationWarning: fractions.gcd() is deprecated. Use math.gcd() instead.
  for j in range(0, rep_rate // gcd(interval, rep_rate)):
.....K.K.....K.....K.....K.....K.....K.....K.....K.....K.....K......S.SS.SS..............S...SS.SS.SS.SS.SSSS.SSSS.SSSS.K......K.............SS.SS..SS.SS.SS.SS..SS.SS.SS...SS..SS...................................................................................................................................................................................................................................................................................................................SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...SS..SS.SS.SS.SS.SS.....SS.SS......FSS.SS.SS.SSK...............................
Bad key "   t   i   m   e   z   o   n   e   " on line 1 in
c:\data\external\pydata\matplotlib\lib\matplotlib\tests\test_utf32_be_rcparams.rc.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution
........................................................................................SS...SS.SS.SS.SS.SS.SS.SS.SS.SS.....SS.SS.SS..SS.SS.SS...........SS........SS.....SS.c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1314: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans

  (prop.get_family(), self.defaultFamily[fontext]))
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=DejaVu Sans:style=normal:variant=normal:weight=400:stretch=normal:size=14.0. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\ttf\STIXSizOneSymBol.ttf
  UserWarning)
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=DejaVu Sans:style=normal:variant=normal:weight=700:stretch=500:size=14.0. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\ttf\STIXSizOneSymBol.ttf
  UserWarning)
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1314: UserWarning: findfont: Font family ['sans serif'] not found. Falling back to DejaVu Sans
  (prop.get_family(), self.defaultFamily[fontext]))
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=DejaVu Sans:style=italic:variant=normal:weight=750:stretch=500:size=14.0. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\ttf\STIXSizOneSymBol.ttf
  UserWarning)
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=DejaVu Sans:style=normal:variant=normal:weight=200:stretch=500:size=14.0. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\ttf\STIXSizOneSymBol.ttf
  UserWarning)
c:\data\external\pydata\matplotlib\lib\matplotlib\font_manager.py:1324: UserWarning: findfont: Could not match :family=DejaVu Sans:style=normal:variant=normal:weight=500:stretch=100:size=14.0. Returning C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\mpl-data\fonts\ttf\STIXSizOneSymBol.ttf
  UserWarning)
FSS.SS.SS.............................................................................................................................................................................SS.SS.SS.SS.SS.SS.SS.SS.SS.SS.SS................SS........................................c:\data\external\pydata\matplotlib\lib\matplotlib\type1font.py:332: PendingDeprecationWarning: generator 'Type1Font._transformer' raised StopIteration
  list(map(buffer.write, transformed))
.....................S......C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  inspect.getargspec(func)
S...SS.SS.SS.SS.SS.SS.SS.SS.SS...SS.SS.SS.SS.SS.....SS....
======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: c:\data\external\pydata\matplotlib\result_images\test_patheffects\collection.png vs. c:\data\external\pydata\matplotlib\result_images\test_patheffects\collection-expected.png (RMS 0.010)

======================================================================
FAIL: matplotlib.tests.test_text.test_font_styles.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: c:\data\external\pydata\matplotlib\result_images\test_text\font_styles.png vs. c:\data\external\pydata\matplotlib\result_images\test_text\font_styles-expected.png (RMS 16.380)

----------------------------------------------------------------------
Ran 5967 tests in 422.857s

FAILED (KNOWNFAIL=37, SKIP=1208, failures=2)

janschulz · Jan 27, 2016

Ok, here are the tests from appveyor (which fail the build now):

TARGET_ARCH=x86, CONDA_PY=27

====================================================================== 
FAIL: matplotlib.tests.test_axes.test_specgram_angle_freqs.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close:
C:\projects\matplotlib\result_images\test_axes\specgram_angle_freqs.png vs. C:\projects\matplotlib\result_images\test_axes\specgram_angle_freqs-expected.png 
(RMS 0.002) 

====================================================================== 
FAIL: matplotlib.tests.test_patches.test_wedge_range.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patches\wedge_range.png vs. C:\projects\matplotlib\result_images\test_patches\wedge_range-expected.png 
(RMS 0.059) 

====================================================================== 
FAIL: matplotlib.tests.test_patheffects.test_collection.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patheffects\collection.png vs. C:\projects\matplotlib\result_images\test_patheffects\collection-expected.png 
(RMS 0.008) 

---------------------------------------------------------------------- 
Ran 5966 tests in 588.929s 

FAILED (KNOWNFAIL=40, SKIP=1210, failures=3)

TARGET_ARCH=x64, CONDA_PY=27

====================================================================== 
FAIL: matplotlib.tests.test_patches.test_wedge_range.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patches\wedge_range.png vs. C:\projects\matplotlib\result_images\test_patches\wedge_range-expected.png 
(RMS 0.059) 

====================================================================== 
FAIL: matplotlib.tests.test_patheffects.test_collection.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patheffects\collection.png vs. C:\projects\matplotlib\result_images\test_patheffects\collection-expected.png
 (RMS 0.012) 

---------------------------------------------------------------------- 
Ran 5966 tests in 411.283s 

FAILED (KNOWNFAIL=40, SKIP=1210, failures=2)

TARGET_ARCH=x64, CONDA_PY=34

======================================================================
FAIL: matplotlib.tests.test_patches.test_wedge_range.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patches\wedge_range.png vs. C:\projects\matplotlib\result_images\test_patches\wedge_range-expected.png
 (RMS 0.059)

======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patheffects\collection.png vs. C:\projects\matplotlib\result_images\test_patheffects\collection-expected.png 
(RMS 0.012)

----------------------------------------------------------------------
Ran 5966 tests in 494.700s

FAILED (KNOWNFAIL=40, SKIP=1210, failures=2)

TARGET_ARCH=x64, CONDA_PY=35

======================================================================
FAIL: matplotlib.tests.test_axes.test_specgram_freqs.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close:
 C:\projects\matplotlib\result_images\test_axes\specgram_freqs.png vs. C:\projects\matplotlib\result_images\test_axes\specgram_freqs-expected.png 
(RMS 0.042)

======================================================================
FAIL: matplotlib.tests.test_axes.test_specgram_freqs.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_axes\specgram_freqs_linear.png vs. C:\projects\matplotlib\result_images\test_axes\specgram_freqs_linear-expected.png
 (RMS 0.042)

======================================================================
FAIL: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_axes\specgram_magnitude_freqs.png vs. C:\projects\matplotlib\result_images\test_axes\specgram_magnitude_freqs-expected.png
 (RMS 0.042)

======================================================================
FAIL: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_axes\specgram_magnitude_freqs_linear.png vs. C:\projects\matplotlib\result_images\test_axes\specgram_magnitude_freqs_linear-expected.png 
(RMS 0.042)

======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_patheffects\collection.png vs. C:\projects\matplotlib\result_images\test_patheffects\collection-expected.png 
(RMS 0.006)

======================================================================
FAIL: matplotlib.tests.test_triangulation.test_tri_smooth_gradient.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: 
C:\projects\matplotlib\result_images\test_triangulation\tri_smooth_gradient.png vs. C:\projects\matplotlib\result_images\test_triangulation\tri_smooth_gradient-expected.png 
(RMS 0.014)

----------------------------------------------------------------------
Ran 5966 tests in 517.163s

FAILED (KNOWNFAIL=40, SKIP=1210, failures=6)

mdboom · Jan 27, 2016

The specgram tests are largely testing numpy, and I don't know how strict they are on ensuring reproducible results... In general, we can probably increase the tolerance on these specific tests (or, if we want to be more careful, only increase the tolerance on Windows)...

janschulz · Jan 27, 2016

BTW, this seems to be out of sync:

def image_comparison(baseline_images=None, extensions=None, tol=0,
                     freetype_version=None, remove_text=False,
                     savefig_kwarg=None, style='classic'):
[...]
      *tol*: (default 13)
        The RMS threshold above which the test is considered failed.

mdboom · Jan 27, 2016

BTW, this seems to be out of sync:

Indeed it is. I'll open another PR for that.

janschulz · Jan 27, 2016

There is also #5647, which lists these two tests as well:

test_axes:test_specgram_freqs
test_axes:test_specgram_magnitude_freqs
[ ]: matplotlib.tests.test_axes.test_specgram_freqs.test (RMS 0.042) (x64,35)
[ ]: matplotlib.tests.test_axes.test_specgram_freqs.test (RMS 0.042) (x64,35)
[ ]: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test (RMS 0.042) (x64,35)
[ ]: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test (RMS 0.042) (x64,35)

-> currently set to 0.03, set tolerance to 0.05 on windows -> done

[ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.006) (x64,35)
[ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.008) (x86,27)
[ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.012) (x64,27)
[ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.012) (x64,34)

This has a black diff, so up the tolerance on windows to 0.013 -> done

[ ]: matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x64,27)
[ ]: matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x64,34)
[ ]: matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x86,27)

This looks actually interesting: it seems that only the middle figure in the last row is different (at least the 32bit, py2.7 which I looked at)

-> also done...

[ ]: matplotlib.tests.test_axes.test_specgram_angle_freqs.test (RMS 0.002) (x86,27)

Also looks black, but only on py27/x86...? -> done

[ ]: matplotlib.tests.test_triangulation.test_tri_smooth_gradient.test (RMS 0.014) (x64,35)

-> done

janschulz · Jan 27, 2016

https://travis-ci.org/matplotlib/matplotlib/jobs/105255492

======================================================================
FAIL: matplotlib.tests.test_mathtext.test_mathtext_cm_48.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/matplotlib/matplotlib/venv/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/travis/build/matplotlib/matplotlib/lib/matplotlib/testing/decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "/home/travis/build/matplotlib/matplotlib/lib/matplotlib/testing/decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: /home/travis/build/matplotlib/matplotlib/result_images/test_mathtext/mathtext_cm_48_svg.png vs. /home/travis/build/matplotlib/matplotlib/result_images/test_mathtext/mathtext_cm_48-expected_svg.png (RMS 4.763)

[I really hate that the image comparison tests do not show the line number but only the decorator code path :-(]

[Edit: seems to be a fluke, at least the next travis build didn't see that error?]

janschulz · Jan 27, 2016

And locally I still have the problem with

======================================================================
FAIL: matplotlib.tests.test_text.test_font_styles.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "c:\data\external\pydata\matplotlib\lib\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close:
 C:\data\external\pydata\matplotlib\result_images\test_text\font_styles.png vs. C:\data\external\pydata\matplotlib\result_images\test_text\font_styles-expected.png
 (RMS 16.380)

janschulz · Jan 27, 2016

I will open a new issue for my local test failure, that looks like the fallback font doesn't work on my system.

[Edit: the problem was a stale font cache which prevent this test to find the fallback fonts... See #5929 for the story... Note to self: before writing issues, use a debugger...]

janschulz · Jan 27, 2016

Ok, the builds in my appveyor instance already build the two py2.7 versions of the tests and they are green: https://ci.appveyor.com/project/JanSchulz/matplotlib/build/1.0.135

janschulz · Jan 27, 2016

Open question: should I remove the visual_tests.py file?

janschulz · Jan 27, 2016

And it's green again (in my build) :-)

janschulz · Feb 1, 2016

@tacaswell @mdboom Should the tests for which I changed the tolerance be added to #5647?

jenshnielsen · Feb 2, 2016

@janschulz Yes it would be good to collect the information about non zero tolerance in #5647

janschulz · Feb 2, 2016

@jenshnielsen ok, done directly in the first comment as additional Todos: #5647

mdboom added the needs_review label Jan 26, 2016

mdboom referenced this pull request Jan 27, 2016
Merged
Update docstring to match kwarg default #5924

janschulz referenced this pull request Jan 27, 2016
Closed
Fallback font doesn't work on windows? #5929

tacaswell added a commit that referenced this pull request Feb 1, 2016

tacaswell Merge pull request #5922 from JanSchulz/fix_tests_on_windows
Fixes for Windows test failures on appveyor
bbe421a

tacaswell merged commit bbe421a into matplotlib:master Feb 1, 2016
2 checks passed

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details

continuous-integration/travis-ci/pr The Travis CI build passed
Details

tacaswell removed the needs_review label Feb 1, 2016

janschulz referenced this pull request Feb 1, 2016
Open
Some tests require non-zero tolerance #5647
0 of 20 tasks complete

QuLogic added this to the 2.1 (next point release) milestone Feb 1, 2016

This was referenced Feb 17, 2016

Merged

Rewrite of image infrastructure #5718

Merged

Remove wrongly commited test.txt #6388

Open

CI: add appveyor script to build Windows wheels #6969

janschulz	`TST: skip test_dviread if kpsewhich is not available` Also fix the docstring to mention that it is part of miktex...	`915590f`
janschulz	`TST: if no converter is found, skip the test`	`581c231`
janschulz	`TST: some tex tests also need to guard against missing gs` Also fixed a problem in an error message when bytes were appended to a string (fails on py3.x) by using %. One of the errors before: ====================================================================== ERROR: matplotlib.tests.test_backend_ps.test_savefig_to_stringio_with_usetex ---------------------------------------------------------------------- Traceback (most recent call last): File "lib\site-packages\nose\case.py", line 198, in runTest self.test(self.arg) File "lib\matplotlib\testing\decorators.py", line 152, in wrapped_callable func(args, *kwargs) File "lib\matplotlib\testing\decorators.py", line 55, in failer result = f(args, *kwargs) File "lib\matplotlib\tests\test_backend_ps.py", line 77, in test_savefig_to_stringio_with_usetex _test_savefig_to_stringio() File "lib\matplotlib\tests\test_backend_ps.py", line 40, in _test_savefig_to_stringio fig.savefig(buffer, format=format) File "lib\matplotlib\figure.py", line 1698, in savefig self.canvas.print_figure(args, kwargs) File "lib\matplotlib\backend_bases.py", line 2232, in print_figure kwargs) File "lib\matplotlib\backends\backend_ps.py", line 985, in print_ps return self._print_ps(outfile, 'ps', args, kwargs) File "lib\matplotlib\backends\backend_ps.py", line 1012, in _print_ps *kwargs) File "lib\matplotlib\backends\backend_ps.py", line 1376, in _print_figure_tex rotated=psfrag_rotated) File "lib\matplotlib\backends\backend_ps.py", line 1539, in gs_distill raise RuntimeError(m % output) RuntimeError: ghostscript was not able to process your image. Here is the full report generated by ghostscript: b''	`f1b52c4`
janschulz	`TST: add helper script to compare images in the browser` This is a script which I developed for ggplot https://github.com/yhat/ggplot/blob/master/visual_tests.py The script will go over the images in result_images and add them to a webpage, sorted by test files. At the top, there will be the test failures, so you can inspect the visually and try to figure out what's wrong with matplotlib (or the tests...) :-)	`82d42eb`

		janschulz	`CI: use visual_tests.py after a test failure`		All checks have passed 2 successful checks continuous-integration/appveyor/pr — AppVeyor build succeeded Details continuous-integration/travis-ci/pr — The Travis CI build passed Details	`f1800af`
		janschulz	`CI: let failing test fail the appveyor build`		Some checks were not successful 1 failing and 1 successful checks continuous-integration/appveyor/pr — AppVeyor build failed Details continuous-integration/travis-ci/pr — The Travis CI build passed Details	`eff6bb0`

matplotlib/matplotlib

Fixes for Windows test failures on appveyor #5922

janschulz commented Jan 26, 2016

mdboom added the needs_review label Jan 26, 2016

QuLogic commented on the diff Jan 26, 2016

janschulz commented Jan 26, 2016

janschulz added some commits Jan 26, 2016

janschulz commented Jan 26, 2016

janschulz added some commits Jan 26, 2016

janschulz commented Jan 27, 2016

mdboom commented Jan 27, 2016

janschulz commented Jan 27, 2016

mdboom commented Jan 27, 2016

mdboom referenced this pull request Jan 27, 2016

Update docstring to match kwarg default #5924

janschulz added some commits Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz added some commits Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz referenced this pull request Jan 27, 2016

Fallback font doesn't work on windows? #5929

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

tacaswell added a commit that referenced this pull request Feb 1, 2016

tacaswell merged commit `bbe421a` into matplotlib:master Feb 1, 2016
2 checks passed

2 checks passed

tacaswell removed the needs_review label Feb 1, 2016

janschulz commented Feb 1, 2016

janschulz referenced this pull request Feb 1, 2016

Some tests require non-zero tolerance #5647

QuLogic added this to the 2.1 (next point release) milestone Feb 1, 2016

jenshnielsen commented Feb 2, 2016

janschulz commented Feb 2, 2016

This was referenced Feb 17, 2016

Rewrite of image infrastructure #5718

Remove wrongly commited test.txt #6388

CI: add appveyor script to build Windows wheels #6969

		@@ -0,0 +1,88 @@
		+#!/usr/bin/env python

matplotlib/matplotlib

Join GitHub today

Fixes for Windows test failures on appveyor #5922

Conversation

janschulz commented Jan 26, 2016

mdboom added the needs_review label Jan 26, 2016

QuLogic commented on the diff Jan 26, 2016

janschulz commented Jan 26, 2016

janschulz added some commits Jan 26, 2016

janschulz commented Jan 26, 2016

janschulz added some commits Jan 26, 2016

All checks have passed

Some checks were not successful

janschulz commented Jan 27, 2016

mdboom commented Jan 27, 2016

janschulz commented Jan 27, 2016

mdboom commented Jan 27, 2016

mdboom referenced this pull request Jan 27, 2016

janschulz added some commits Jan 27, 2016

Some checks were not successful

All checks have failed

janschulz commented Jan 27, 2016

janschulz added some commits Jan 27, 2016

All checks have passed

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz referenced this pull request Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

janschulz commented Jan 27, 2016

tacaswell added a commit that referenced this pull request Feb 1, 2016

Hide details View details tacaswell merged commit bbe421a into matplotlib:master Feb 1, 2016 2 checks passed

2 checks passed

tacaswell removed the needs_review label Feb 1, 2016

janschulz commented Feb 1, 2016

janschulz referenced this pull request Feb 1, 2016

QuLogic added this to the 2.1 (next point release) milestone Feb 1, 2016

jenshnielsen commented Feb 2, 2016

janschulz commented Feb 2, 2016

This was referenced Feb 17, 2016

tacaswell merged commit `bbe421a` into matplotlib:master Feb 1, 2016
2 checks passed