Windows test failures on appveyor

[jankatins]

Currently, test failures do not trigger a failed build, so appveyor is still green, but there are a few test failures. I will try to get it built with the local_freetype version, but I think there are also a few other errors below:

[These are from https://ci.appveyor.com/project/mdboom/matplotlib/build/1.0.366/job/1cqc4t3s9gavr3ep]

python tests.py || cmd /c "exit /b 0"
c:\projects\matplotlib\lib\matplotlib\__init__.py:1513: UserWarning: matplotlib is not built with the correct FreeType version to run tests.  Set local_freetype=True in setup.cfg and rebuild. Expect many image comparison failures below.
  "matplotlib is not built with the correct FreeType version to run "
....KKKKKKKK...KK.KK......KK...KK..KK.KK.KK.....KK.KK.KK.KK...KK..........................KK.KK.KK.KK...KK.KK.KK.KK.KK.KK.KK.KK.KK...KK..KK.KK.KK.KK.KK.KK.....KK.KK.KK..KK.KK.KK.KK.KK.KK.KK.KK.KK......KK........KK.KK.KK.KK......KK...KK.KK.KK.KK.KK.KK..KK.KK.KK...KK...........KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.....KK...KK..KK.KK.KK.KK.............KK.KK.KK.KK.KK.KK.KK.KK.KK....KK...........................................................KK....'gs' is not recognized as an internal or external command,
operable program or batch file.
'gs' is not recognized as an internal or external command,
operable program or batch file.
'latex' is not recognized as an internal or external command,
operable program or batch file.
'latex' is not recognized as an internal or external command,
operable program or batch file.
SSSSSSKKK.................KKKKKKKKKKKKKKK.KK.....K..KK.KK.KK.KK..........................SS..KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK..............c:\projects\matplotlib\lib\matplotlib\collections.py:364: UserWarning: Collection picker None could not be converted to float
  % self._picker)
....KK......................................KK..c:\projects\matplotlib\lib\matplotlib\colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 4
  colors = np.asarray(colors)[igood]
c:\projects\matplotlib\lib\matplotlib\colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3
  colors = np.asarray(colors)[igood]
..............................K.K.....K.....K.....K.....K.....K.....K.....K.....K.....K....E.K.KK.KK............K...KK.KK.KK.KK.KKKK.KKKK.KKKK.K.....K............KK.KK..KK.KK.KK.KK..KK.KK.KK...KK..KK...................................................................................................................................................................................................................................................................................................................KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...KK..KK.KK.KK.KKFKK.....KKFKK......FKK.KK.KK.KKK.............................. 
Bad key "timezone" on line 1 in
c:\projects\matplotlib\lib\matplotlib\tests\test_utf32_be_rcparams.rc.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution
........................................................................................KK...KK.KK.KK.KK.KK.KK.KK.KK.KK.....KK.KK.KK..KK.KK.KK...........KK........KK.....KK..KK.KK.KK............................................................................................................................................................................KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK................KK.............................................................S......S...KK.KK.KK.KK.KK.KK.KK.KK.KK...KK.KK.KK.KK.KK.....KK... 
======================================================================
ERROR: matplotlib.tests.test_dviread.test_dviread
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34_64\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\tests\test_dviread.py", line 74, in test_dviread
    for page in dvi]
  File "c:\projects\matplotlib\lib\matplotlib\tests\test_dviread.py", line 68, in <listcomp>
    data = [{'text': [[t.x, t.y,
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 239, in __iter__
    have_page = self._read()
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 300, in _read
    self._dtable[byte](self, byte)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 164, in wrapper
    return method(self, *[f(self, byte-min) for f in get_args])
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 447, in _fnt_def
    self._fnt_def_real(k, c, s, d, a, l)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 452, in _fnt_def_real
    tfm = _tfmfile(fontname)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 1016, in _tfmfile
    return _fontfile(texname, Tfm, '.tfm', _tfmcache)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 1005, in _fontfile
    filename = find_tex_file(texname + suffix)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 985, in find_tex_file
    stderr=subprocess.PIPE)
  File "C:\Python34_64\envs\test-environment\lib\subprocess.py", line 859, in __init__
    restore_signals, start_new_session)
  File "C:\Python34_64\envs\test-environment\lib\subprocess.py", line 1112, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

======================================================================
FAIL: matplotlib.tests.test_patches.test_wedge_range.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34_64\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 54, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 245, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: C:\projects\matplotlib\result_images\test_patches\wedge_range.png vs. C:\projects\matplotlib\result_images\test_patches\wedge_range-expected.png (RMS 0.059)

======================================================================
FAIL: matplotlib.tests.test_path.test_xkcd.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34_64\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 54, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 245, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: C:\projects\matplotlib\result_images\test_path\xkcd.png vs. C:\projects\matplotlib\result_images\test_path\xkcd-expected.png (RMS 9.607)

======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python34_64\envs\test-environment\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 54, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 245, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: C:\projects\matplotlib\result_images\test_patheffects\collection.png vs. C:\projects\matplotlib\result_images\test_patheffects\collection-expected.png (RMS 0.012)

----------------------------------------------------------------------
Ran 5938 tests in 366.738s

FAILED (KNOWNFAIL=1228, SKIP=10, errors=1, failures=3)

[tacaswell]

The xkcd one does not worry me too much, it currently uses the system rand to wiggle the path which is (iirc) vendor dependent. This should be fixed by #5728 which uses our own RNG, but it has a different failure on windows.

[jenshnielsen]

I am seeing test_patheffects.test_collection fail with a similar diff on OSX see my comments in #5647

[jankatins]

Current status (from a py27, 64bit run):

python tests.py || cmd /c "exit /b 0"
c:\projects\matplotlib\lib\matplotlib\__init__.py:1513: UserWarning: matplotlib is not built with the correct FreeType version to run tests.  Set local_freetype=True in setup.cfg and rebuild. Expect many image comparison failures below.
  "matplotlib is not built with the correct FreeType version to run " 
.....KKKKKK.KK...KK.KK.......KK...KK..KK.KK.KK.....KK.KK.KK.KK...KK..........................KK.KK.KK.KK...KK.KK.KK.KK.KK.KK.KK.KK.KK...KK..KK.KK.KK.KK.KK.KK.....KK.KK.KK..KK.KK.KK.KK.KK.KK.KK.KK.KK......KK........KK.KK.KK.KK......KK...KK.KK.KK.KK.KK.KK..KK.KK.KK...KK...........KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.....KK...KK..KK.KK.KK.KK.............KK.KK.KK.KK.KK.KK.KK.KK.KK....KK......................................SS.......................KK....SSSSSS'gs' is not recognized as an internal or external command,
operable program or batch file.
'gs' is not recognized as an internal or external command,
operable program or batch file.
K'latex' is not recognized as an internal or external command,
operable program or batch file.
K'latex' is not recognized as an internal or external command,
operable program or batch file.
K.................KKKKKKKKKKKKKKK.KK....'latex' is not recognized as an internal or external command,
operable program or batch file.
Traceback (most recent call last): 
  File "<string>", line 1, in <module>
  File "c:\projects\matplotlib\lib\matplotlib\tests\test_backend_svg.py", line 151, in _test_determinism_save
    FigureCanvasSVG(fig).print_svg(filename)
  File "c:\projects\matplotlib\lib\matplotlib\backends\backend_svg.py", line 1220, in print_svg
    return self._print_svg(filename, svgwriter, fh_to_close, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\backends\backend_svg.py", line 1248, in _print_svg
    self.figure.draw(renderer)
  File "c:\projects\matplotlib\lib\matplotlib\artist.py", line 62, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\figure.py", line 1292, in draw
    func(*args)
  File "c:\projects\matplotlib\lib\matplotlib\artist.py", line 62, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\axes\_base.py", line 2392, in draw
    a.draw(renderer)
  File "c:\projects\matplotlib\lib\matplotlib\artist.py", line 62, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\axis.py", line 1130, in draw
    renderer)
  File "c:\projects\matplotlib\lib\matplotlib\axis.py", line 1070, in _get_tick_bboxes
    extent = tick.label1.get_window_extent(renderer)
  File "c:\projects\matplotlib\lib\matplotlib\text.py", line 961, in get_window_extent
    bbox, info, descent = self._get_layout(self._renderer)
  File "c:\projects\matplotlib\lib\matplotlib\text.py", line 352, in _get_layout
    ismath=False)
  File "c:\projects\matplotlib\lib\matplotlib\backends\backend_svg.py", line 1197, in get_text_width_height_descent
    return self._text2path.get_text_width_height_descent(s, prop, ismath)
  File "c:\projects\matplotlib\lib\matplotlib\textpath.py", line 101, in get_text_width_height_descent
    renderer=None)
  File "c:\projects\matplotlib\lib\matplotlib\texmanager.py", line 675, in get_text_width_height_descent
    dvifile = self.make_dvi(tex, fontsize)
  File "c:\projects\matplotlib\lib\matplotlib\texmanager.py", line 422, in make_dvi
    report))
RuntimeError: LaTeX was not able to process the following string:
'lp'
Here is the full report generated by LaTeX: 


K.K..KK.KK.KK.KK..........................SS..KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK..............c:\projects\matplotlib\lib\matplotlib\collections.py:364: UserWarning: Collection picker None could not be converted to float
  % self._picker)
....KK........................................c:\projects\matplotlib\lib\matplotlib\colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 4
  colors = np.asarray(colors)[igood]
c:\projects\matplotlib\lib\matplotlib\colorbar.py:539: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 5 but corresponding boolean dimension is 3
  colors = np.asarray(colors)[igood]
..............................K.K.....K.....K.....K.....K.....K.....K.....K.....K.....K....E.K.KK.KK............K...KK.KK.KK.KK.KKKK.KKKK.KKKK.K.....KC:\conda\envs\test-environment\lib\xml\etree\ElementTree.py:1627: DeprecationWarning: This method of XMLParser is deprecated.  Define doctype() method on the TreeBuilder target.
  DeprecationWarning,
.............KK.KK..KK.KK.KK.KK..KK.KK.KK...KK..KK...................................................................................................................................................................................................................................................................................................................KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K...KK..KK.KK.KK.KKFKK.....KK.KK......FKK.KK.KK.KKK.............................. 
Bad key "timezone" on line 1 in
c:\projects\matplotlib\lib\matplotlib\tests\test_utf32_be_rcparams.rc.
You probably need to get an updated matplotlibrc file from
http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
or from the matplotlib source distribution 
........................................................................................KK...KK.KK.KK.KK.KK.KK.KK.KK.KK.....KK.KK.KK..KK.KK.KK...........KK........KK.....KK..KK.KK.KK.............................................................................................................................................................................KK.KK.KK.KK.KK.KK.KK.KK.KK.KK.KK................KK.............................................................S......S...KK.KK.KK.KK.KK.KK.KK.KK.KK...KK.KK.KK.KK.KK.....KK.... 
====================================================================== 
ERROR: matplotlib.tests.test_dviread.test_dviread 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\tests\test_dviread.py", line 74, in test_dviread
    for page in dvi]
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 239, in __iter__
    have_page = self._read()
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 300, in _read
    self._dtable[byte](self, byte)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 164, in wrapper
    return method(self, *[f(self, byte-min) for f in get_args])
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 447, in _fnt_def
    self._fnt_def_real(k, c, s, d, a, l)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 452, in _fnt_def_real
    tfm = _tfmfile(fontname)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 1016, in _tfmfile
    return _fontfile(texname, Tfm, '.tfm', _tfmcache) 
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 1005, in _fontfile
    filename = find_tex_file(texname + suffix)
  File "c:\projects\matplotlib\lib\matplotlib\dviread.py", line 985, in find_tex_file
    stderr=subprocess.PIPE)
  File "C:\conda\envs\test-environment\lib\subprocess.py", line 710, in __init__
    errread, errwrite)
  File "C:\conda\envs\test-environment\lib\subprocess.py", line 958, in _execute_child
    startupinfo)
WindowsError: [Error 2] The system cannot find the file specified 

====================================================================== 
FAIL: matplotlib.tests.test_patches.test_wedge_range.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 54, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 245, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: C:\projects\matplotlib\result_images\test_patches\wedge_range.png vs. C:\projects\matplotlib\result_images\test_patches\wedge_range-expected.png (RMS 0.059) 

====================================================================== 
FAIL: matplotlib.tests.test_patheffects.test_collection.test 
---------------------------------------------------------------------- 
Traceback (most recent call last):
  File "C:\conda\envs\test-environment\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 54, in failer
    result = f(*args, **kwargs)
  File "c:\projects\matplotlib\lib\matplotlib\testing\decorators.py", line 245, in do_test
    '(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: C:\projects\matplotlib\result_images\test_patheffects\collection.png vs. C:\projects\matplotlib\result_images\test_patheffects\collection-expected.png (RMS 0.012) 

---------------------------------------------------------------------- 
Ran 5962 tests in 1246.825s 

FAILED (KNOWNFAIL=1237, SKIP=12, errors=1, failures=2) 

[jankatins]

Oh my, on my local windows 3.5 build with local freetype, it's even worse:

======================================================================
ERROR: matplotlib.tests.test_backend_ps.test_savefig_to_stringio_with_usetex
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 152, in wrapped_callable
    func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_ps.py", line 77, in test_savefig_to_stringio_with_usetex
    _test_savefig_to_stringio()
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_ps.py", line 40, in _test_savefig_to_stringio
    fig.savefig(buffer, format=format)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\figure.py", line 1698, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backend_bases.py", line 2232, in print_figure
    **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 985, in print_ps
    return self._print_ps(outfile, 'ps', *args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 1012, in _print_ps
    **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 1376, in _print_figure_tex
    rotated=psfrag_rotated)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 1534, in gs_distill
    your image.\nHere is the full report generated by ghostscript:\n\n' + fh.read())
TypeError: Can't convert 'bytes' object to str implicitly

======================================================================
ERROR: matplotlib.tests.test_backend_ps.test_savefig_to_stringio_with_usetex_eps
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 152, in wrapped_callable
    func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_ps.py", line 96, in test_savefig_to_stringio_with_usetex_eps
    _test_savefig_to_stringio(format='eps')
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_ps.py", line 40, in _test_savefig_to_stringio
    fig.savefig(buffer, format=format)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\figure.py", line 1698, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backend_bases.py", line 2232, in print_figure
    **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 988, in print_eps
    return self._print_ps(outfile, 'eps', *args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 1012, in _print_ps
    **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 1376, in _print_figure_tex
    rotated=psfrag_rotated)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\backends\backend_ps.py", line 1534, in gs_distill
    your image.\nHere is the full report generated by ghostscript:\n\n' + fh.read())
TypeError: Can't convert 'bytes' object to str implicitly

======================================================================
FAIL: matplotlib.tests.test_backend_pgf.test_xelatex
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 422, in backend_switcher
    result = func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 92, in test_xelatex
    compare_figure('pgf_xelatex.pdf')
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 50, in compare_figure
    err = compare_images(expected, actual, tol=14)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 309, in compare_images
    actual = convert(actual, False)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 170, in convert
    "Don't know how to convert %s files to png" % extension)
matplotlib.testing.exceptions.ImageComparisonFailure: Don't know how to convert pdf files to png

======================================================================
FAIL: matplotlib.tests.test_backend_pgf.test_pdflatex
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 422, in backend_switcher
    result = func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 108, in test_pdflatex
    compare_figure('pgf_pdflatex.pdf')
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 50, in compare_figure
    err = compare_images(expected, actual, tol=14)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 309, in compare_images
    actual = convert(actual, False)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 170, in convert
    "Don't know how to convert %s files to png" % extension)
matplotlib.testing.exceptions.ImageComparisonFailure: Don't know how to convert pdf files to png

======================================================================
FAIL: matplotlib.tests.test_backend_pgf.test_rcupdate
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 422, in backend_switcher
    result = func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 137, in test_rcupdate
    compare_figure('pgf_rcupdate%d.pdf' % (i + 1))
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 50, in compare_figure
    err = compare_images(expected, actual, tol=14)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 309, in compare_images
    actual = convert(actual, False)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 170, in convert
    "Don't know how to convert %s files to png" % extension)
matplotlib.testing.exceptions.ImageComparisonFailure: Don't know how to convert pdf files to png

======================================================================
FAIL: matplotlib.tests.test_backend_pgf.test_mixedmode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 422, in backend_switcher
    result = func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 171, in test_mixedmode
    compare_figure('pgf_mixedmode.pdf')
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 50, in compare_figure
    err = compare_images(expected, actual, tol=14)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 309, in compare_images
    actual = convert(actual, False)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 170, in convert
    "Don't know how to convert %s files to png" % extension)
matplotlib.testing.exceptions.ImageComparisonFailure: Don't know how to convert pdf files to png

======================================================================
FAIL: matplotlib.tests.test_backend_pgf.test_bbox_inches
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 422, in backend_switcher
    result = func(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 193, in test_bbox_inches
    compare_figure('pgf_bbox_inches.pdf', savefig_kwargs={'bbox_inches': bbox})
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\tests\test_backend_pgf.py", line 50, in compare_figure
    err = compare_images(expected, actual, tol=14)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 309, in compare_images
    actual = convert(actual, False)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\compare.py", line 170, in convert
    "Don't know how to convert %s files to png" % extension)
matplotlib.testing.exceptions.ImageComparisonFailure: Don't know how to convert pdf files to png

======================================================================
FAIL: matplotlib.tests.test_patheffects.test_collection.test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\nose\case.py", line 198, in runTest
    self.test(*self.arg)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 55, in failer
    result = f(*args, **kwargs)
  File "C:\portabel\miniconda\envs\matplotlib_build\lib\site-packages\matplotlib-1.5.1+1064.g4f14fc8-py3.5-win-amd64.egg\matplotlib\testing\decorators.py", line 259, in do_test
    '(RMS %(rms).3f)'%err)
matplotlib.testing.exceptions.ImageComparisonFailure: images not close: c:\data\external\pydata\matplotlib\result_images\test_patheffects\collection.png vs. c:\data\external\pydata\matplotlib\result_images\test_patheffects\collection-expected.png (RMS 0.010)

[jankatins]

Ok, IMO this should be changed to a Skip:

def convert(filename, cache):
    """
    Convert the named file into a png file.  Returns the name of the
    created file.

    If *cache* is True, the result of the conversion is cached in
    `matplotlib._get_cachedir() + '/test_cache/'`.  The caching is based
    on a hash of the exact contents of the input file.  The is no limit
    on the size of the cache, so it may need to be manually cleared
    periodically.

    """
    base, extension = filename.rsplit('.', 1)
    if extension not in converter:
        raise ImageComparisonFailure(
            "Don't know how to convert %s files to png" % extension)