Fixes for Windows test failures on appveyor #5922
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jankatins
force-pushed
the
fix_tests_on_windows
branch
from
faf2347
to
573e1dd
Compare
| @@ -0,0 +1,88 @@ | |||
| #!/usr/bin/env python | |||
There was a problem hiding this comment.
Probably interesting, but I think this file is not actually related to this PR?
There was a problem hiding this comment.
That's handy. There's also tools/test_triage.py that does something similar in PyQt4 in case you're not aware of it.
There was a problem hiding this comment.
yep, at least in should not belong to that commit... Also needed a fix for the image comparison decorator test images ...
There was a problem hiding this comment.
I left that in for the moment, should I add it or remove it? if "keep": in a different PR? I would run this script on failure on appveyor so that the html file is included in the result_images.zip, which is IMO nice for a quick overview...
jankatins
force-pushed
the
fix_tests_on_windows
branch
from
573e1dd
to
9c353f0
Compare
|
The image comparison tests for some the failures are IMO to sensitive, at least I can't see much of a difference in that images. The only "real" difference is the font_styles, which is simply empty (-> all text missing)? |
Also fix the docstring to mention that it is part of miktex...
Also fixed a problem in an error message when bytes were appended to
a string (fails on py3.x) by using %.
One of the errors before:
======================================================================
ERROR: matplotlib.tests.test_backend_ps.test_savefig_to_stringio_with_usetex
----------------------------------------------------------------------
Traceback (most recent call last):
File "lib\site-packages\nose\case.py", line 198, in runTest
self.test(*self.arg)
File "lib\matplotlib\testing\decorators.py", line 152, in wrapped_callable
func(*args, **kwargs)
File "lib\matplotlib\testing\decorators.py", line 55, in failer
result = f(*args, **kwargs)
File "lib\matplotlib\tests\test_backend_ps.py", line 77, in test_savefig_to_stringio_with_usetex
_test_savefig_to_stringio()
File "lib\matplotlib\tests\test_backend_ps.py", line 40, in _test_savefig_to_stringio
fig.savefig(buffer, format=format)
File "lib\matplotlib\figure.py", line 1698, in savefig
self.canvas.print_figure(*args, **kwargs)
File "lib\matplotlib\backend_bases.py", line 2232, in print_figure
**kwargs)
File "lib\matplotlib\backends\backend_ps.py", line 985, in print_ps
return self._print_ps(outfile, 'ps', *args, **kwargs)
File "lib\matplotlib\backends\backend_ps.py", line 1012, in _print_ps
**kwargs)
File "lib\matplotlib\backends\backend_ps.py", line 1376, in _print_figure_tex
rotated=psfrag_rotated)
File "lib\matplotlib\backends\backend_ps.py", line 1539, in gs_distill
raise RuntimeError(m % output)
RuntimeError: ghostscript was not able to process your image.
Here is the full report generated by ghostscript:
b''
This is a script which I developed for ggplot https://github.com/yhat/ggplot/blob/master/visual_tests.py The script will go over the images in result_images and add them to a webpage, sorted by test files. At the top, there will be the test failures, so you can inspect the visually and try to figure out what's wrong with matplotlib (or the tests...) :-)
jankatins
force-pushed
the
fix_tests_on_windows
branch
from
411e038
to
27b14ab
Compare
|
Current output in my local py35 test run: |
jankatins
force-pushed
the
fix_tests_on_windows
branch
from
27b14ab
to
f1800af
Compare
|
Ok, here are the tests from appveyor (which fail the build now): TARGET_ARCH=x86, CONDA_PY=27 TARGET_ARCH=x64, CONDA_PY=27 TARGET_ARCH=x64, CONDA_PY=34 TARGET_ARCH=x64, CONDA_PY=35 |
|
The |
|
BTW, this seems to be out of sync: |
Indeed it is. I'll open another PR for that. |
There seems to be an error here, so don't silence 7z too much and only filter out the individual files beeing compressed, but not the status messages.
jankatins
force-pushed
the
fix_tests_on_windows
branch
from
3b12326
to
e1194c7
Compare
* [ ]: matplotlib.tests.test_axes.test_specgram_freqs.test (RMS 0.042) (x64,35) * [ ]: matplotlib.tests.test_axes.test_specgram_freqs.test (RMS 0.042) (x64,35) * [ ]: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test (RMS 0.042) (x64,35) * [ ]: matplotlib.tests.test_axes.test_specgram_magnitude_freqs.test (RMS 0.042) (x64,35) -> before set to 0.03, set tolerance to 0.05 on windows
* [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.006) (x64,35) * [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.008) (x86,27) * [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.012) (x64,27) * [ ]: matplotlib.tests.test_patheffects.test_collection.test (RMS 0.012) (x64,34) This has a black diff, so up the tolerance on windows to 0.013
|
There is also #5647, which lists these two tests as well:
-> currently set to 0.03, set tolerance to 0.05 on windows -> done
This has a black diff, so up the tolerance on windows to 0.013 -> done
This looks actually interesting: it seems that only the middle figure in the last row is different (at least the 32bit, py2.7 which I looked at) -> also done...
Also looks black, but only on py27/x86...? -> done
-> done |
affected: matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x64,27) matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x64,34) matplotlib.tests.test_patches.test_wedge_range.test (RMS 0.059) (x86,27) it seems that only the middle figure in the last row is different. Up the tolerance on windows to let the tests pass.
affected: * matplotlib.tests.test_axes.test_specgram_angle_freqs.test (RMS 0.002) (x86,27)
affected: * matplotlib.tests.test_triangulation.test_tri_smooth_gradient.test (RMS 0.014) (x64,35) The diff looks pitch black to me... -> up the tolerance...
|
https://travis-ci.org/matplotlib/matplotlib/jobs/105255492 [I really hate that the image comparison tests do not show the line number but only the decorator code path :-(] [Edit: seems to be a fluke, at least the next travis build didn't see that error?] |
jankatins
force-pushed
the
fix_tests_on_windows
branch
from
4896839
to
47481d7
Compare
|
And locally I still have the problem with |
|
I will open a new issue for my local test failure, that looks like the fallback font doesn't work on my system. [Edit: the problem was a stale font cache which prevent this test to find the fallback fonts... See #5929 for the story... Note to self: before writing issues, use a debugger...] |
|
Ok, the builds in my appveyor instance already build the two py2.7 versions of the tests and they are green: https://ci.appveyor.com/project/JanSchulz/matplotlib/build/1.0.135 |
|
Open question: should I remove the |
|
And it's green again (in my build) :-) |
tacaswell
added a commit
that referenced
this pull request
Feb 1, 2016
Fixes for Windows test failures on appveyor
|
@tacaswell @mdboom Should the tests for which I changed the tolerance be added to #5647? |
20 tasks
|
@JanSchulz Yes it would be good to collect the information about non zero tolerance in #5647 |
|
@jenshnielsen ok, done directly in the first comment as additional Todos: #5647 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes the windows test errors as seen on appveyor and makes appveyor test failures into build failures.
Closes: #5748