Use some more pytest plugins: warnings & rerunfailures #8346

QuLogic · 2017-03-20T00:00:00Z

Install pytest-warnings so we can see warnings produced by each test instead of being entirely hidden and unnoticed (cf. #7334 (comment)).

With pytest-rerunfailures, we can mark a few tests that are flaky so that they get rerun automatically. I'd rather flaky tests be fixed of course, but this will hopefully help speed up CI a bit since we won't have to wait the 12+ minutes for a full rebuild.

I have only marked two tests as flaky because they are inherently difficult to fix. If there are any suggestions for other tests to mark, I will add them, but hopefully there will be few.

QuLogic · 2017-03-20T00:39:59Z

With 3 test run failures out of the last 2 builds, test_mplot3d/mixedsubplot.svg might be a good addition as well. I actually have an idea of how to fix this one, but haven't had a chance to get through all the other required work to actually implement it.

anntzer · 2017-03-20T03:56:09Z

appveyor.yml

@@ -93,6 +93,7 @@ install:
  - cmd: IF %PYTHON_VERSION% == 2.7 conda install -q backports.functools_lru_cache
  # pytest-cov>=2.3.1 due to https://github.com/pytest-dev/pytest-cov/issues/124
  - conda install -q pytest "pytest-cov>=2.3.1" pytest-timeout pytest-xdist
+  - pip install -q pytest-rerunfailures pytest-warnings


Can we keep everything installed from the same source?

Not available in conda or conda-forge, but I could switch them all to pip.

let's use pip for everything

phobson · 2017-03-21T17:23:29Z

I was going to suggest the test_mixedsubplot-svg as well.

In my notes, I also have these:

lib/matplotlib/tests/test_mathtext.py::test_mathfont_stix_18
lib/matplotlib/tests/test_axes.py::test_formatter_ticker[1-formatter_ticker_002-pdf]

Though I haven't checked recently if they are still flaky

QuLogic · 2017-03-23T04:55:29Z

lib/matplotlib/tests/test_mathtext.py::test_mathfont_stix_18

This one's covered by c3aca56.

lib/matplotlib/tests/test_axes.py::test_formatter_ticker[1-formatter_ticker_002-pdf]

I don't recall seeing this one very often lately.

QuLogic · 2017-03-23T06:32:02Z

Hmm, you know what; I think the warnings are still not printed... I will investigate tomorrow.

QuLogic · 2017-03-23T06:43:29Z

Coincidentally, pytest-warnings just got merged into pytest: pytest-dev/pytest#2072

dstansby · 2017-03-25T15:56:22Z

The doc build isn't finding the command deactivate, so is erroring - my guess is that this should be source deactivate?

QuLogic · 2017-03-26T01:43:51Z

Aha, I've figured out why warnings are not correctly captured. The image_comparison decorator runs the test code as part of the setup and only runs a compare in the actual test part, which means that it's not covered by pytest-warnings.

QuLogic · 2017-03-26T20:15:30Z

test_savefig_to_stringio_with_usetex has been a bit flaky as well; maybe should be added too?

Edit: Actually, that build is a lock error which I've not seen before. I do recall a test with a similar name (or just this one) failing semi-regularly though.

dstansby · 2017-04-12T10:20:17Z

ci/travis/test_script.sh

@@ -1,5 +1,7 @@
 #! /bin/bash

+set -e


This is causing the doc build to fail on line 39 below, which can just be deleted (I can't work out how to open a PR on your branch, but https://travis-ci.org/dstansby/matplotlib/builds/221288308 is my travis build)

should probably just drop line 39

QuLogic · 2017-04-15T21:16:12Z

FYI, I'd like to get #8380 in first so that the warning filter plugin starts working correctly.

tacaswell · 2017-04-17T02:27:46Z

@QuLogic #8380 is in and this needs a rebase.

QuLogic · 2017-04-21T00:58:38Z

Well, that's unfortunate; it looks like mixedsubplots still fails even with a rerun. (Unfortunately, with xdist, you only see the last result, but I do know rerun is working as I made an explicitly broken test to try it out.) This is especially disappointing as mixedsubplots seems to have been failing quite a lot recently.

phobson · 2017-04-21T01:57:00Z

can we just disable the SVG portion of that test?

QuLogic · 2017-04-23T19:18:35Z

I marked it as xfail now.

See matplotlib#7911 for why these tests are flaky and matplotlib#7107 for why they are not so easy to fix.

Since it depends on "speed", it may sometimes fail, so try again a couple of times.

Even with a rerun, it doesn't appear to fix itself, and it's been acting up a lot lately, so just ignore any failures for now.

QuLogic · 2017-04-28T06:50:33Z

This is rebased now that the documentation is building again.

WeatherGod · 2017-04-28T21:14:53Z

As the creator of mixedsubplots, I am completely flummoxed why it would even be flakey... ever. I worry that there might actually be a real bug somewhere here. A race condition or something, perhaps.

QuLogic · 2017-04-28T21:25:02Z

Not sure it's a race condition as marking it to rerun a few times did not seem to work. Once it's broken, it's broken, but I don't know what breaks it.

WeatherGod · 2017-04-28T21:30:57Z

Could you try taking out the antialiased argument in the plot_surface(). It is an anachronism from a bygone era. Perhaps it is causing trouble.

QuLogic · 2017-04-28T23:41:23Z

According to #8549, that didn't work, so I think we should just go ahead and merge this.

QuLogic · 2017-04-30T01:26:19Z

🎊 Hopefully there's at least a little reduction in restarted jobs...

QuLogic added the topic: testing label Mar 20, 2017

QuLogic added this to the 2.1 (next point release) milestone Mar 20, 2017

anntzer reviewed Mar 20, 2017

View reviewed changes

anntzer approved these changes Mar 23, 2017

View reviewed changes

anntzer changed the title ~~Use some more pytest plugins: warnings & rerunfailures~~ [MRG+1] Use some more pytest plugins: warnings & rerunfailures Mar 23, 2017

QuLogic force-pushed the pytest-plugins branch from 387b251 to 83fb3fb Compare March 24, 2017 23:42

QuLogic mentioned this pull request Mar 26, 2017

Make image_comparison more pytest-y #8380

Merged

dstansby reviewed Apr 12, 2017

View reviewed changes

QuLogic force-pushed the pytest-plugins branch from 2419f62 to bc85a82 Compare April 17, 2017 06:31

QuLogic force-pushed the pytest-plugins branch 2 times, most recently from b2528c5 to 76e4655 Compare April 23, 2017 18:53

QuLogic mentioned this pull request Apr 24, 2017

Sphinx Gallery API pages + deprecating old examples folder #8525

Merged

QuLogic added 4 commits April 27, 2017 23:11

ci: Install pytest-warnings to report warnings.

c0ef143

Add pytest-rerunfailures plugin.

35b13b4

ci: Error out of test script on first error.

c310c08

TST: Mark mathtext tests as flaky.

8a6b84e

See matplotlib#7911 for why these tests are flaky and matplotlib#7107 for why they are not so easy to fix.

QuLogic added 3 commits April 27, 2017 23:11

TST: Mark test_invisible_Line_rendering as flaky.

4c95844

Since it depends on "speed", it may sometimes fail, so try again a couple of times.

TST: Use only pip for pytest packages.

086e3d4

TST: Mark mixedsubplots SVG test as xfail.

28c8fef

Even with a rerun, it doesn't appear to fix itself, and it's been acting up a lot lately, so just ignore any failures for now.

QuLogic force-pushed the pytest-plugins branch from 76e4655 to 28c8fef Compare April 28, 2017 03:12

phobson mentioned this pull request Apr 28, 2017

TST: remove out-dated antialiased param #8549

Closed

2 tasks

phobson approved these changes Apr 29, 2017

View reviewed changes

phobson changed the title ~~[MRG+1] Use some more pytest plugins: warnings & rerunfailures~~ Use some more pytest plugins: warnings & rerunfailures Apr 29, 2017

phobson merged commit 405e595 into matplotlib:master Apr 29, 2017

QuLogic deleted the pytest-plugins branch April 30, 2017 01:24

QuLogic mentioned this pull request May 21, 2017

Consolidate and move examples out of pylab #8266

Merged

anntzer mentioned this pull request Jul 18, 2018

Don't use pytest -rw now that pytest-warnings is builtin. #11700

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use some more pytest plugins: warnings & rerunfailures #8346

Use some more pytest plugins: warnings & rerunfailures #8346

QuLogic commented Mar 20, 2017

QuLogic commented Mar 20, 2017 •

edited

Loading

anntzer Mar 20, 2017

QuLogic Mar 20, 2017

anntzer Mar 20, 2017

phobson commented Mar 21, 2017

QuLogic commented Mar 23, 2017

QuLogic commented Mar 23, 2017

QuLogic commented Mar 23, 2017

dstansby commented Mar 25, 2017

QuLogic commented Mar 26, 2017

QuLogic commented Mar 26, 2017 •

edited

Loading

dstansby Apr 12, 2017

tacaswell Apr 17, 2017

QuLogic commented Apr 15, 2017

tacaswell commented Apr 17, 2017

QuLogic commented Apr 21, 2017

phobson commented Apr 21, 2017

QuLogic commented Apr 23, 2017

QuLogic commented Apr 28, 2017

WeatherGod commented Apr 28, 2017

QuLogic commented Apr 28, 2017

WeatherGod commented Apr 28, 2017

QuLogic commented Apr 28, 2017

QuLogic commented Apr 30, 2017

Use some more pytest plugins: warnings & rerunfailures #8346

Use some more pytest plugins: warnings & rerunfailures #8346

Conversation

QuLogic commented Mar 20, 2017

QuLogic commented Mar 20, 2017 • edited Loading

anntzer Mar 20, 2017

Choose a reason for hiding this comment

QuLogic Mar 20, 2017

Choose a reason for hiding this comment

anntzer Mar 20, 2017

Choose a reason for hiding this comment

phobson commented Mar 21, 2017

QuLogic commented Mar 23, 2017

QuLogic commented Mar 23, 2017

QuLogic commented Mar 23, 2017

dstansby commented Mar 25, 2017

QuLogic commented Mar 26, 2017

QuLogic commented Mar 26, 2017 • edited Loading

dstansby Apr 12, 2017

Choose a reason for hiding this comment

tacaswell Apr 17, 2017

Choose a reason for hiding this comment

QuLogic commented Apr 15, 2017

tacaswell commented Apr 17, 2017

QuLogic commented Apr 21, 2017

phobson commented Apr 21, 2017

QuLogic commented Apr 23, 2017

QuLogic commented Apr 28, 2017

WeatherGod commented Apr 28, 2017

QuLogic commented Apr 28, 2017

WeatherGod commented Apr 28, 2017

QuLogic commented Apr 28, 2017

QuLogic commented Apr 30, 2017

QuLogic commented Mar 20, 2017 •

edited

Loading

QuLogic commented Mar 26, 2017 •

edited

Loading