What is the interaction with pytest's -x flag

[birdsarah]

What I'd like to happen is that if I run

py.test --rerun 5 -x

If a test fails it reruns it up to 5 times, but if it fails, pytest then exits.

[florianpilz]

Hi @birdsarah, nice to see you again! (We did a Sprint together at EuroPython 2014.)

I guess what you want to achieve is that pytest continues if -x was given and a flaky test succeeds due to --rerun.

I have looked into the issue, but nothing related to -x seems to trigger the exit. :/ I will try to get help from the mailing list.

[birdsarah]

Hi @florianpilz - indeed!

Yes, you've got it - rerun takes precedence so:

• please rerun this flaky test N times
• if it fails after N times, and -x is given, then please bail the whole test run.

Having looked into it, this is what happens:

Here's output from a simple test run, that I setup to fail:

(bokeh_34)➜  tests git:(contain-bokeh-style) ✗ py.test -k test_colors -x --rerun=5 -v
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.4.4, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /Users/caged/miniconda3/envs/bokeh_34/bin/python
cachedir: ../../.cache
rootdir: /Users/caged/Dev/bokeh/bokeh, inifile: setup.cfg
plugins: cov-1.8.1, html-1.7, selenium-1.1, variables-1.3, rerunfailures-1.0.1, timeout-1.0.0, xdist-1.14
collected 246 items 

test_colors.py::TestColor::test_abstract PASSED
test_colors.py::TestColor::test_basic RERUN
test_colors.py::TestColor::test_basic RERUN
test_colors.py::TestColor::test_basic RERUN
test_colors.py::TestColor::test_basic RERUN
test_colors.py::TestColor::test_basic RERUN
test_colors.py::TestColor::test_basic FAILED

=================================================================================================== FAILURES ===================================================================================================
_____________________________________________________________________________________________ TestColor.test_basic _____________________________________________________________________________________________

self = <test_colors.TestColor testMethod=test_basic>

    def test_basic(self):
        c = colors.Color()
>       assert False
E       AssertionError: assert False

test_colors.py:11: AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================================================== 226 tests deselected by '-ktest_colors' ====================================================================================

With xdist it does not do the same:

(bokeh_34)➜  tests git:(contain-bokeh-style) ✗ py.test -k test_colors -x --rerun=5 -v -n3
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.4.4, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /Users/caged/miniconda3/envs/bokeh_34/bin/python
cachedir: ../../.cache
rootdir: /Users/caged/Dev/bokeh/bokeh, inifile: setup.cfg
plugins: cov-1.8.1, html-1.7, selenium-1.1, variables-1.3, rerunfailures-1.0.1, timeout-1.0.0, xdist-1.14
[gw0] darwin Python 3.4.4 cwd: /Users/caged/Dev/bokeh/bokeh/bokeh/tests
[gw1] darwin Python 3.4.4 cwd: /Users/caged/Dev/bokeh/bokeh/bokeh/tests
[gw2] darwin Python 3.4.4 cwd: /Users/caged/Dev/bokeh/bokeh/bokeh/tests
[gw0] Python 3.4.4 |Continuum Analytics, Inc.| (default, Jan  9 2016, 17:30:09)  -- [GCC 4.2.1 (Apple Inc. build 5577)]
[gw1] Python 3.4.4 |Continuum Analytics, Inc.| (default, Jan  9 2016, 17:30:09)  -- [GCC 4.2.1 (Apple Inc. build 5577)]
[gw2] Python 3.4.4 |Continuum Analytics, Inc.| (default, Jan  9 2016, 17:30:09)  -- [GCC 4.2.1 (Apple Inc. build 5577)]
gw0 [20] / gw1 [20] / gw2 [20]
scheduling tests via LoadScheduling

test_colors.py::TestColor::test_clamp 
test_colors.py::TestColor::test_basic 
test_colors.py::TestColor::test_abstract 
[gw2] PASSED test_colors.py::TestColor::test_clamp 
[gw1] PASSED test_colors.py::TestColor::test_abstract 
test_colors.py::TestColor::test_repr 
test_colors.py::TestColor::test_darken 
[gw2] PASSED test_colors.py::TestColor::test_repr 
[gw1] PASSED test_colors.py::TestColor::test_darken 
test_colors.py::TestRGB::test_basic 
test_colors.py::TestRGB::test_from_rgb 
[gw2] PASSED test_colors.py::TestRGB::test_basic 
[gw1] PASSED test_colors.py::TestRGB::test_from_rgb 
test_colors.py::TestRGB::test_repr 
test_colors.py::TestRGB::test_to_css 
[gw2] PASSED test_colors.py::TestRGB::test_repr 
[gw1] PASSED test_colors.py::TestRGB::test_to_css 
test_colors.py::TestRGB::test_to_hex 
test_colors.py::TestRGB::test_to_rgb 
[gw2] PASSED test_colors.py::TestRGB::test_to_hex 
[gw1] PASSED test_colors.py::TestRGB::test_to_rgb 
test_colors.py::TestHSL::test_from_hsl 
test_colors.py::TestHSL::test_basic 
[gw2] PASSED test_colors.py::TestHSL::test_basic 
[gw1] PASSED test_colors.py::TestHSL::test_from_hsl 
test_colors.py::TestHSL::test_repr 
test_colors.py::TestHSL::test_to_css 
[gw2] PASSED test_colors.py::TestHSL::test_repr 
test_colors.py::TestHSL::test_to_hsl 
[gw1] PASSED test_colors.py::TestHSL::test_to_css 
[gw2] PASSED test_colors.py::TestHSL::test_to_hsl 
test_colors.py::TestNamedColor::test_basic 
[gw1] PASSED test_colors.py::TestNamedColor::test_basic 
test_colors.py::TestColor::test_basic 
[gw0] FAILED test_colors.py::TestColor::test_basic 

=================================================================================================== FAILURES ===================================================================================================
_____________________________________________________________________________________________ TestColor.test_basic _____________________________________________________________________________________________
[gw0] darwin -- Python 3.4.4 /Users/caged/miniconda3/envs/bokeh_34/bin/python
self = <test_colors.TestColor testMethod=test_basic>

    def test_basic(self):
        c = colors.Color()
>       assert False
E       AssertionError: assert False

test_colors.py:11: AssertionError
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! xdist.dsession.Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 1 failed, 16 passed in 1.46 seconds ======================================================================================

The reruns don't occur.

Edit: I've updated this comment, now I'm reading the trace correctly. Sorry for the confusion.

[birdsarah]

Oh wow! I can't read a set of test results.

Looking at the above results I posted, rerun is not doing what I hoped, the test run bails after the first failure when using xdist.

I've updated the above comment to reflect that.

[florianpilz]

Now I'm confused. What exactly did you expect?

From my understanding, the current implementation will RERUN a test and stop execution of the whole test suite if it still fails on the last RERUN when --exitfirst is given. Be aware that with xdist only the final output is logged, so I assume that the test was RERUN 5 times, but since it still failed the logged outcome is FAILED. (xdist will terminate a worker on the first logged outcome of a test, thus we cannot log intermediate reruns.)

However, during this discussion I found out that with --exitfirst the whole test suite will be stopped after a RERUN, even in case it succeeded! This is definitely a bug, but I have not yet found a solution to this.

[birdsarah]

Now I'm confused. What exactly did you expect?

@florianpilz - me too. Sorry, I totally dropped the ball on this thread. Feel free to do with this thread as you wish.

If we found a bug that's great. But it may be better to document in a new issue without the above ramblings from me.

Best,

Bird

[florianpilz]

OK, I extracted the bug to #32.