pytest-parallel exit code always 0

[etragas-fathom]

Unsure if the issue is specific to me, added the info below to reproduce.

tst.py =

import pytest
def test_tst():
    assert False

etragas$ pytest tst.py
test session starts 
platform darwin -- Python 3.6.6, pytest-3.8.1, py-1.6.0, pluggy-0.7.1
rootdir: /Users/etragas/diseaseTools, inifile: pytest.ini
plugins: xdist-1.23.2, profiling-1.3.0, parallel-0.0.5, mock-1.10.0, forked-0.2, cov-2.6.0
collected 1 item

tst.py F                                                                                                                                                 [100%]

=== FAILURES ===

    def test_tst():
>       assert False
E       assert False

tst.py:3: AssertionError
1 failed in 0.08 seconds 

etragas$ echo $?
1

etragas$ pytest tst.py --workers auto
test session starts 
platform darwin -- Python 3.6.6, pytest-3.8.1, py-1.6.0, pluggy-0.7.1
rootdir: /Users/etragas/diseaseTools, inifile: pytest.ini
plugins: xdist-1.23.2, profiling-1.3.0, parallel-0.0.5, mock-1.10.0, forked-0.2, cov-2.6.0
collected 1 item
pytest-parallel: 12 workers (processes), 1 test per worker (thread)
F                                                                                                                                                        [100%]
=== FAILURES ===

    def test_tst():
>       assert False
E       assert False

tst.py:3: AssertionError
1 failed in 0.12 seconds 

HUAWEI_P10-796cab753f8233:diseaseTools etragas$ echo $?
0

[kevlened]

Interesting. I wonder if this also happens when using --tests-per-worker auto. I'll write a test to repro the original problem.

[etragas-fathom]

I tested, and this issue doesn't occur if you set --workers 1

[etragas-fathom]

I suspect there is a diff between running process_with_threads directly and wrapping that in a process

[kevlened]

pytest-parallel just does a normal run when --workers is 1, so that makes sense. What's odd is there's a test to ensure failures are reported: https://github.com/browsertron/pytest-parallel/blob/2f6a094ebb5a6d274adcdb1942a62dd124252711/tests/test_parallel.py#L4-L18

There must be something else in the reporting that's missing.

[etragas-fathom]

But the exit code isn't the same thing as the number of failures right? So would the above test catch that?

[kevlened]

It's not the same, but it means there's a disconnect between how pytest does its reporting and how it decides to exit the process.

[kevlened]

That test needs assert result.ret == 1

[kevlened]

The multiprocessing portion of this library works by monkeypatching values on pytest classes using the multiprocessing module. Ideally, we'd monkeypatch the testsfailed counter. The problem is getting a multiprocessing counter to work like a normal integer is non-trivial.

[kevlened]

Thanks for reporting this! It should be resolved in 0.0.6, which is on PyPI now.

[etragas-fathom]

Haha excellent! I just came here to report that session.testsfailed was probably the issue. Thank you for fixing this so quickly 👏