Exit pytest if a collection error occours #1421

Closed
fontealpina opened this Issue Feb 29, 2016 · 20 comments

Comments

Projects
None yet
6 participants
@fontealpina

If a collection error occurs, is there a way to terminate immediately pytest execution after collection, avoiding the execution of all the tests (even the ones properly collected)?
Basically I need to start test execution only if there are no import errors.
So for instance, now if there is a collection error I get:

$ pytest test/test_01/test_01.py test/test_02/test_02.py test/test_03/test_03.py -v -s
============================ test session starts =============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /usr/bin/python
collected 2 items / 1 errors

test/test_01/test_01.py::TestBug::test_n_01 PASSED
test/test_02/test_02.py::TestBug::test_n_02 PASSED

=================================== ERRORS ===================================
________________ ERROR collecting test/test_03/test_03.py _________________
import file mismatch:
a description of the import error
================ 2 passed, 1 error in 0.17 seconds ================

I would get something like this instead:

$ pytest test/test_01/test_01.py test/test_02/test_02.py test/test_03/test_03.py -v -s
============================ test session starts =============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /usr/bin/python
collected 2 items / 1 errors
=================================== ERRORS =============================
________________ ERROR collecting test/test_03/test_03.py _______________________
import file mismatch:
a description of the import error
================================ 1 error in 0.17 seconds ======================

Is there an options that allows that?
Or maybe this can be done with a plugin?
Thanks

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Feb 29, 2016

Member

You can easily do this with a local plugin, e.g. by putting this in a conftest.py:

import pytest

def pytest_collectreport(report):
    if report.failed:
        raise pytest.UsageError("Errors during collection, aborting")

Then even if tests are collected, your hook will exit before they are run:

collecting 1 items / 1 errors
================================================ ERRORS =================================================
_____________________________________ ERROR collecting test_two.py ______________________________________
test_two.py:1: in <module>
    import asifjsafpo
E   ImportError: No module named 'asifjsafpo'
======================================== 1 error in 0.01 seconds ========================================
ERROR: Errors during collection, aborting
Member

The-Compiler commented Feb 29, 2016

You can easily do this with a local plugin, e.g. by putting this in a conftest.py:

import pytest

def pytest_collectreport(report):
    if report.failed:
        raise pytest.UsageError("Errors during collection, aborting")

Then even if tests are collected, your hook will exit before they are run:

collecting 1 items / 1 errors
================================================ ERRORS =================================================
_____________________________________ ERROR collecting test_two.py ______________________________________
test_two.py:1: in <module>
    import asifjsafpo
E   ImportError: No module named 'asifjsafpo'
======================================== 1 error in 0.01 seconds ========================================
ERROR: Errors during collection, aborting
@fontealpina

This comment has been minimized.

Show comment
Hide comment
@fontealpina

fontealpina Feb 29, 2016

Great, it works as I was expecting!
Thanks

Great, it works as I was expecting!
Thanks

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Feb 29, 2016

Member

@fontealpina would you consider writing a short blog post with this trick in pytest-tricks? 😉

cc @hackebrot

Member

nicoddemus commented Feb 29, 2016

@fontealpina would you consider writing a short blog post with this trick in pytest-tricks? 😉

cc @hackebrot

@fontealpina

This comment has been minimized.

Show comment
Hide comment
@fontealpina

fontealpina Feb 29, 2016

@nicoddemus it is a pleasure for me. I'll do it as soon as possible.

@nicoddemus it is a pleasure for me. I'll do it as soon as possible.

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Feb 29, 2016

Member

Nice, thanks! 😎

Member

nicoddemus commented Feb 29, 2016

Nice, thanks! 😎

@hpk42

This comment has been minimized.

Show comment
Hide comment
@hpk42

hpk42 Mar 2, 2016

Contributor

Actually i think it's worthwhile to consider if we want to consider changing the default to stop if a collection error occurs. Having good defaults is a major goal IMHO. Who ever likes to execute tests when there are collection errors? Whenever i see them i usually want to first see what's going wrong with collection before considering anything else. If there is a real need we can introduce a --continue-on-collection-errors option or so.

Contributor

hpk42 commented Mar 2, 2016

Actually i think it's worthwhile to consider if we want to consider changing the default to stop if a collection error occurs. Having good defaults is a major goal IMHO. Who ever likes to execute tests when there are collection errors? Whenever i see them i usually want to first see what's going wrong with collection before considering anything else. If there is a real need we can introduce a --continue-on-collection-errors option or so.

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Mar 2, 2016

Member

Hmm, I actually agree with that - let's reopen this

Member

The-Compiler commented Mar 2, 2016

Hmm, I actually agree with that - let's reopen this

@The-Compiler The-Compiler reopened this Mar 2, 2016

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Mar 2, 2016

Member

I agree... if I see collection errors the first thing I do is to immediately hit CTRL+C to stop the run. 😁

Member

nicoddemus commented Mar 2, 2016

I agree... if I see collection errors the first thing I do is to immediately hit CTRL+C to stop the run. 😁

@nicoddemus nicoddemus added this to the 2.10 milestone Mar 2, 2016

@RonnyPfannschmidt RonnyPfannschmidt modified the milestones: 3.0, 2.10 Mar 2, 2016

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Mar 2, 2016

Member

since the collection behavior is a "breaking" change, we should do it on a major release

Member

RonnyPfannschmidt commented Mar 2, 2016

since the collection behavior is a "breaking" change, we should do it on a major release

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Mar 2, 2016

Member

IMHO this could go into a 2.10.0 release... it is a small behavior change, and to restore the previous behavior (which I doubt there are people depending on) is just a matter of adding a line to pytest.ini:

[pytest]
addopts = --continue-on-collection-errors

What IMHO should definitely go into a major release are API changes which break entire test suites.

Member

nicoddemus commented Mar 2, 2016

IMHO this could go into a 2.10.0 release... it is a small behavior change, and to restore the previous behavior (which I doubt there are people depending on) is just a matter of adding a line to pytest.ini:

[pytest]
addopts = --continue-on-collection-errors

What IMHO should definitely go into a major release are API changes which break entire test suites.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Mar 2, 2016

Member

we are certain to break behind a firewall ci setup in a minor release then

Member

RonnyPfannschmidt commented Mar 2, 2016

we are certain to break behind a firewall ci setup in a minor release then

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Mar 2, 2016

Member

Sorry, what do you mean by "firewall ci setup"?

Member

nicoddemus commented Mar 2, 2016

Sorry, what do you mean by "firewall ci setup"?

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Mar 2, 2016

Member

as in we are certain to "break" testsuites in private setups that previously ran in spite of a few minor collection errors

Member

RonnyPfannschmidt commented Mar 2, 2016

as in we are certain to "break" testsuites in private setups that previously ran in spite of a few minor collection errors

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Mar 2, 2016

Member

But those test suites were failing to anyway, no? 😉 After all a collection error does fail a test suite.

Member

nicoddemus commented Mar 2, 2016

But those test suites were failing to anyway, no? 😉 After all a collection error does fail a test suite.

@RonnyPfannschmidt RonnyPfannschmidt modified the milestones: 2.10, 3.0 Mar 2, 2016

@hpk42

This comment has been minimized.

Show comment
Hide comment
@hpk42

hpk42 Mar 2, 2016

Contributor

i think the compat aim of pytest minor releases (like 2.10 compared to 2.9) is that all previously passing test suites will still pass. Wouldn't make too big promises on details such as the one we are discussing.

Contributor

hpk42 commented Mar 2, 2016

i think the compat aim of pytest minor releases (like 2.10 compared to 2.9) is that all previously passing test suites will still pass. Wouldn't make too big promises on details such as the one we are discussing.

@ghostsquad

This comment has been minimized.

Show comment
Hide comment
@ghostsquad

ghostsquad Mar 2, 2016

why does pytest exit with code 0 when there is a collection error? I'm using --strict but it doesn't seem to have any effect on the exit code (which is needed for Continuous Build/Integration)

============ 16 passed, 1 pytest-warnings, 1 error in 1.72 seconds =============

Process finished with exit code 0

why does pytest exit with code 0 when there is a collection error? I'm using --strict but it doesn't seem to have any effect on the exit code (which is needed for Continuous Build/Integration)

============ 16 passed, 1 pytest-warnings, 1 error in 1.72 seconds =============

Process finished with exit code 0
@ghostsquad

This comment has been minimized.

Show comment
Hide comment
@ghostsquad

ghostsquad Mar 2, 2016

oh weird, this seems to only occur in PyCharm....

$ python3 -m pytest
...
16 passed, 1 pytest-warnings, 1 error in 0.65 seconds 
$ echo $?
1

oh weird, this seems to only occur in PyCharm....

$ python3 -m pytest
...
16 passed, 1 pytest-warnings, 1 error in 0.65 seconds 
$ echo $?
1
@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Mar 2, 2016

Member

@ghostsquad which version are you using? I just did a quick test here and things seem to work as expected:

# contents of foo.py
import kjanskjan
def test_foo():
    pass

# contents of bar.py
def test():
    pass
$ py.test foo.py bar.py -q
.
=================================== ERRORS ====================================
___________________________ ERROR collecting foo.py ___________________________
foo.py:1: in <module>
    import kjanskjan
E   ImportError: No module named 'kjanskjan'
1 passed, 1 error in 0.02 seconds

$ echo %ERRORLEVEL%
1

(Tested in pytest 2.9.0 and 2.8.7)

Member

nicoddemus commented Mar 2, 2016

@ghostsquad which version are you using? I just did a quick test here and things seem to work as expected:

# contents of foo.py
import kjanskjan
def test_foo():
    pass

# contents of bar.py
def test():
    pass
$ py.test foo.py bar.py -q
.
=================================== ERRORS ====================================
___________________________ ERROR collecting foo.py ___________________________
foo.py:1: in <module>
    import kjanskjan
E   ImportError: No module named 'kjanskjan'
1 passed, 1 error in 0.02 seconds

$ echo %ERRORLEVEL%
1

(Tested in pytest 2.9.0 and 2.8.7)

@ghostsquad

This comment has been minimized.

Show comment
Hide comment
@ghostsquad

ghostsquad Mar 2, 2016

@nicoddemus ya, it seems fine on the terminal, but within PyCharm CE 5.0.4, it is all "green" and exit code 0. py.tests configuration in PyCharm is default with options: --verbose --strict

I might have to bring this up to Jetbrains.

@nicoddemus ya, it seems fine on the terminal, but within PyCharm CE 5.0.4, it is all "green" and exit code 0. py.tests configuration in PyCharm is default with options: --verbose --strict

I might have to bring this up to Jetbrains.

@nicoddemus nicoddemus modified the milestones: 2.10, 3.0 Jun 26, 2016

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Jul 7, 2016

Member

Fixed in #1628 - thanks @omarkohl!

Member

The-Compiler commented Jul 7, 2016

Fixed in #1628 - thanks @omarkohl!

@pyup-bot pyup-bot referenced this issue in gasparka/pyha_demo_project Dec 18, 2017

Closed

Pin pytest to latest version 3.3.1 #6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment