Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exit pytest if a collection error occours #1421

Closed
fontealpina opened this issue Feb 29, 2016 · 20 comments
Closed

Exit pytest if a collection error occours #1421

fontealpina opened this issue Feb 29, 2016 · 20 comments

Comments

@fontealpina
Copy link

@fontealpina fontealpina commented Feb 29, 2016

If a collection error occurs, is there a way to terminate immediately pytest execution after collection, avoiding the execution of all the tests (even the ones properly collected)?
Basically I need to start test execution only if there are no import errors.
So for instance, now if there is a collection error I get:

$ pytest test/test_01/test_01.py test/test_02/test_02.py test/test_03/test_03.py -v -s
============================ test session starts =============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /usr/bin/python
collected 2 items / 1 errors

test/test_01/test_01.py::TestBug::test_n_01 PASSED
test/test_02/test_02.py::TestBug::test_n_02 PASSED

=================================== ERRORS ===================================
________________ ERROR collecting test/test_03/test_03.py _________________
import file mismatch:
a description of the import error
================ 2 passed, 1 error in 0.17 seconds ================

I would get something like this instead:

$ pytest test/test_01/test_01.py test/test_02/test_02.py test/test_03/test_03.py -v -s
============================ test session starts =============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /usr/bin/python
collected 2 items / 1 errors
=================================== ERRORS =============================
________________ ERROR collecting test/test_03/test_03.py _______________________
import file mismatch:
a description of the import error
================================ 1 error in 0.17 seconds ======================

Is there an options that allows that?
Or maybe this can be done with a plugin?
Thanks

@The-Compiler
Copy link
Member

@The-Compiler The-Compiler commented Feb 29, 2016

You can easily do this with a local plugin, e.g. by putting this in a conftest.py:

import pytest

def pytest_collectreport(report):
    if report.failed:
        raise pytest.UsageError("Errors during collection, aborting")

Then even if tests are collected, your hook will exit before they are run:

collecting 1 items / 1 errors
================================================ ERRORS =================================================
_____________________________________ ERROR collecting test_two.py ______________________________________
test_two.py:1: in <module>
    import asifjsafpo
E   ImportError: No module named 'asifjsafpo'
======================================== 1 error in 0.01 seconds ========================================
ERROR: Errors during collection, aborting
@fontealpina
Copy link
Author

@fontealpina fontealpina commented Feb 29, 2016

Great, it works as I was expecting!
Thanks

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Feb 29, 2016

@fontealpina would you consider writing a short blog post with this trick in pytest-tricks? 😉

cc @hackebrot

@fontealpina
Copy link
Author

@fontealpina fontealpina commented Feb 29, 2016

@nicoddemus it is a pleasure for me. I'll do it as soon as possible.

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Feb 29, 2016

Nice, thanks! 😎

@hpk42
Copy link
Contributor

@hpk42 hpk42 commented Mar 2, 2016

Actually i think it's worthwhile to consider if we want to consider changing the default to stop if a collection error occurs. Having good defaults is a major goal IMHO. Who ever likes to execute tests when there are collection errors? Whenever i see them i usually want to first see what's going wrong with collection before considering anything else. If there is a real need we can introduce a --continue-on-collection-errors option or so.

@The-Compiler
Copy link
Member

@The-Compiler The-Compiler commented Mar 2, 2016

Hmm, I actually agree with that - let's reopen this

@The-Compiler The-Compiler reopened this Mar 2, 2016
@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Mar 2, 2016

I agree... if I see collection errors the first thing I do is to immediately hit CTRL+C to stop the run. 😁

@nicoddemus nicoddemus added this to the 2.10 milestone Mar 2, 2016
@RonnyPfannschmidt RonnyPfannschmidt modified the milestones: 3.0, 2.10 Mar 2, 2016
@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Mar 2, 2016

since the collection behavior is a "breaking" change, we should do it on a major release

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Mar 2, 2016

IMHO this could go into a 2.10.0 release... it is a small behavior change, and to restore the previous behavior (which I doubt there are people depending on) is just a matter of adding a line to pytest.ini:

[pytest]
addopts = --continue-on-collection-errors

What IMHO should definitely go into a major release are API changes which break entire test suites.

@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Mar 2, 2016

we are certain to break behind a firewall ci setup in a minor release then

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Mar 2, 2016

Sorry, what do you mean by "firewall ci setup"?

@RonnyPfannschmidt
Copy link
Member

@RonnyPfannschmidt RonnyPfannschmidt commented Mar 2, 2016

as in we are certain to "break" testsuites in private setups that previously ran in spite of a few minor collection errors

@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Mar 2, 2016

But those test suites were failing to anyway, no? 😉 After all a collection error does fail a test suite.

@RonnyPfannschmidt RonnyPfannschmidt modified the milestones: 2.10, 3.0 Mar 2, 2016
@hpk42
Copy link
Contributor

@hpk42 hpk42 commented Mar 2, 2016

i think the compat aim of pytest minor releases (like 2.10 compared to 2.9) is that all previously passing test suites will still pass. Wouldn't make too big promises on details such as the one we are discussing.

@ghostsquad
Copy link

@ghostsquad ghostsquad commented Mar 2, 2016

why does pytest exit with code 0 when there is a collection error? I'm using --strict but it doesn't seem to have any effect on the exit code (which is needed for Continuous Build/Integration)

============ 16 passed, 1 pytest-warnings, 1 error in 1.72 seconds =============

Process finished with exit code 0
@ghostsquad
Copy link

@ghostsquad ghostsquad commented Mar 2, 2016

oh weird, this seems to only occur in PyCharm....

$ python3 -m pytest
...
16 passed, 1 pytest-warnings, 1 error in 0.65 seconds 
$ echo $?
1
@nicoddemus
Copy link
Member

@nicoddemus nicoddemus commented Mar 2, 2016

@ghostsquad which version are you using? I just did a quick test here and things seem to work as expected:

# contents of foo.py
import kjanskjan
def test_foo():
    pass

# contents of bar.py
def test():
    pass
$ py.test foo.py bar.py -q
.
=================================== ERRORS ====================================
___________________________ ERROR collecting foo.py ___________________________
foo.py:1: in <module>
    import kjanskjan
E   ImportError: No module named 'kjanskjan'
1 passed, 1 error in 0.02 seconds

$ echo %ERRORLEVEL%
1

(Tested in pytest 2.9.0 and 2.8.7)

@ghostsquad
Copy link

@ghostsquad ghostsquad commented Mar 2, 2016

@nicoddemus ya, it seems fine on the terminal, but within PyCharm CE 5.0.4, it is all "green" and exit code 0. py.tests configuration in PyCharm is default with options: --verbose --strict

I might have to bring this up to Jetbrains.

@nicoddemus nicoddemus modified the milestones: 2.10, 3.0 Jun 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.