Raise CollectError if pytest.skip() is called during collection #1519
I'll add the
This change effectively disables using
This earlier comment by @hpk42 is also an option:
I tried checking the type of the
Here's a quick checklist that should be present in PRs:
pytest.skip() must not be used at module level because it can easily be
This is unexpected for users used to the @unittest.skip decorator and therefore
The pytest equivalent of @unittest.skip is @pytest.mark.skip .
Adapt existing tests that were actually relying on this behaviour and add a
good thinking to just catch it - that way around is far more easy to detect, and requires far less logic by simply requiring the user to correct his misuse
i'm a bit surprised by the "unrelated" tests removals,
i'm also surprised that most testing is via junit-xml, i believe/hope this can be done with the normal inline run at least
could you take a look at how easy/feasible that is
@RonnyPfannschmidt I know what you mean by 'unrelated tests removal' (even though they are not really unrelated because the removed/modified tests all rely on pytest.skip at a module level).
This is what I previously wrote to @nicoddemus by mail:
He recommended creating a pull request and discussing it here :-)
I agree that this commit breaks external contracts because currently it is possible to use
This could be legitimate:
import pytest pytest.skip("This whole module is currently broken") def test_one(): ...
This is misleading:
import pytest @pytest.skip("Test is broken") def test_one(): ...
The main test for my change is
thanks for the clarifications, it makes me wonder if we should test junitxml output in a more localized manner in future
with the "unrelated" test i meant the change in result-log - its basically a legacy test result output format
i would appreciate the return of removed test in runner as it is a very local,
also well done
@hpk42 and @nicoddemus : i think we need a target branch for this one as it cant be part of just a feature release, since its a "breaking change" - alternatively we can bump the next version in the features branch to 3.0 ?
You mean avoid removing
I don't understand what you mean. Could you give some more details?
Hi @omarkohl, first of all thanks for all the work!
That example could easily be changed to:
import pytest pytestmark = pytest.mark.skip(reason="This whole module is currently broken") def test_one(): ...
So IMHO this would be reasonable to ask users to change this rather unusual use-case in face of the benefits of someone misusing
I'm not entirely sure this qualifies as an API change, it seems to me more that it fixes an API misuse. On the other hand, pytest's own test suite relied on this, so one might argue that it is actually a feature change.
I'm fine with this going to 3.0 only.
@nicoddemus i agree that the old behavior was unintended/bad, however fixing it required a change in how certain exceptions propagate, and thus how py.test interacts with the testing of stuff
this will turns skips at user projects to collection errors, thus "breaking" their builds on update
if we want to be able to push this into a feature release, it would have to us a different mechanism to warn the user of his missuse
pytest.skip() must not be used at module level because it can easily be misunderstood and used as a decorator instead of pytest.mark.skip, causing the whole module to be skipped instead of just the test being decorated. This is unexpected for users used to the @unittest.skip decorator and therefore it is best to bail out with a clean error when it happens. The pytest equivalent of @unittest.skip is @pytest.mark.skip . Adapt existing tests that were actually relying on this behaviour and add a test that explicitly test that collection fails. fix #607