New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skipif + parametrize not skipping tests #1296

Closed
arxanas opened this Issue Dec 30, 2015 · 12 comments

Comments

Projects
7 participants
@arxanas

arxanas commented Dec 30, 2015

I have this testing code:

import pytest


def params():
    dont_skip = pytest.mark.skipif(False, reason="don't skip")
    return [dont_skip("foo"), dont_skip("bar")]


@pytest.mark.skipif(True, reason="always skip")
@pytest.mark.parametrize("param", params())
@pytest.mark.skipif(True, reason="really always skip please")
def test_foo(param):
    assert False

The expected behavior is that if any of the skipif conditions returns True, the test is skipped. The actual behavior is that the skipif condition seems to depend entirely on the value of the dont_skip parameter. For example, the following test case causes the tests to be skipped (the Trues and Falses are swapped around):

import pytest


def params():
    dont_skip = pytest.mark.skipif(True, reason="don't skip")
    return [dont_skip("foo"), dont_skip("bar")]


@pytest.mark.skipif(False, reason="always skip")
@pytest.mark.parametrize("param", params())
@pytest.mark.skipif(False, reason="really always skip please")
def test_foo(param):
    assert False

Here's the full output of the first code sample:

$ py.test test_foo.py
============================= test session starts ==============================
platform darwin -- Python 2.7.10, pytest-2.8.6.dev1, py-1.4.31, pluggy-0.3.1
rootdir: /Volumes/Home/Users/Waleed/tmp/foo, inifile: 
collected 2 items

test_foo.py FF

=================================== FAILURES ===================================
________________________________ test_foo[foo] _________________________________

param = 'foo'

    @pytest.mark.skipif(True, reason="always skip")
    @pytest.mark.parametrize("param", params())
    @pytest.mark.skipif(True, reason="really always skip please")
    def test_foo(param):
>       assert False
E       assert False

test_foo.py:13: AssertionError
________________________________ test_foo[bar] _________________________________

param = 'bar'

    @pytest.mark.skipif(True, reason="always skip")
    @pytest.mark.parametrize("param", params())
    @pytest.mark.skipif(True, reason="really always skip please")
    def test_foo(param):
>       assert False
E       assert False

test_foo.py:13: AssertionError
=========================== 2 failed in 0.01 seconds ===========================

The issue manifests under:

  • Python 3.5 + Pytest 2.8.5
  • Python 2.7.10 + Pytest 2.8.5
  • Python 2.7.10 + Pytest HEAD (0ef73ed).

I'm running OS X 10.9.5.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Dec 30, 2015

Thanks for the report! 😄

@kdexd

This comment has been minimized.

kdexd commented Dec 12, 2016

I have recently experienced this issue. It still persists in pytest 3.0.0.

@kdexd kdexd referenced this issue Dec 12, 2016

Closed

Move to py.test? #411

@alessss

This comment has been minimized.

alessss commented Jan 9, 2017

Is there any solution for the issue? Still experiencing it in pytest==3.0.5

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Jan 9, 2017

@alessss markers have not been fixed, this issue is not easy to resolve due to legacy

@alessss

This comment has been minimized.

alessss commented Jan 9, 2017

If i use plain True in skipif decorator it works well:
@pytest.mark.skipif(True, reason="always skip")

But if there is some function, returning True or False - tests don't start.
from distutils.version import StrictVersion
from versions import get_version
@pytest.mark.skipif(StrictVersion(get_version()) < StrictVersion('1.8.0'), reason='WRONG VERSION')

Is there any workaround for this?

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Jan 9, 2017

@alessss I don't see how that'd be possible - and if you don't also have skipif markers on parameters, this issue is totally unrelated.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Jan 17, 2017

For reference, #1921 discusses how to start fixing markers in general.

@scopatz

This comment has been minimized.

scopatz commented Feb 12, 2017

This is a non-obvious pain point I am experiencing on v3.0.6

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Feb 13, 2017

@hpk42 i beleive this one breaks apart due to yet another marker transfer detail included in parametrization

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Jun 23, 2017

we can fix this by landing #2522 as well

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Apr 10, 2018

fixed in #3317

@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Apr 10, 2018

fixed in #3317

@RonnyPfannschmidt RonnyPfannschmidt moved this from in progress to done in resolve the mark fallout Apr 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment