filter expressions unreliable when parametrizing test

[rasschaert]

Filter expressions (-k) become very unreliable when you try to match the parameters in parameterized fixtures. The filters seem to match things they shouldn't.

Minimal example:

import pytest

@pytest.mark.parametrize(
    'bar', [
        'xyz',
        'zzz',
        'ttt',
        'qqq'
    ],
)
def test_foo(bar):
    assert(True)

Filtering for 'z' matches everything (which is, I'm pretty sure, incorrect behaviour).

[kenny@pc test]$ pytest -v -p no:sugar -k 'z'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[xyz] PASSED                                         [ 25%]
test_foo.py::test_foo[zzz] PASSED                                         [ 50%]
test_foo.py::test_foo[ttt] PASSED                                         [ 75%]
test_foo.py::test_foo[qqq] PASSED                                         [100%]

=========================== 4 passed in 0.01 seconds ============================

Different filters do work as expected.

[kenny@pc test]$ pytest -v -p no:sugar -k 'q or x'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[xyz] PASSED                                         [ 50%]
test_foo.py::test_foo[qqq] PASSED                                         [100%]

============================== 2 tests deselected ===============================
==================== 2 passed, 2 deselected in 0.01 seconds =====================

My system is Arch Linux, the relevant software versions are in my pytest output.
pip list:

acdcli (0.3.2)
apipkg (1.4)
appdirs (1.4.3)
Archey3 (0.4)
asciinema (2.0.0)
asn1crypto (0.24.0)
attrs (17.4.0)
Automat (0.6.0)
Babel (2.5.3)
bcrypt (3.1.4)
beautifulsoup4 (4.6.0)
bottle (0.12.13)
broadlink (0.3)
cached-property (1.3.1)
caffeine-ng (3.4.0)
cairocffi (0.6)
CairoSVG (1.0.9)
cffi (1.11.4)
chardet (3.0.4)
click (6.7)
cliff (2.11.0)
cmd2 (0.8.0)
colorama (0.3.9)
constantly (15.1.0)
cryptography (2.1.4)
cssselect (0.9.1)
cupshelpers (1.0)
cycler (0.10.0)
debtcollector (1.19.0)
decorator (4.2.1)
deprecation (1.0.1)
docker (3.0.1)
docker-compose (1.19.0)
docker-pycreds (0.2.2)
dockerpty (0.4.1)
docopt (0.6.2)
dogpile.cache (0.6.4)
ds4drv (0.5.1)
evdev (0.7.0)
ewmh (0.1.6)
execnet (1.5.0)
Flask (0.12.2)
Flask-SQLAlchemy (2.3.2.dev20171011)
fusepy (2.0.4)
Glances (2.11.1)
gscreenshot (2.10.0)
html5lib (0.999)
hyperlink (17.3.1)
idna (2.6)
incremental (17.5.0)
iotop (0.6)
ipykernel (4.6.1)
ipython (6.2.1)
ipython-genutils (0.1.0)
ipywidgets (6.0.0)
isc (2.0)
iso8601 (0.1.12)
itsdangerous (0.24)
jedi (0.11.1)
Jinja2 (2.7.3)
jsonpatch (1.21)
jsonpointer (2.0)
jsonschema (2.6.0)
jupyter-client (5.1.0)
jupyter-console (5.2.0)
jupyter-core (4.4.0)
jupyterthemes (0.18.2)
keystoneauth1 (3.4.0)
keyutils (0.5)
lesscpy (0.13.0)
louis (3.4.0)
lxc (0.1)
lxml (3.4.0)
Markdown (2.5.1)
MarkupSafe (0.23)
matplotlib (2.1.2)
mccabe (0.6.1)
meld (3.18.0)
mistune (0.8.1)
msgpack (0.5.4)
multi-key-dict (2.0.3)
namcap (3.2.7)
nbformat (4.4.0)
netaddr (0.7.19)
netifaces (0.10.6)
nose (1.3.7)
notebook (5.0.0)
numpy (1.14.0)
olefile (0.45.1)
openstacksdk (0.9.19)
os-client-config (1.28.0)
osc-lib (1.9.0)
oslo.config (5.2.0)
oslo.i18n (3.19.0)
oslo.serialization (2.24.0)
oslo.utils (3.35.0)
packaging (16.8)
paramiko (2.4.0)
parsel (1.4.0)
parso (0.1.1)
path.py (10.4)
pbr (3.1.1)
peewee (3.0.18)
pexpect (4.3.1)
pickleshare (0.7.4)
Pillow (5.0.0)
pip (9.0.1)
pluggy (0.6.0)
ply (3.10)
positional (1.1.2)
prettytable (0.7.2)
prompt-toolkit (1.0.15)
psutil (5.4.3)
ptpython (0.41)
ptyprocess (0.5.2)
py (1.5.2)
pyalpm (0.8)
pyasn1 (0.4.2)
pyasn1-modules (0.2.1)
PyBluez (0.22)
pycairo (1.16.2)
pycodestyle (2.3.1)
pycparser (2.10)
pycrypto (2.6.1)
pycups (1.9.73)
pycurl (7.43.0.1)
PyDispatcher (2.0.5)
pydocstyle (2.1.1)
pyelftools (0.24)
pyflakes (1.6.0)
Pygments (2.2.0)
pygobject (3.26.1)
pylama (7.4.3)
PyNaCl (1.2.1)
pyOpenSSL (17.5.0)
pyparsing (2.2.0)
pyperclip (1.6.0)
Pyphen (0.9.1)
pytest (3.4.1)
pytest-forked (0.2)
pytest-interactive (0.1.4)
pytest-sugar (0.9.1)
pytest-xdist (1.22.1)
python-cinderclient (3.2.0)
python-dateutil (2.6.1)
python-glanceclient (2.9.1)
python-jenkins (0.4.15)
python-keystoneclient (3.15.0)
python-novaclient (10.1.0)
python-openstackclient (3.14.0)
python-xlib (0.21)
pytz (2018.3)
pyudev (0.21.0.dev20161225)
pyxdg (0.25)
PyYAML (3.11)
pyzmq (16.0.3)
qtconsole (4.3.1)
queuelib (1.4.2)
ranger-fm (1.9.0)
requests (2.18.4)
requests-toolbelt (0.8.0)
requestsexceptions (1.4.0)
rfc3986 (1.1.0)
Scrapy (1.5.0)
service-identity (17.0.0)
setproctitle (1.1.10)
setuptools (38.5.1)
simplegeneric (0.8.1)
simplejson (3.13.2)
six (1.11.0)
snowballstemmer (1.2.1)
speedtest-cli (1.0.7)
SQLAlchemy (1.2.3)
sshuttle (0.78.3)
stevedore (1.28.0)
tablib (0.11.3)
team (1.0)
termcolor (1.1.0)
terminado (0.8.1)
testinfra (1.10.1)
texttable (1.2.1)
tinycss (0.3)
tornado (4.5.2)
tqdm (4.19.5)
traitlets (4.3.2)
Twisted (17.9.0)
udiskie (1.7.3)
urllib3 (1.22)
w3lib (1.19.0)
warlock (1.3.0)
wcwidth (0.1.7)
WeasyPrint (0.23)
websocket-client (0.46.0)
Werkzeug (0.14.1)
wheel (0.30.0)
widgetsnbextension (2.0.0)
wrapt (1.10.11)
zeroconf (0.19.1)
zope.interface (4.4.3)

Being able to select/deselect certain tests based on the parameter names is quite important to my use case. Please let me know if I can further help narrow down the issue.

[RonnyPfannschmidt]

i suspect the keyword expression is matching the parametrize mark as well

[rasschaert]

Hah! Good catch!

That means I constructed my minimal example poorly, since that's not the case in the real-world issue I'm having.

This one demonstrates the issue I'm having better:

import pytest

@pytest.mark.parametrize(
    'bar', [
        'abc-1',
        'def-1',
        'ghi-1',
        'jkl-1'
    ],
)
def test_foo(bar):
    assert(True)

[kenny@pc test]$ pytest -v -p no:sugar -k 'abc-1 and jkl-1'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[def-1] PASSED                                       [ 50%]
test_foo.py::test_foo[ghi-1] PASSED                                       [100%]

============================== 2 tests deselected ===============================
==================== 2 passed, 2 deselected in 0.01 seconds =====================


[kenny@pc test]$ pytest -v -p no:sugar -k 'abc-1 and ghi-1'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[def-1] PASSED                                       [ 50%]
test_foo.py::test_foo[jkl-1] PASSED                                       [100%]

============================== 2 tests deselected ===============================
==================== 2 passed, 2 deselected in 0.01 seconds =====================


[kenny@pc test]$ pytest -v -p no:sugar -k 'abc-1 or ghi-1'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[abc-1] PASSED                                       [ 25%]
test_foo.py::test_foo[def-1] PASSED                                       [ 50%]
test_foo.py::test_foo[ghi-1] PASSED                                       [ 75%]
test_foo.py::test_foo[jkl-1] PASSED                                       [100%]

=========================== 4 passed in 0.01 seconds ============================

Take out the dashes from the parameters and filters and it behaves as expected:

[kenny@pc test]$ pytest -v -p no:sugar -k 'abc1 and ghi1'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

============================== 4 tests deselected ===============================
========================= 4 deselected in 0.01 seconds ==========================
[kenny@pc test]$ pytest -v -p no:sugar -k 'abc1 or ghi1'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[abc1] PASSED                                        [ 50%]
test_foo.py::test_foo[ghi1] PASSED                                        [100%]

============================== 2 tests deselected ===============================
==================== 2 passed, 2 deselected in 0.01 seconds =====================

I've found elsewhere in this issue tracker that filters are treated as Python expressions. That probably means that the dash is interpreted as a minus sign instead of a literal character.

Unfortunately I really need those dashes in my parameters and I need to be able to filter on them.

[rasschaert]

Meanwhile I've found an ugly workaround that allows me to apply the filters I want.

Replacing the dashes in with underscores in the ids and then also using underscores in the filter works well, without actually changing the original parameter that is fed into my test.

import pytest

@pytest.mark.parametrize(
    'bar', [
        'abc-1',
        'def-1',
        'ghi-1',
        'jkl-1'
    ],
    ids=lambda x: x.replace('-', '_')
)
def test_foo(bar):
    assert("-" in bar and "_" not in bar)

[kenny@pc test]$ pytest -v -p no:sugar -k 'abc_1 or jkl_1'
============================== test session starts ==============================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/kenny/test, inifile:
plugins: testinfra-1.10.1, xdist-1.22.1, interactive-0.1.4, forked-0.2, pylama-7.4.3
collected 4 items                                                               

test_foo.py::test_foo[abc_1] PASSED                                       [ 50%]
test_foo.py::test_foo[jkl_1] PASSED                                       [100%]

============================== 2 tests deselected ===============================
==================== 2 passed, 2 deselected in 0.01 seconds =====================

[RonnyPfannschmidt]

@rasschaert thans for showing your solution , its a fine hack to work around that current shortcoming of pytest

we hope to eventually enable better selection fo marks, but thats still far off

[nicoddemus]

Closing as this has not seen activity in awhile.

[RonnyPfannschmidt]

@nicoddemus this one hasnt been fixed as far as i can tell

[nicoddemus]

Thanks @RonnyPfannschmidt for catching my slip up.

[nicoddemus]

Looking again at this, I'm not sure if this is solvable: -k matches against mark names as well, so this is a "feature".

If we want to change how -k works (personally I'm not sure if it is worth the trouble), should we create a new proposal ticket for it?

[RonnyPfannschmidt]

personally i wonder if we should just introduce a new option to express filters and then at a later point deprecate -m and -k