Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-k EXPRESSION bug for disjunctions of parametrized tests #5881

Closed
leezu opened this issue Sep 24, 2019 · 4 comments · Fixed by #7122
Closed

-k EXPRESSION bug for disjunctions of parametrized tests #5881

leezu opened this issue Sep 24, 2019 · 4 comments · Fixed by #7122
Labels
topic: selection related to test selection from the command line

Comments

@leezu
Copy link
Contributor

leezu commented Sep 24, 2019

With pytest 5.1.3 on Python 3.7, pytest -k 'test_method or test_other' works as specified in the manual whereas pytest -k 'test_method[True] or test_other' fails with the following traceback:

Traceback (most recent call last):
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/main.py", line 191, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/main.py", line 234, in _main
    config.hook.pytest_collection(session=session)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/manager.py", line 92, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/manager.py", line 86, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/main.py", line 244, in pytest_collection
    return session.perform_collect()
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/main.py", line 440, in perform_collect
    session=self, config=self.config, items=items
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/manager.py", line 92, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/manager.py", line 86, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/mark/__init__.py", line 144, in pytest_collection_modifyitems
    deselect_by_keyword(items, config)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/mark/__init__.py", line 113, in deselect_by_keyword
    if keywordexpr and not matchkeyword(colitem, keywordexpr):
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/mark/legacy.py", line 100, in matchkeyword
    return eval(keywordexpr, {}, mapping)
  File "<string>", line 1, in <module>
TypeError: 'bool' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/NAME/.pyenv/versions/3.7.3/bin/pytest", line 10, in <module>
    sys.exit(main())
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/config/__init__.py", line 77, in main
    return config.hook.pytest_cmdline_main(config=config)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/manager.py", line 92, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/manager.py", line 86, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 208, in _multicall
    return outcome.get_result()
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/main.py", line 228, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/main.py", line 211, in wrap_session
    config.notify_exception(excinfo, config.option)
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/config/__init__.py", line 747, in notify_exception
    funcargs=True, showlocals=getattr(option, "showlocals", False), style=style
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/_code/code.py", line 580, in getrepr
    self._getreprcrash(),
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/_code/code.py", line 531, in _getreprcrash
    entry = self.traceback.getcrashentry()
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/_code/code.py", line 346, in getcrashentry
    if not entry.ishidden():
  File "/home/NAME/.pyenv/versions/3.7.3/lib/python3.7/site-packages/_pytest/_code/code.py", line 248, in ishidden
    tbh = f.f_locals.get(
AttributeError: 'KeywordMapping' object has no attribute 'get'
@ionelmc ionelmc transferred this issue from pytest-dev/pytest-cov Sep 25, 2019
@ionelmc
Copy link
Member

ionelmc commented Sep 25, 2019

@leezu I've transferred your issue to the right bugtracker.

@leezu
Copy link
Contributor Author

leezu commented Sep 25, 2019

@ionelmc thanks. I opened at pytest-dev/pytest-cov by accident..

@asottile
Copy link
Member

seems there's a combination of things happening here

first is that I don't believe -k directly supports parametrized names at all, the "expression" in -k is meant to be a python expression so what's attempted here is True[True] or True and so it is crashing

It's then additionally crashing in the crash-reporting code since we have a traceback frame that doesn't act like a dictionary -- it's the fake frame that implements -k heh

oddly enough, because True is also a valid expression, the only thing I could get working was -k (test_method and T and rue) or test_other -- not pretty but it ~kinda works -- you're probably better off using the selection by test id directly

@Zac-HD Zac-HD added the topic: selection related to test selection from the command line label Oct 2, 2019
@toslunar
Copy link

I found a workaround: pytest -k "(lambda: locals)()()['test_method[True]'] or test_other"

uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Jan 13, 2020
This package installs its 'tests' module into a non-package-specific
location in the root site-packages directory. Python packages should/usually
address this by excluding tests from installation using
setup.py: find_packages(exclude[ "<package-glob>" ]).

Unfortunately the above 'simple' change cannot be used because it doesn't
allow the package to be tested on Python 3.x, as the tests package is not
processed by 2to3, and the tests need to be run on the processed (by 2to3)
sources. This may be the original reason upstream didn't remove them from
the build, not grokking the ramifications.

Accordingly, this change moves the tests into the main package namespace
and runs the tests on the processed sources in the resulting build/*
directory. Remove CONFLICTS_INSTALL (to py-suds) accordingly.

While I'm here:

Patch the tests to support pytest > 4 and hack around the
inability to reference parametrized tests (test[foo]) in pytests -k
expressions. [1][2]

Level up ports compliance: LICENSE_FILE, USE{S} ordering, match COMMENT
to setup.py:description, match pkg-descr WWW URL to setup.py:homepage

QA: All tests pass on Python 2.7-3.8

[1] pytest-dev/pytest#5881
[2] pytest-dev/pytest#6177

PR:		226077
Reported by:	yuri [1]
Approved by:	portmgr (blanket: ports compliance)
MFH:		2020Q1 (blanket: ports compliance, port bugfixes)


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@522860 35697150-7ecd-e111-bb59-0022644237b5
uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Jan 13, 2020
This package installs its 'tests' module into a non-package-specific
location in the root site-packages directory. Python packages should/usually
address this by excluding tests from installation using
setup.py: find_packages(exclude[ "<package-glob>" ]).

Unfortunately the above 'simple' change cannot be used because it doesn't
allow the package to be tested on Python 3.x, as the tests package is not
processed by 2to3, and the tests need to be run on the processed (by 2to3)
sources. This may be the original reason upstream didn't remove them from
the build, not grokking the ramifications.

Accordingly, this change moves the tests into the main package namespace
and runs the tests on the processed sources in the resulting build/*
directory. Remove CONFLICTS_INSTALL (to py-suds) accordingly.

While I'm here:

Patch the tests to support pytest > 4 and hack around the
inability to reference parametrized tests (test[foo]) in pytests -k
expressions. [1][2]

Level up ports compliance: LICENSE_FILE, USE{S} ordering, match COMMENT
to setup.py:description, match pkg-descr WWW URL to setup.py:homepage

QA: All tests pass on Python 2.7-3.8

[1] pytest-dev/pytest#5881
[2] pytest-dev/pytest#6177

PR:		226077
Reported by:	yuri [1]
Approved by:	portmgr (blanket: ports compliance)
MFH:		2020Q1 (blanket: ports compliance, port bugfixes)
uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Jan 13, 2020
net/py-suds-jurko: Fix tests installed into shared location [1]

This package installs its 'tests' module into a non-package-specific
location in the root site-packages directory. Python packages should/usually
address this by excluding tests from installation using
setup.py: find_packages(exclude[ "<package-glob>" ]).

Unfortunately the above 'simple' change cannot be used because it doesn't
allow the package to be tested on Python 3.x, as the tests package is not
processed by 2to3, and the tests need to be run on the processed (by 2to3)
sources. This may be the original reason upstream didn't remove them from
the build, not grokking the ramifications.

Accordingly, this change moves the tests into the main package namespace
and runs the tests on the processed sources in the resulting build/*
directory. Remove CONFLICTS_INSTALL (to py-suds) accordingly.

While I'm here:

Patch the tests to support pytest > 4 and hack around the
inability to reference parametrized tests (test[foo]) in pytests -k
expressions. [1][2]

Level up ports compliance: LICENSE_FILE, USE{S} ordering, match COMMENT
to setup.py:description, match pkg-descr WWW URL to setup.py:homepage

QA: All tests pass on Python 2.7-3.8

[1] pytest-dev/pytest#5881
[2] pytest-dev/pytest#6177

PR:		226077
Reported by:	yuri [1]
Approved by:	portmgr (blanket: ports compliance)

net/py-suds-jurko: Bump PORTREVISION missed in ports r522860

PR:	226077

Approved by:	ports-secteam (blanket(s): ports compliance, port bugfixes)
Jehops pushed a commit to Jehops/freebsd-ports-legacy that referenced this issue Jan 13, 2020
This package installs its 'tests' module into a non-package-specific
location in the root site-packages directory. Python packages should/usually
address this by excluding tests from installation using
setup.py: find_packages(exclude[ "<package-glob>" ]).

Unfortunately the above 'simple' change cannot be used because it doesn't
allow the package to be tested on Python 3.x, as the tests package is not
processed by 2to3, and the tests need to be run on the processed (by 2to3)
sources. This may be the original reason upstream didn't remove them from
the build, not grokking the ramifications.

Accordingly, this change moves the tests into the main package namespace
and runs the tests on the processed sources in the resulting build/*
directory. Remove CONFLICTS_INSTALL (to py-suds) accordingly.

While I'm here:

Patch the tests to support pytest > 4 and hack around the
inability to reference parametrized tests (test[foo]) in pytests -k
expressions. [1][2]

Level up ports compliance: LICENSE_FILE, USE{S} ordering, match COMMENT
to setup.py:description, match pkg-descr WWW URL to setup.py:homepage

QA: All tests pass on Python 2.7-3.8

[1] pytest-dev/pytest#5881
[2] pytest-dev/pytest#6177

PR:		226077
Reported by:	yuri [1]
Approved by:	portmgr (blanket: ports compliance)
MFH:		2020Q1 (blanket: ports compliance, port bugfixes)


git-svn-id: svn+ssh://svn.freebsd.org/ports/head@522860 35697150-7ecd-e111-bb59-0022644237b5
bluetech added a commit to bluetech/pytest that referenced this issue Apr 25, 2020
Previously, the expressions given to the `-m` and `-k` options were
evaluated with `eval`. This causes a few issues:

- Python keywords cannot be used.

- Constants like numbers, None, True, False are not handled correctly.

- Various syntax like numeric operators and `X if Y else Z` is supported
  unintentionally.

- `eval()` is somewhat dangerous for arbitrary input.

- Can fail in many ways so requires `except Exception`.

The format we want to support is quite simple, so change to a custom
parser. This fixes the issues above, and gives us full control of the
format, so can be documented comprehensively and even be extended in the
future if we wish.

Fixes pytest-dev#1141.
Fixes pytest-dev#3573.
Fixes pytest-dev#5881.
Fixes pytest-dev#6822.
Fixes pytest-dev#7112.
bluetech added a commit to bluetech/pytest that referenced this issue Apr 25, 2020
Previously, the expressions given to the `-m` and `-k` options were
evaluated with `eval`. This causes a few issues:

- Python keywords cannot be used.

- Constants like numbers, None, True, False are not handled correctly.

- Various syntax like numeric operators and `X if Y else Z` is supported
  unintentionally.

- `eval()` is somewhat dangerous for arbitrary input.

- Can fail in many ways so requires `except Exception`.

The format we want to support is quite simple, so change to a custom
parser. This fixes the issues above, and gives us full control of the
format, so can be documented comprehensively and even be extended in the
future if we wish.

Fixes pytest-dev#1141.
Fixes pytest-dev#3573.
Fixes pytest-dev#5881.
Fixes pytest-dev#6822.
Fixes pytest-dev#7112.
uqs pushed a commit to freebsd/freebsd-ports that referenced this issue Apr 1, 2021
net/py-suds-jurko: Fix tests installed into shared location [1]

This package installs its 'tests' module into a non-package-specific
location in the root site-packages directory. Python packages should/usually
address this by excluding tests from installation using
setup.py: find_packages(exclude[ "<package-glob>" ]).

Unfortunately the above 'simple' change cannot be used because it doesn't
allow the package to be tested on Python 3.x, as the tests package is not
processed by 2to3, and the tests need to be run on the processed (by 2to3)
sources. This may be the original reason upstream didn't remove them from
the build, not grokking the ramifications.

Accordingly, this change moves the tests into the main package namespace
and runs the tests on the processed sources in the resulting build/*
directory. Remove CONFLICTS_INSTALL (to py-suds) accordingly.

While I'm here:

Patch the tests to support pytest > 4 and hack around the
inability to reference parametrized tests (test[foo]) in pytests -k
expressions. [1][2]

Level up ports compliance: LICENSE_FILE, USE{S} ordering, match COMMENT
to setup.py:description, match pkg-descr WWW URL to setup.py:homepage

QA: All tests pass on Python 2.7-3.8

[1] pytest-dev/pytest#5881
[2] pytest-dev/pytest#6177

PR:		226077
Reported by:	yuri [1]
Approved by:	portmgr (blanket: ports compliance)

net/py-suds-jurko: Bump PORTREVISION missed in ports r522860

PR:	226077

Approved by:	ports-secteam (blanket(s): ports compliance, port bugfixes)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: selection related to test selection from the command line
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants