Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Error when rerun occurs on tests that are members of a class using mark. #3555

Closed
shaleh opened this Issue Jun 8, 2018 · 27 comments

Comments

Projects
None yet
4 participants
@shaleh
Copy link

shaleh commented Jun 8, 2018

(Updated, added into about the marker registration)

We have some integration tests using selenium that pytest runs for us.

When a test fails and is rerun we see this:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/main.py", line 236, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pytest_rerunfailures.py", line 146, in pytest_runtest_protocol
INTERNALERROR>     reports = runtestprotocol(item, nextitem=nextitem, log=False)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/runner.py", line 73, in runtestprotocol
INTERNALERROR>     rep = call_and_report(item, "setup", log)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/runner.py", line 161, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 617, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/integrations/testrail/plugin.py", line 65, in pytest_runtest_makereport
INTERNALERROR>     if item.get_marker(TESTRAIL_PREFIX):
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/nodes.py", line 231, in get_marker
INTERNALERROR>     return MarkInfo(markers)
INTERNALERROR>   File "<attrs generated init ca81e36d524e8623cbf31cadd240224be62448db>", line 6, in __init__
INTERNALERROR>     self.combined = __attr_factory_combined(self)
INTERNALERROR>   File "/Users/seanperry/repos/work/integration-tests/env/lib/python2.7/site-packages/_pytest/mark/structures.py", line 288, in <lambda>
INTERNALERROR>     lambda self: reduce(Mark.combined_with, self._marks), takes_self=True
INTERNALERROR> TypeError: unbound method combined_with() must be called with Mark instance as first argument (got MarkDecorator instance instead)

The tests that are being rerun are methods of a test class declared like this:

from integrations.testrail.plugin import report_testrail

@pytest.mark.thing1
class TestSomethingWithinThing1(object)
    @pytest.fixture(autouse=True)
    def store_thing1_item(self, thing1_item):
        self.thing1_item = thing1_item

    @report_testrail('ABC123')
    def test_blah(self):
        # test something that fails and is rerun.....

There are no marks on the fixtures. The thing1 mark is registered in pytest.ini like so:

[pytest]
markers =
    thing1: Thing1 does this.....
    thing2: Thing2 does that....

Python version

$ env/bin/python --version
Python 2.7.15

pip list says:

env/bin/pip list
Package              Version
-------------------- ---------
atomicwrites         1.1.5
attrs                18.1.0
certifi              2018.4.16
chardet              3.0.4
click                6.7
configparser         3.5.0
EasyProcess          0.2.3
enum34               1.1.6
first                2.0.1
flake8               3.5.0
funcsigs             1.0.2
idna                 2.6
invoke               1.0.0
mccabe               0.6.1
mock                 2.0.0
more-itertools       4.2.0
pbr                  4.0.4
pip                  10.0.1
pip-tools            2.0.2
pluggy               0.6.0
py                   1.5.3
pycodestyle          2.3.1
pyflakes             1.6.0
pytest               3.6.1
pytest-rerunfailures 4.1
pytz                 2018.4
PyVirtualDisplay     0.2.1
requests             2.18.4
selenium             3.12.0
setuptools           39.2.0
six                  1.11.0
urllib3              1.22
wheel                0.31.1

I am running on OSX:

uname -a
Darwin My-MacBook-Pro.local 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Thank you.

@pytestbot

This comment has been minimized.

Copy link

pytestbot commented Jun 8, 2018

GitMate.io thinks possibly related issues are #3163 (Pytest internal error due to infinite recursion on final test), #50 (Select tests according to their mark), #3494 (Cannot select test by mark using lower case param), #120 (py.test.mark.parameterize works with test functions, but not classes), and #814 (Mark for flaky tests).

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 8, 2018

Thanks @pytestbot but none of those seem relevant. (Yes, I know this is a bot).

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 9, 2018

the issue report is incomplete, off-hand i have no idea what could be wrong please provide at least a runable example

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

I posted the code as best I could. It is not publicly licensed. I will need to make a new repo and try to reproduce it independently. I was hoping the code above was enough for someone to tells us we are doing something wrong.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

@shaleh its completely unclear where the markdeorator is from without the code of the decorators as well

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

also the missing traceback leaves no clue for a indication

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

I did not include the traceback because it did not appear helpful. I will see what I can do about adding it.

I did not think about the definition of the marker. Thank you, I will update the Issue with a recreation of it.

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

Ok, I added more info about the marker registration.

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

And full traceback is now included.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

this is a bug in custom marker application, please post the helper

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

I do not have any special custom markers. I use the registered 'thing1' or 'thing2' nothing more.

grep -r mark applications/ | grep -v thing1 | grep -v thing2 returns nothing.

Which helper do you need?

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

that report_testrail helper seems like a possible cause - other projects had similar issues with bugs in markers where they took a markinfo marker collection instead of a marker

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

Thanks for the pointer. I will go look.

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

def report_testrail(case_id):
    def decorator(function):
        function.case_id = case_id
        return function
    return decorator

Really simple.

Is it supposed to take a marker parameter? What about tests without a marker?

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

@shaleh in the integration something else is going on, please show the complete code

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

No really, that is it :-) I pasted that directly from the real source.

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

It just adds the id to the function. Code elsewhere assumes the function has a case_id and uses it.

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

However, this may be relevant:

    @pytest.hookimpl(tryfirst=True, hookwrapper=True)
    def pytest_runtest_makereport(self, item, call):
        outcome = yield
        if item.get_marker(TESTRAIL_PREFIX):
            rep = outcome.get_result()
            case_ids = item.get_marker(TESTRAIL_PREFIX).kwargs.get('ids')
            if rep.when == 'call' and case_ids:
                self.add_result(
                    clean_case_ids(case_ids),
                    get_test_outcome(outcome.result.outcome),
                )
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

that part is where the error happens, but the trigger is somewhere else, likely something with a add_marker call - i ask for the complete code there because i'm very aware of the internal interactions and dont have to guess whats relevant

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

yeah. I understand. Thanks, I will go hunting. I appreciate the pointer.

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

I think I have the culprit...

@pytest.fixture(autouse=True)
def testrail(request):
    testrail = request.config.pluginmanager.get_plugin('testrail')
    # Check if the plugin is configured. It might not be configured if invoked
    # as `pytest --testrail-config=`.
    if testrail:
        case_id = getattr(request.function, 'case_id', None)
        if not case_id:
            raise Exception(request.function.__name__ + " must declare a testrail case_id")
        marker = pytest.mark.testrail(ids=[case_id])
        request.node.add_marker(marker)
    yield

That is in the top level conftest.py. It is the only add_marker call.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

thanks for the update, this seems to be a regression introduced by the legacy marker support

for now you have to pin the older pytest i beleive, i#ll try to come up with a minimal acceptance test later and come up with a propper solution

@shaleh

This comment has been minimized.

Copy link
Author

shaleh commented Jun 11, 2018

Excellent. Glad I could help! Thanks for your time.

Updating our code to use get_closest_marker instead of get_marker resolved our issue locally.

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jun 11, 2018

@RonnyPfannschmidt is this related to #3501? I ask because the error message is the same.

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 11, 2018

@nicoddemus unrelated - just triggers the same issue - the code in #3501 deliberately triggers it by putting a markdecorator into a place where only marks should be

the code here doesn't

@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jun 11, 2018

I see, thanks. I still need to get back to #3501, hopefully will have time this week after a major Python 3 conversion at work. 😁

@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 12, 2018

@shaleh if you find some time please verify #3577

guykisel added a commit to guykisel/inline-plz that referenced this issue Jul 6, 2018

Update pytest to 3.6.3 (#302)
This PR updates [pytest](https://pypi.org/project/pytest) from **3.5.1** to **3.6.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.6.2
   ```
   =========================

Bug Fixes
---------

- Fix regression in ``Node.add_marker`` by extracting the mark object of a
  ``MarkDecorator``. (`3555
  &lt;https://github.com/pytest-dev/pytest/issues/3555&gt;`_)

- Warnings without ``location`` were reported as ``None``. This is corrected to
  now report ``&lt;undetermined location&gt;``. (`3563
  &lt;https://github.com/pytest-dev/pytest/issues/3563&gt;`_)

- Continue to call finalizers in the stack when a finalizer in a former scope
  raises an exception. (`3569
  &lt;https://github.com/pytest-dev/pytest/issues/3569&gt;`_)

- Fix encoding error with `print` statements in doctests (`3583
  &lt;https://github.com/pytest-dev/pytest/issues/3583&gt;`_)


Improved Documentation
----------------------

- Add documentation for the ``--strict`` flag. (`3549
  &lt;https://github.com/pytest-dev/pytest/issues/3549&gt;`_)


Trivial/Internal Changes
------------------------

- Update old quotation style to parens in fixture.rst documentation. (`3525
  &lt;https://github.com/pytest-dev/pytest/issues/3525&gt;`_)

- Improve display of hint about ``--fulltrace`` with ``KeyboardInterrupt``.
  (`3545 &lt;https://github.com/pytest-dev/pytest/issues/3545&gt;`_)

- pytest&#39;s testsuite is no longer runnable through ``python setup.py test`` --
  instead invoke ``pytest`` or ``tox`` directly. (`3552
  &lt;https://github.com/pytest-dev/pytest/issues/3552&gt;`_)

- Fix typo in documentation (`3567
  &lt;https://github.com/pytest-dev/pytest/issues/3567&gt;`_)
   ```
   
  
  
   ### 3.6.1
   ```
   =========================

Bug Fixes
---------

- Fixed a bug where stdout and stderr were logged twice by junitxml when a test
  was marked xfail. (`3491
  &lt;https://github.com/pytest-dev/pytest/issues/3491&gt;`_)

- Fix ``usefixtures`` mark applyed to unittest tests by correctly instantiating
  ``FixtureInfo``. (`3498
  &lt;https://github.com/pytest-dev/pytest/issues/3498&gt;`_)

- Fix assertion rewriter compatibility with libraries that monkey patch
  ``file`` objects. (`3503
  &lt;https://github.com/pytest-dev/pytest/issues/3503&gt;`_)


Improved Documentation
----------------------

- Added a section on how to use fixtures as factories to the fixture
  documentation. (`3461 &lt;https://github.com/pytest-dev/pytest/issues/3461&gt;`_)


Trivial/Internal Changes
------------------------

- Enable caching for pip/pre-commit in order to reduce build time on
  travis/appveyor. (`3502
  &lt;https://github.com/pytest-dev/pytest/issues/3502&gt;`_)

- Switch pytest to the src/ layout as we already suggested it for good practice
  - now we implement it as well. (`3513
  &lt;https://github.com/pytest-dev/pytest/issues/3513&gt;`_)

- Fix if in tests to support 3.7.0b5, where a docstring handling in AST got
  reverted. (`3530 &lt;https://github.com/pytest-dev/pytest/issues/3530&gt;`_)

- Remove some python2.5 compatibility code. (`3529
  &lt;https://github.com/pytest-dev/pytest/issues/3529&gt;`_)
   ```
   
  
  
   ### 3.6.0
   ```
   =========================

Features
--------

- Revamp the internals of the ``pytest.mark`` implementation with correct per
  node handling which fixes a number of long standing bugs caused by the old
  design. This introduces new ``Node.iter_markers(name)`` and
  ``Node.get_closest_mark(name)`` APIs. Users are **strongly encouraged** to
  read the `reasons for the revamp in the docs
  &lt;https://docs.pytest.org/en/latest/mark.htmlmarker-revamp-and-iteration&gt;`_,
  or jump over to details about `updating existing code to use the new APIs
  &lt;https://docs.pytest.org/en/latest/mark.htmlupdating-code&gt;`_. (`3317
  &lt;https://github.com/pytest-dev/pytest/issues/3317&gt;`_)

- Now when ``pytest.fixture`` is applied more than once to the same function a
  ``ValueError`` is raised. This buggy behavior would cause surprising problems
  and if was working for a test suite it was mostly by accident. (`2334
  &lt;https://github.com/pytest-dev/pytest/issues/2334&gt;`_)

- Support for Python 3.7&#39;s builtin ``breakpoint()`` method, see `Using the
  builtin breakpoint function
  &lt;https://docs.pytest.org/en/latest/usage.htmlbreakpoint-builtin&gt;`_ for
  details. (`3180 &lt;https://github.com/pytest-dev/pytest/issues/3180&gt;`_)

- ``monkeypatch`` now supports a ``context()`` function which acts as a context
  manager which undoes all patching done within the ``with`` block. (`3290
  &lt;https://github.com/pytest-dev/pytest/issues/3290&gt;`_)

- The ``--pdb`` option now causes KeyboardInterrupt to enter the debugger,
  instead of stopping the test session. On python 2.7, hitting CTRL+C again
  exits the debugger. On python 3.2 and higher, use CTRL+D. (`3299
  &lt;https://github.com/pytest-dev/pytest/issues/3299&gt;`_)

- pytest not longer changes the log level of the root logger when the
  ``log-level`` parameter has greater numeric value than that of the level of
  the root logger, which makes it play better with custom logging configuration
  in user code. (`3307 &lt;https://github.com/pytest-dev/pytest/issues/3307&gt;`_)


Bug Fixes
---------

- A rare race-condition which might result in corrupted ``.pyc`` files on
  Windows has been hopefully solved. (`3008
  &lt;https://github.com/pytest-dev/pytest/issues/3008&gt;`_)

- Also use iter_marker for discovering the marks applying for marker
  expressions from the cli to avoid the bad data from the legacy mark storage.
  (`3441 &lt;https://github.com/pytest-dev/pytest/issues/3441&gt;`_)

- When showing diffs of failed assertions where the contents contain only
  whitespace, escape them using ``repr()`` first to make it easy to spot the
  differences. (`3443 &lt;https://github.com/pytest-dev/pytest/issues/3443&gt;`_)


Improved Documentation
----------------------

- Change documentation copyright year to a range which auto-updates itself each
  time it is published. (`3303
  &lt;https://github.com/pytest-dev/pytest/issues/3303&gt;`_)


Trivial/Internal Changes
------------------------

- ``pytest`` now depends on the `python-atomicwrites
  &lt;https://github.com/untitaker/python-atomicwrites&gt;`_ library. (`3008
  &lt;https://github.com/pytest-dev/pytest/issues/3008&gt;`_)

- Update all pypi.python.org URLs to pypi.org. (`3431
  &lt;https://github.com/pytest-dev/pytest/issues/3431&gt;`_)

- Detect `pytest_` prefixed hooks using the internal plugin manager since
  ``pluggy`` is deprecating the ``implprefix`` argument to ``PluginManager``.
  (`3487 &lt;https://github.com/pytest-dev/pytest/issues/3487&gt;`_)

- Import ``Mapping`` and ``Sequence`` from ``_pytest.compat`` instead of
  directly from ``collections`` in ``python_api.py::approx``. Add ``Mapping``
  to ``_pytest.compat``, import it from ``collections`` on python 2, but from
  ``collections.abc`` on Python 3 to avoid a ``DeprecationWarning`` on Python
  3.7 or newer. (`3497 &lt;https://github.com/pytest-dev/pytest/issues/3497&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Repo: https://github.com/pytest-dev/pytest/issues
  - Homepage: http://pytest.org
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.