RecursionError in pluggy #1794

Closed
The-Compiler opened this Issue Aug 5, 2016 · 21 comments

Comments

Projects
None yet
4 participants
@The-Compiler
Member

The-Compiler commented Aug 5, 2016

When I try to run the qutebrowser testsuite with the current git features branch, I get:

_____ ERROR at teardown of test_timeout_set_interval _____

hook = <_HookCaller 'pytest_fixture_post_finalizer'>
methods = [<_pytest.vendored_packages.pluggy.HookImpl object at 0x7f3dd6aa7b38>]
kwargs = {'fixturedef': <FixtureDef name='httpbin' scope='session' baseid='tests/end2end' >}

    self._inner_hookexec = lambda hook, methods, kwargs: \
>       _MultiCall(methods, kwargs, hook.spec_opts).execute()
E   RecursionError: maximum recursion depth exceeded

.../_pytest/vendored_packages/pluggy.py:333: RecursionError

test_timeout_set_interval seems unrelated, as this is just the teardown of a session-scoped fixture. However, when I only run pytest on tests/end2end I can't reproduce this...

The httpbin fixture is defined here.

I have no idea what's happening though... Taking the freedom to mark this as critical as it might be a serious 3.0 regression for other testsuites too.

@The-Compiler The-Compiler added this to the 3.0 milestone Aug 5, 2016

@The-Compiler The-Compiler changed the title from RecursionError in pluggy [regression] to RecursionError in pluggy Aug 5, 2016

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 9, 2016

Member

@The-Compiler, thanks, I plan to start investigating this tomorrow. It seems related to the new pytest_fixture_post_finalizer hook.

Meanwhile, if you have the time, could you try to bisect this from 2.9.2 and try to find the culprit commit?

Member

nicoddemus commented Aug 9, 2016

@The-Compiler, thanks, I plan to start investigating this tomorrow. It seems related to the new pytest_fixture_post_finalizer hook.

Meanwhile, if you have the time, could you try to bisect this from 2.9.2 and try to find the culprit commit?

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 10, 2016

Member

Trying to bisect, but I ran into some trouble with this not being reproducible anymore first (not sure why). I think things are looking better now, but it'll still take me a bit.

Member

The-Compiler commented Aug 10, 2016

Trying to bisect, but I ran into some trouble with this not being reproducible anymore first (not sure why). I think things are looking better now, but it'll still take me a bit.

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 10, 2016

Member

Bisected to 7751008 ("Implement invocation-scoped fixtures"). @nicoddemus, your turn 😉

Member

The-Compiler commented Aug 10, 2016

Bisected to 7751008 ("Implement invocation-scoped fixtures"). @nicoddemus, your turn 😉

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 10, 2016

Member

Thanks for the bisect! 😁

I will try to tackle this tonight.

Is there a simple non-obvious command to reproduce the error in qutebrowser, for example py.test tests/some_test_file.py --someoption, or does it happen only when running the full test suite?

Member

nicoddemus commented Aug 10, 2016

Thanks for the bisect! 😁

I will try to tackle this tonight.

Is there a simple non-obvious command to reproduce the error in qutebrowser, for example py.test tests/some_test_file.py --someoption, or does it happen only when running the full test suite?

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 10, 2016

Member

Here's roughly what I did:

  • create/activate a virtualenv
  • pip install -rmisc/requirements/requirements-tests.txt
  • pip install -rrequirements.txt
  • pip install git+https://github.com/pytest-dev/pytest.git@features
  • /usr/bin/python3 scripts/link_pyqt.py .venv
  • edit conftest.py to always skip BDD tests in pytest_ignore_collect (simply set skip_bdd = True)
  • py.test

I initially tried running a subset but then the error went away, so I guess better save than sorry. Still takes around 1min 40s per run though...

Member

The-Compiler commented Aug 10, 2016

Here's roughly what I did:

  • create/activate a virtualenv
  • pip install -rmisc/requirements/requirements-tests.txt
  • pip install -rrequirements.txt
  • pip install git+https://github.com/pytest-dev/pytest.git@features
  • /usr/bin/python3 scripts/link_pyqt.py .venv
  • edit conftest.py to always skip BDD tests in pytest_ignore_collect (simply set skip_bdd = True)
  • py.test

I initially tried running a subset but then the error went away, so I guess better save than sorry. Still takes around 1min 40s per run though...

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 11, 2016

Member

On this for some hours now, made some progress but still no idea what's the problem. 😭 I'll just dump some information here in the hope that sparks some heureka moment in someone.

I could not reproduce the problem on my Windows computer because of PyQt5.QtWebKit not being easily available and not being in the mood to compile it myself, so I had to investigate this on a spare Unix box which didn't have my usual dev environment on it, so progress has been slow. 😔

Anyways, I've tried to reduce the number of tests one had to run to find the issue, and I managed to remove quite a few test files until I had a small enough set of tests that still manifested the problem, reducing the test suite time to 8 seconds. I pushed a branch to my fork in order to continue this tomorrow or if others want to take a look.

Some facts:

  • When commenting out the pytest_fixture_post_finalizer hook call in fixtures.py, the traceback changes and shows where the actual recursion is happening:

    self = <FixtureDef name='qapp' scope='session' baseid='tests' >                                                                                                                                                                           
    
    def finish(self):                                                                                                                                                                                                                     
        try:                                                                                                                                                                                                                              
            while self._finalizer:                                                                                                                                                                                                        
                func = self._finalizer.pop()                                                                                                                                                                                              
    >               func()                                                                                                                                                                                                                    
    E               RuntimeError: maximum recursion depth exceeded                                                                                                                                                                            
    
    ../pytest/_pytest/fixtures.py:751: RuntimeError            
    

    So it seems some finalizer is registering itself, somehow.

  • It's definitely related to either or both httpbin and httpbin_after_test fixtures. On my branch if I remove their autouse status or don't execute tests/end2end/test_dummy.py (which exists solely that tests/end2end/conftest.py is considered) the error no longer happens.

  • httpbin and other fixtures were being imported directly into tests/end2end/conftest.py:

    from end2end.fixtures.webserver import httpbin, httpbin_after_test, ssl_server

    This might be a problem because depending on where they are used they might be initialized/finalized more than once because pytest will think they are defined in different places and are different fixtures. The correct approach would be to use pytest_plugins variable to declare those, but that doesn't seem to be the issue here: I did try to rename the fixture function to httpbin_ and mark it with @pytest.fixture(name='httpbin', ... to see if other places imported it, but doesn't seem to be the case.

  • I couldn't really manage to see which actual function was the func variable in the error traceback; I focused more on trying to narrow the suite, but I guess that is a good starting point for tomorrow.

Well that's what I can remember now with my tired brain at this late hour. I plan to continue debugging this tomorrow, but any comments or insights are welcome. 😅

Member

nicoddemus commented Aug 11, 2016

On this for some hours now, made some progress but still no idea what's the problem. 😭 I'll just dump some information here in the hope that sparks some heureka moment in someone.

I could not reproduce the problem on my Windows computer because of PyQt5.QtWebKit not being easily available and not being in the mood to compile it myself, so I had to investigate this on a spare Unix box which didn't have my usual dev environment on it, so progress has been slow. 😔

Anyways, I've tried to reduce the number of tests one had to run to find the issue, and I managed to remove quite a few test files until I had a small enough set of tests that still manifested the problem, reducing the test suite time to 8 seconds. I pushed a branch to my fork in order to continue this tomorrow or if others want to take a look.

Some facts:

  • When commenting out the pytest_fixture_post_finalizer hook call in fixtures.py, the traceback changes and shows where the actual recursion is happening:

    self = <FixtureDef name='qapp' scope='session' baseid='tests' >                                                                                                                                                                           
    
    def finish(self):                                                                                                                                                                                                                     
        try:                                                                                                                                                                                                                              
            while self._finalizer:                                                                                                                                                                                                        
                func = self._finalizer.pop()                                                                                                                                                                                              
    >               func()                                                                                                                                                                                                                    
    E               RuntimeError: maximum recursion depth exceeded                                                                                                                                                                            
    
    ../pytest/_pytest/fixtures.py:751: RuntimeError            
    

    So it seems some finalizer is registering itself, somehow.

  • It's definitely related to either or both httpbin and httpbin_after_test fixtures. On my branch if I remove their autouse status or don't execute tests/end2end/test_dummy.py (which exists solely that tests/end2end/conftest.py is considered) the error no longer happens.

  • httpbin and other fixtures were being imported directly into tests/end2end/conftest.py:

    from end2end.fixtures.webserver import httpbin, httpbin_after_test, ssl_server

    This might be a problem because depending on where they are used they might be initialized/finalized more than once because pytest will think they are defined in different places and are different fixtures. The correct approach would be to use pytest_plugins variable to declare those, but that doesn't seem to be the issue here: I did try to rename the fixture function to httpbin_ and mark it with @pytest.fixture(name='httpbin', ... to see if other places imported it, but doesn't seem to be the case.

  • I couldn't really manage to see which actual function was the func variable in the error traceback; I focused more on trying to narrow the suite, but I guess that is a good starting point for tomorrow.

Well that's what I can remember now with my tired brain at this late hour. I plan to continue debugging this tomorrow, but any comments or insights are welcome. 😅

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 11, 2016

Member

I can confirm that subset reproduces it for me. I could also get rid of test_split_hypothesis.py and test_log.py.

When I comment out ihook.pytest_fixture_post_finalizer(fixturedef=self) then, I don't get a RecursionError anymore. After re-adding the test files I removed above, it's there again, but at another location:

Traceback (most recent call last):
  File "/home/florian/proj/pytest/_pytest/main.py", line 570, in gethookproxy
    return self._fs2hookproxy[fspath]
RecursionError: maximum recursion depth exceeded

So it sounds like it's - pure speculation - something with the stack getting bigger with more test items, and then at some point too big?

By the way: import pdb; pdb.set_trace() will segfault for some reason I didn't investigate yet, but pip install pdbpp helps.

I also tried uninstalling pytest-repeat and pytest-rerunfailures (and removing --strict), and the issue still happens.


When I catch that RecursionError and show the stack via pdb/pdbpp, I get:

(Pdb++) bt
[0]   /home/florian/proj/pytest/.venv/bin/pytest(9)<module>()
-> load_entry_point('pytest', 'console_scripts', 'pytest')()
[1]   /home/florian/proj/pytest/_pytest/config.py(57)main()
-> return config.hook.pytest_cmdline_main(config=config)
[2]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[3]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[4]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[5]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[6]   /home/florian/proj/pytest/_pytest/main.py(125)pytest_cmdline_main()
-> return wrap_session(config, _main)
[7]   /home/florian/proj/pytest/_pytest/main.py(96)wrap_session()
-> session.exitstatus = doit(config, session) or 0
[8]   /home/florian/proj/pytest/_pytest/main.py(131)_main()
-> config.hook.pytest_runtestloop(session=session)
[9]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[10]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[11]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[12]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[13]   /home/florian/proj/pytest/_pytest/main.py(152)pytest_runtestloop()
-> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
[14]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[15]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[16]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[17]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[18]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[19]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[20]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[21]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[22]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[23]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[24]   /home/florian/proj/pytest/.venv/lib/python3.5/site-packages/pytest_rerunfailures.py(73)pytest_runtest_protocol()
-> reports = runtestprotocol(item, nextitem=nextitem, log=False)
[25]   /home/florian/proj/pytest/_pytest/runner.py(81)runtestprotocol()
-> nextitem=nextitem))
[26]   /home/florian/proj/pytest/_pytest/runner.py(133)call_and_report()
-> call = call_runtest_hook(item, when, **kwds)
[27]   /home/florian/proj/pytest/_pytest/runner.py(151)call_runtest_hook()
-> return CallInfo(lambda: ihook(item=item, **kwds), when=when)
[28]   /home/florian/proj/pytest/_pytest/runner.py(163)__init__()
-> self.result = func()
[29]   /home/florian/proj/pytest/_pytest/runner.py(151)<lambda>()
-> return CallInfo(lambda: ihook(item=item, **kwds), when=when)
[30]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[31]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[32]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[33]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[34]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[35]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[36]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[37]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[38]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[39]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[40]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[41]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[42]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[43]   /home/florian/proj/pytest/_pytest/runner.py(116)pytest_runtest_teardown()
-> item.session._setupstate.teardown_exact(item, nextitem)
[44]   /home/florian/proj/pytest/_pytest/runner.py(400)teardown_exact()
-> self._teardown_towards(needed_collectors)
[45]   /home/florian/proj/pytest/_pytest/runner.py(406)_teardown_towards()
-> self._pop_and_teardown()
[46]   /home/florian/proj/pytest/_pytest/runner.py(366)_pop_and_teardown()
-> self._teardown_with_finalization(colitem)
[47]   /home/florian/proj/pytest/_pytest/runner.py(384)_teardown_with_finalization()
-> self._callfinalizers(colitem)
[48]   /home/florian/proj/pytest/_pytest/runner.py(374)_callfinalizers()
-> fin()
[49]   /home/florian/proj/pytest/_pytest/fixtures.py(751)finish()
-> func()

[...]

[958]   /home/florian/proj/pytest/_pytest/fixtures.py(751)finish()
-> func()
[959]   /home/florian/proj/pytest/_pytest/fixtures.py(753)finish()
-> ihook = self._fixturemanager.session.ihook
[960]   /home/florian/proj/pytest/_pytest/main.py(266)ihook()
-> return self.session.gethookproxy(self.fspath)
[961] > /home/florian/proj/pytest/_pytest/main.py(572)gethookproxy()->None
-> import pdb; pdb.set_trace()

I then tried finding out what func is - turns out inside a RecursionError you have no way to debug because your stack is full 😆 - however, this helped:

diff --git a/_pytest/fixtures.py b/_pytest/fixtures.py
index 383e41b..6d3a8d9 100644
--- a/_pytest/fixtures.py
+++ b/_pytest/fixtures.py
@@ -748,7 +748,11 @@ class FixtureDef:
         try:
             while self._finalizer:
                 func = self._finalizer.pop()
-                func()
+                try:
+                    func()
+                except RecursionError:
+                    sys.setrecursionlimit(sys.getrecursionlimit() + 100)
+                    import pdb; pdb.set_trace()
         finally:
             ihook = self._fixturemanager.session.ihook
             ihook.pytest_fixture_post_finalizer(fixturedef=self)

Turns out it's indeed the finalizer of httpbin calling itself somehow:

[954] > /home/florian/proj/pytest/_pytest/fixtures.py(749)finish()
-> while self._finalizer:
(Pdb++) func
<bound method FixtureDef.finish of <FixtureDef name='httpbin' scope='session' baseid='' >>

So I placed a breakpoint in the fixture after yield:

[53] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(176)httpbin()
-> httpbin.cleanup()
(Pdb++) s
--Call--
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(163)cleanup()
-> def cleanup(self):
(Pdb++) s
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(165)cleanup()
-> self.proc.terminate()
(Pdb++) s
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(166)cleanup()
-> self.proc.waitForFinished()
(Pdb++) s
--Return--
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(166)cleanup()->None
-> self.proc.waitForFinished()
(Pdb++) s
--Return--
[53] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(176)httpbin()->None
-> httpbin.cleanup()
(Pdb++) s
StopIteration
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(714)teardown()
-> next(it)
(Pdb++) s
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(715)teardown()
-> except StopIteration:
(Pdb++) s
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(716)teardown()
-> pass
(Pdb++) s
--Return--
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(716)teardown()->None
-> pass

Okay, the yield_fixture is done now, let's see what happens next:

(Pdb++) s
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(749)finish()
-> while self._finalizer:
(Pdb++) s
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(753)finish()
-> ihook = self._fixturemanager.session.ihook
(Pdb++) self._finalizer
[]

No finalizers left. Let's continue stepping:

(Pdb++) s
--Call--
[52] > /home/florian/proj/pytest/_pytest/main.py(263)ihook()
-> @property
(Pdb++) s
[52] > /home/florian/proj/pytest/_pytest/main.py(266)ihook()
-> return self.session.gethookproxy(self.fspath)
(Pdb++) s
--Call--
[53] > /home/florian/proj/pytest/_pytest/main.py(568)gethookproxy()
-> def gethookproxy(self, fspath):
(Pdb++) s

Inside some hook stuff I don't understand, so I started using n to step over
stuff:

[53] > /home/florian/proj/pytest/_pytest/main.py(569)gethookproxy()
-> try:
(Pdb++) n
[53] > /home/florian/proj/pytest/_pytest/main.py(570)gethookproxy()
-> return self._fs2hookproxy[fspath]
(Pdb++) n
--Return--
[53] > /home/florian/proj/pytest/_pytest/main.py(570)gethookproxy()-><_pytest.main...x7fa990126f98>
-> return self._fs2hookproxy[fspath]
(Pdb++) n
--Return--
[52] > /home/florian/proj/pytest/_pytest/main.py(266)ihook()-><_pytest.main...x7fa990126f98>
-> return self.session.gethookproxy(self.fspath)
(Pdb++) n
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(754)finish()
-> ihook.pytest_fixture_post_finalizer(fixturedef=self)
(Pdb++) n
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(757)finish()
-> if hasattr(self, "cached_result"):
(Pdb++) n
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(758)finish()
-> del self.cached_result
(Pdb++) n
--Return--
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(758)finish()->None
-> del self.cached_result
(Pdb++) n
[50] > /home/florian/proj/pytest/_pytest/fixtures.py(749)finish()
-> while self._finalizer:

Oh, we ended up here again. Now how does self._finalizer look like?

(Pdb++) n
[50] > /home/florian/proj/pytest/_pytest/fixtures.py(750)finish()
-> func = self._finalizer.pop()
(Pdb++) self._finalizer
[<bound method FixtureDef.finish of <FixtureDef name='httpbin' scope='session' baseid='' >>, <bound method FixtureDef.finish of <FixtureDef name='qapp' scope='session' baseid='tests' >>, <bound method FixtureDef.finish of <FixtureDef name='httpbin' scope='session' baseid='' >>, <bound method FixtureDef.finish of <FixtureDef name='qapp' scope='session' baseid='tests' >>, ...]
(Pdb++) len(self._finalizer)
1942

😱


I have no idea how this happens, but I have a qapp fixture in my conftest.py to override the one from pytest-qt:

@pytest.fixture(scope='session')
def qapp(qapp):
    """Change the name of the QApplication instance."""
    qapp.setApplicationName('qute_test')
    return qapp

If I don't override qapp, or don't request it from httpbin and do this in my conftest.py instead:

from PyQt5.QtWidgets import QApplication
early_qapp = QApplication([])

things work fine again.

Member

The-Compiler commented Aug 11, 2016

I can confirm that subset reproduces it for me. I could also get rid of test_split_hypothesis.py and test_log.py.

When I comment out ihook.pytest_fixture_post_finalizer(fixturedef=self) then, I don't get a RecursionError anymore. After re-adding the test files I removed above, it's there again, but at another location:

Traceback (most recent call last):
  File "/home/florian/proj/pytest/_pytest/main.py", line 570, in gethookproxy
    return self._fs2hookproxy[fspath]
RecursionError: maximum recursion depth exceeded

So it sounds like it's - pure speculation - something with the stack getting bigger with more test items, and then at some point too big?

By the way: import pdb; pdb.set_trace() will segfault for some reason I didn't investigate yet, but pip install pdbpp helps.

I also tried uninstalling pytest-repeat and pytest-rerunfailures (and removing --strict), and the issue still happens.


When I catch that RecursionError and show the stack via pdb/pdbpp, I get:

(Pdb++) bt
[0]   /home/florian/proj/pytest/.venv/bin/pytest(9)<module>()
-> load_entry_point('pytest', 'console_scripts', 'pytest')()
[1]   /home/florian/proj/pytest/_pytest/config.py(57)main()
-> return config.hook.pytest_cmdline_main(config=config)
[2]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[3]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[4]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[5]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[6]   /home/florian/proj/pytest/_pytest/main.py(125)pytest_cmdline_main()
-> return wrap_session(config, _main)
[7]   /home/florian/proj/pytest/_pytest/main.py(96)wrap_session()
-> session.exitstatus = doit(config, session) or 0
[8]   /home/florian/proj/pytest/_pytest/main.py(131)_main()
-> config.hook.pytest_runtestloop(session=session)
[9]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[10]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[11]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[12]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[13]   /home/florian/proj/pytest/_pytest/main.py(152)pytest_runtestloop()
-> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
[14]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[15]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[16]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[17]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[18]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[19]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[20]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[21]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[22]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[23]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[24]   /home/florian/proj/pytest/.venv/lib/python3.5/site-packages/pytest_rerunfailures.py(73)pytest_runtest_protocol()
-> reports = runtestprotocol(item, nextitem=nextitem, log=False)
[25]   /home/florian/proj/pytest/_pytest/runner.py(81)runtestprotocol()
-> nextitem=nextitem))
[26]   /home/florian/proj/pytest/_pytest/runner.py(133)call_and_report()
-> call = call_runtest_hook(item, when, **kwds)
[27]   /home/florian/proj/pytest/_pytest/runner.py(151)call_runtest_hook()
-> return CallInfo(lambda: ihook(item=item, **kwds), when=when)
[28]   /home/florian/proj/pytest/_pytest/runner.py(163)__init__()
-> self.result = func()
[29]   /home/florian/proj/pytest/_pytest/runner.py(151)<lambda>()
-> return CallInfo(lambda: ihook(item=item, **kwds), when=when)
[30]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(724)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[31]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(338)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[32]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(333)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[33]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[34]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[35]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[36]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[37]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[38]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[39]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(595)execute()
-> return _wrapped_call(hook_impl.function(*args), self.execute)
[40]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(247)_wrapped_call()
-> call_outcome = _CallOutcome(func)
[41]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(264)__init__()
-> self.result = func()
[42]   /home/florian/proj/pytest/_pytest/vendored_packages/pluggy.py(596)execute()
-> res = hook_impl.function(*args)
[43]   /home/florian/proj/pytest/_pytest/runner.py(116)pytest_runtest_teardown()
-> item.session._setupstate.teardown_exact(item, nextitem)
[44]   /home/florian/proj/pytest/_pytest/runner.py(400)teardown_exact()
-> self._teardown_towards(needed_collectors)
[45]   /home/florian/proj/pytest/_pytest/runner.py(406)_teardown_towards()
-> self._pop_and_teardown()
[46]   /home/florian/proj/pytest/_pytest/runner.py(366)_pop_and_teardown()
-> self._teardown_with_finalization(colitem)
[47]   /home/florian/proj/pytest/_pytest/runner.py(384)_teardown_with_finalization()
-> self._callfinalizers(colitem)
[48]   /home/florian/proj/pytest/_pytest/runner.py(374)_callfinalizers()
-> fin()
[49]   /home/florian/proj/pytest/_pytest/fixtures.py(751)finish()
-> func()

[...]

[958]   /home/florian/proj/pytest/_pytest/fixtures.py(751)finish()
-> func()
[959]   /home/florian/proj/pytest/_pytest/fixtures.py(753)finish()
-> ihook = self._fixturemanager.session.ihook
[960]   /home/florian/proj/pytest/_pytest/main.py(266)ihook()
-> return self.session.gethookproxy(self.fspath)
[961] > /home/florian/proj/pytest/_pytest/main.py(572)gethookproxy()->None
-> import pdb; pdb.set_trace()

I then tried finding out what func is - turns out inside a RecursionError you have no way to debug because your stack is full 😆 - however, this helped:

diff --git a/_pytest/fixtures.py b/_pytest/fixtures.py
index 383e41b..6d3a8d9 100644
--- a/_pytest/fixtures.py
+++ b/_pytest/fixtures.py
@@ -748,7 +748,11 @@ class FixtureDef:
         try:
             while self._finalizer:
                 func = self._finalizer.pop()
-                func()
+                try:
+                    func()
+                except RecursionError:
+                    sys.setrecursionlimit(sys.getrecursionlimit() + 100)
+                    import pdb; pdb.set_trace()
         finally:
             ihook = self._fixturemanager.session.ihook
             ihook.pytest_fixture_post_finalizer(fixturedef=self)

Turns out it's indeed the finalizer of httpbin calling itself somehow:

[954] > /home/florian/proj/pytest/_pytest/fixtures.py(749)finish()
-> while self._finalizer:
(Pdb++) func
<bound method FixtureDef.finish of <FixtureDef name='httpbin' scope='session' baseid='' >>

So I placed a breakpoint in the fixture after yield:

[53] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(176)httpbin()
-> httpbin.cleanup()
(Pdb++) s
--Call--
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(163)cleanup()
-> def cleanup(self):
(Pdb++) s
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(165)cleanup()
-> self.proc.terminate()
(Pdb++) s
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(166)cleanup()
-> self.proc.waitForFinished()
(Pdb++) s
--Return--
[54] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(166)cleanup()->None
-> self.proc.waitForFinished()
(Pdb++) s
--Return--
[53] > /home/florian/proj/pytest/qutebrowser/tests/end2end/fixtures/webserver.py(176)httpbin()->None
-> httpbin.cleanup()
(Pdb++) s
StopIteration
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(714)teardown()
-> next(it)
(Pdb++) s
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(715)teardown()
-> except StopIteration:
(Pdb++) s
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(716)teardown()
-> pass
(Pdb++) s
--Return--
[52] > /home/florian/proj/pytest/_pytest/fixtures.py(716)teardown()->None
-> pass

Okay, the yield_fixture is done now, let's see what happens next:

(Pdb++) s
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(749)finish()
-> while self._finalizer:
(Pdb++) s
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(753)finish()
-> ihook = self._fixturemanager.session.ihook
(Pdb++) self._finalizer
[]

No finalizers left. Let's continue stepping:

(Pdb++) s
--Call--
[52] > /home/florian/proj/pytest/_pytest/main.py(263)ihook()
-> @property
(Pdb++) s
[52] > /home/florian/proj/pytest/_pytest/main.py(266)ihook()
-> return self.session.gethookproxy(self.fspath)
(Pdb++) s
--Call--
[53] > /home/florian/proj/pytest/_pytest/main.py(568)gethookproxy()
-> def gethookproxy(self, fspath):
(Pdb++) s

Inside some hook stuff I don't understand, so I started using n to step over
stuff:

[53] > /home/florian/proj/pytest/_pytest/main.py(569)gethookproxy()
-> try:
(Pdb++) n
[53] > /home/florian/proj/pytest/_pytest/main.py(570)gethookproxy()
-> return self._fs2hookproxy[fspath]
(Pdb++) n
--Return--
[53] > /home/florian/proj/pytest/_pytest/main.py(570)gethookproxy()-><_pytest.main...x7fa990126f98>
-> return self._fs2hookproxy[fspath]
(Pdb++) n
--Return--
[52] > /home/florian/proj/pytest/_pytest/main.py(266)ihook()-><_pytest.main...x7fa990126f98>
-> return self.session.gethookproxy(self.fspath)
(Pdb++) n
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(754)finish()
-> ihook.pytest_fixture_post_finalizer(fixturedef=self)
(Pdb++) n
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(757)finish()
-> if hasattr(self, "cached_result"):
(Pdb++) n
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(758)finish()
-> del self.cached_result
(Pdb++) n
--Return--
[51] > /home/florian/proj/pytest/_pytest/fixtures.py(758)finish()->None
-> del self.cached_result
(Pdb++) n
[50] > /home/florian/proj/pytest/_pytest/fixtures.py(749)finish()
-> while self._finalizer:

Oh, we ended up here again. Now how does self._finalizer look like?

(Pdb++) n
[50] > /home/florian/proj/pytest/_pytest/fixtures.py(750)finish()
-> func = self._finalizer.pop()
(Pdb++) self._finalizer
[<bound method FixtureDef.finish of <FixtureDef name='httpbin' scope='session' baseid='' >>, <bound method FixtureDef.finish of <FixtureDef name='qapp' scope='session' baseid='tests' >>, <bound method FixtureDef.finish of <FixtureDef name='httpbin' scope='session' baseid='' >>, <bound method FixtureDef.finish of <FixtureDef name='qapp' scope='session' baseid='tests' >>, ...]
(Pdb++) len(self._finalizer)
1942

😱


I have no idea how this happens, but I have a qapp fixture in my conftest.py to override the one from pytest-qt:

@pytest.fixture(scope='session')
def qapp(qapp):
    """Change the name of the QApplication instance."""
    qapp.setApplicationName('qute_test')
    return qapp

If I don't override qapp, or don't request it from httpbin and do this in my conftest.py instead:

from PyQt5.QtWidgets import QApplication
early_qapp = QApplication([])

things work fine again.

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 11, 2016

Member

FYI: I edited the post above multiple times to provide more and more information - so don't view it via email 😉

Member

The-Compiler commented Aug 11, 2016

FYI: I edited the post above multiple times to provide more and more information - so don't view it via email 😉

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 11, 2016

Member

@The-Compiler thanks a lot for the debugging help! I will continue this tonight. 😅

Me this morning: "Hmm if only I knew of a project which used PyQt5 and QtWebKit on Windows, so I can see which package they use to install it hassle-free..."... duh. I face-palmed so hard I almost knocked myself out. 😆

Member

nicoddemus commented Aug 11, 2016

@The-Compiler thanks a lot for the debugging help! I will continue this tonight. 😅

Me this morning: "Hmm if only I knew of a project which used PyQt5 and QtWebKit on Windows, so I can see which package they use to install it hassle-free..."... duh. I face-palmed so hard I almost knocked myself out. 😆

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 11, 2016

Member

Installing Python 3.4 and PyQt 5.5.1 (e.g. from here, or probably still available on sourceforge) might be the easiest way.

Member

The-Compiler commented Aug 11, 2016

Installing Python 3.4 and PyQt 5.5.1 (e.g. from here, or probably still available on sourceforge) might be the easiest way.

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 11, 2016

Member

Yep, the face-palm moment was due to the fact that I could just have seen how qutebrowser did it. 😁

Member

nicoddemus commented Aug 11, 2016

Yep, the face-palm moment was due to the fact that I could just have seen how qutebrowser did it. 😁

@The-Compiler

This comment has been minimized.

Show comment
Hide comment
@The-Compiler

The-Compiler Aug 11, 2016

Member

Right - seems like we both need more ☕️ 😉

Member

The-Compiler commented Aug 11, 2016

Right - seems like we both need more ☕️ 😉

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 12, 2016

Member

Quick update: I've made some progress and I understood what the problem is, but I'm still in the process of reproducing this in an isolated test. 😬

Member

nicoddemus commented Aug 12, 2016

Quick update: I've made some progress and I understood what the problem is, but I'm still in the process of reproducing this in an isolated test. 😬

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Aug 16, 2016

Revert all invocation-fixtures code
Due to a serious regression found in #1794, it was decided to pull off
invocation features from 3.0 so it can be (hopefully) re-introduced
in 3.1

@nicoddemus nicoddemus modified the milestones: 3.1, 3.0 Aug 16, 2016

nicoddemus added a commit to nicoddemus/blog.pytest.org that referenced this issue Aug 16, 2016

Remove invocation scoped fixtures section
As discussed in pytest-dev/pytest#1794, it was decided to
postpone this until 3.1

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Aug 17, 2016

Revert all invocation-fixtures code
Due to a serious regression found in #1794, it was decided to pull off
invocation features from 3.0 so it can be (hopefully) re-introduced
in 3.1

@nicoddemus nicoddemus changed the title from RecursionError in pluggy to Bring invocation scoped fixtures back (was: RecursionError in pluggy) Aug 26, 2016

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 26, 2016

Member

(@The-Compiler I changed the title of this issue, I hope that's OK)

Member

nicoddemus commented Aug 26, 2016

(@The-Compiler I changed the title of this issue, I hope that's OK)

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 26, 2016

Member

When this is tackled, make sure to revert the fix for #1872

Member

nicoddemus commented Aug 26, 2016

When this is tackled, make sure to revert the fix for #1872

@blueyed

This comment has been minimized.

Show comment
Hide comment
@blueyed

blueyed Oct 11, 2016

Contributor

What's the state of this?
Any progress on the front of this nice feature?
Is there a PR / branch where it's being worked on?

Contributor

blueyed commented Oct 11, 2016

What's the state of this?
Any progress on the front of this nice feature?
Is there a PR / branch where it's being worked on?

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Oct 11, 2016

Member

Any progress on the front of this nice feature?

No progress, unfortunately.

There were two approaches that were discussed/implemented:

  1. Change the fixture caching mechanism to also cache based on scope; this was tried during the sprint but hit some roadblocks.
  2. Group FixtureDef on the Request objects not only by name, but also by scope. That was the implementation I had for 3.0 but broke down when faced with fixtures being overwritten in the same name.

I gave it another try right after 3.0 was out but I'm not sure it is possible with the current architecture. We would probably have to rethink how things are implemented in order to support this in all the possible use cases.

In summary, I don't think this will land in 3.1 unless someone else gives it a shot. 😞

Member

nicoddemus commented Oct 11, 2016

Any progress on the front of this nice feature?

No progress, unfortunately.

There were two approaches that were discussed/implemented:

  1. Change the fixture caching mechanism to also cache based on scope; this was tried during the sprint but hit some roadblocks.
  2. Group FixtureDef on the Request objects not only by name, but also by scope. That was the implementation I had for 3.0 but broke down when faced with fixtures being overwritten in the same name.

I gave it another try right after 3.0 was out but I'm not sure it is possible with the current architecture. We would probably have to rethink how things are implemented in order to support this in all the possible use cases.

In summary, I don't think this will land in 3.1 unless someone else gives it a shot. 😞

@blueyed

This comment has been minimized.

Show comment
Hide comment
@blueyed

blueyed Oct 12, 2016

Contributor

Too bad, but don't feel 😞! :)

Is there some WIP branch somewhere?

I've came back to this because I've started for at least the 2nd time to have some foo_session fixtures etc - basically what this would automate.

Contributor

blueyed commented Oct 12, 2016

Too bad, but don't feel 😞! :)

Is there some WIP branch somewhere?

I've came back to this because I've started for at least the 2nd time to have some foo_session fixtures etc - basically what this would automate.

@RonnyPfannschmidt

This comment has been minimized.

Show comment
Hide comment
@RonnyPfannschmidt

RonnyPfannschmidt Oct 12, 2016

Member

@blueyed main reason there is no wip is, that the current internals of fixtures make multi scoped fixtures on top of them a mess that can easily break apart

my current suspicion is that we need to rewrite the internals from ground up to enable it :(

Member

RonnyPfannschmidt commented Oct 12, 2016

@blueyed main reason there is no wip is, that the current internals of fixtures make multi scoped fixtures on top of them a mess that can easily break apart

my current suspicion is that we need to rewrite the internals from ground up to enable it :(

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Oct 12, 2016

Member

@blueyed for the record I pulled it back in a single commit: 707b6b5 (well and e92d373 but that was just docs), it should be possible to revert it and continue from there.

Member

nicoddemus commented Oct 12, 2016

@blueyed for the record I pulled it back in a single commit: 707b6b5 (well and e92d373 but that was just docs), it should be possible to revert it and continue from there.

@nicoddemus nicoddemus changed the title from Bring invocation scoped fixtures back (was: RecursionError in pluggy) to Invocation scoped fixtures Jul 6, 2017

@nicoddemus nicoddemus changed the title from Invocation scoped fixtures to RecursionError in pluggy Jul 6, 2017

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Jul 6, 2017

Member

We have reverted invocation scoped fixtures and I've reopened the original issue in #1681, so I'm closing this for now.

Member

nicoddemus commented Jul 6, 2017

We have reverted invocation scoped fixtures and I've reopened the original issue in #1681, so I'm closing this for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment