INTERNALERROR> IndexError: list index out of range in hypothesis and pytest-bdd

[hroncok]

Hello pytest folks.

When testing the impact of the pytest upgrade form 7.2.x to 7.3.0 in Fedora, I found two interesting failures in

https://copr.fedorainfracloud.org/coprs/churchyard/pytest-7.3.0/package/python-hypothesis/ 6.62.1

platform linux -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0
rootdir: /tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_suppress_health_check_function_scoped_fixture0
plugins: hypothesis-6.62.1, xdist-3.2.0
collected 3 items

test_suppress_health_check_function_scoped_fixture.py 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 348, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/reports.py", line 363, in from_item_and_call
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>                ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 1833, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/nodes.py", line 484, in _repr_failure_py
INTERNALERROR>     return excinfo.getrepr(
INTERNALERROR>            ^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 671, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 987, in repr_excinfo
INTERNALERROR>     return ExceptionChainRepr(repr_chain)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 1039, in __init__
INTERNALERROR>     reprtraceback=chain[-1][0],
INTERNALERROR>                   ~~~~~^^^^
INTERNALERROR> IndexError: list index out of range

And with https://copr.fedorainfracloud.org/coprs/churchyard/pytest-7.3.0/package/python-pytest-bdd/ 6.1.1

platform linux -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_step_trace2
configfile: tox.ini
plugins: bdd-6.1.1, xdist-3.2.0
collecting ... collected 4 items / 3 deselected / 1 selected

test_step_trace.py::test_when_not_found <- ../../../../../builddir/build/BUILDROOT/python-pytest-bdd-6.1.1-7.fc39.x86_64/usr/lib/python3.11/site-packages/pytest_bdd/scenario.py 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>              ^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in <lambda>
INTERNALERROR>     lambda: oldcall(hook_name, hook_impls, kwargs, firstresult)
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 348, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>              ^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in <lambda>
INTERNALERROR>     lambda: oldcall(hook_name, hook_impls, kwargs, firstresult)
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call
INTERNALERROR>     result = func()
INTERNALERROR>              ^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in <lambda>
INTERNALERROR>     lambda: oldcall(hook_name, hook_impls, kwargs, firstresult)
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/reports.py", line 363, in from_item_and_call
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>                ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 1833, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/nodes.py", line 484, in _repr_failure_py
INTERNALERROR>     return excinfo.getrepr(
INTERNALERROR>            ^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 671, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 987, in repr_excinfo
INTERNALERROR>     return ExceptionChainRepr(repr_chain)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 1039, in __init__
INTERNALERROR>     reprtraceback=chain[-1][0],
INTERNALERROR>                   ~~~~~^^^^
INTERNALERROR> IndexError: list index out of range

I will try to retest with the latest versions of hypothesis, but pytest-bdd is up to date.

I am puzzleed on where to report this. Is this a bug in pytest? Or in the tests?

[nicoddemus]

Strange indeed... seems like the traceback in chained exception is empty? 🤔

[bluetech]

I ran out of time to look into this today. I'm assuming this didn't fail on 7.2.x which means this is a 7.3.0 regression. So first order of business would be to bisect. Will do it when I get a chance, though I'll be happy if someone beats me to it :)

[musicinmybrain]

I ran out of time to look into this today. I'm assuming this didn't fail on 7.2.x which means this is a 7.3.0 regression. So first order of business would be to bisect. Will do it when I get a chance, though I'll be happy if someone beats me to it :)

Working in a venv, I used git bisect between 7.2.2 and 7.3.0 with the pytest-bdd test suite (at master, pytest-dev/pytest-bdd@d45c543) and identified 431ec6d as the commit that introduced this regression.

[bluetech]

Thanks @musicinmybrain, very helpful. I'm looking into it.

431ec6d is backported to 7.2.x as well, though that wasn't (and will not be) released. In case anyone is confused why it still fails on 7.2.x branch :)

[bluetech]

I'm converting this to an issue for further discussion

[hroncok]

#10903