Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

INTERNALERROR when raising exception from pytest_runtest_call() after failed tests #333

Closed
pytestbot opened this Issue Jul 14, 2013 · 4 comments

Comments

Projects
None yet
1 participant
@pytestbot
Copy link

pytestbot commented Jul 14, 2013

Originally reported by: Nikolaus Rath (BitBucket: nikratio, GitHub: nikratio)


When raising an exception from pytest_runtest_call() after a test has been executed and failed, py.test crashes with an internal error.

If the test completed successfully, py.test works as expected and reports a test failure due to the exception in pytest_runtest_call().

I've attached a testcase. For me, the output is:

#!text

[0] nikratio@vostro:~/tmp$ py.test tests/test_good.py 
=============================================== test session starts ================================================
platform linux -- Python 3.3.2 -- pytest-2.3.5 -- /usr/bin/python3.3
collected 1 items 

tests/test_good.py:6: GoodTest.test_it FAILED

===================================================== FAILURES =====================================================
_________________________________________________ GoodTest.test_it _________________________________________________
Traceback (most recent call last):
  File "/home/nikratio/tmp/tests/conftest.py", line 20, in pytest_runtest_call
    raise AssertionError('Suspicious output to stderr')
AssertionError: Suspicious output to stderr
------------------------------------------------- Captured stderr --------------------------------------------------
Exception! Something went wrong in another thread!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================= 1 failed in 0.02 seconds =============================================
[2] nikratio@vostro:~/tmp$ py.test tests/test_crash.py 
=============================================== test session starts ================================================
platform linux -- Python 3.3.2 -- pytest-2.3.5 -- /usr/bin/python3.3
collected 1 items 

tests/test_crash.py:6: CrashTest.test_it 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/main.py", line 81, in wrap_session
INTERNALERROR>     doit(config, session)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/main.py", line 115, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 441, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 452, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 370, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/main.py", line 135, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 441, in __call__
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 452, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 370, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/runner.py", line 62, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/runner.py", line 72, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/runner.py", line 108, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/main.py", line 159, in call_matching_hooks
INTERNALERROR>     return hookmethod.pcall(plugins, **kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 445, in pcall
INTERNALERROR>     return self._docall(methods, kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 452, in _docall
INTERNALERROR>     res = mc.execute()
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 370, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/capture.py", line 171, in pytest_runtest_makereport
INTERNALERROR>     rep = __multicall__.execute()
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/core.py", line 370, in execute
INTERNALERROR>     res = method(**kwargs)
INTERNALERROR>   File "/home/nikratio/.local/lib/python3.3/site-packages/_pytest/unittest.py", line 150, in pytest_runtest_makereport
INTERNALERROR>     del call.result
INTERNALERROR> AttributeError: result

=================================================  in 0.01 seconds =================================================

I am not sure what the best behavior in this case would be, but an internal error seems like the worst alternative :-).


@pytestbot

This comment has been minimized.

Copy link
Author

pytestbot commented Sep 5, 2013

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):


Internal errors are to be expected when implementing hooks, and especially when using multicall.execute() so we don't consider this a bug, at least. (reporting could be nicer, maybe).

That being said, you could also improve your solution like this:

#!python

import sys
import pytest
# content of conftest.py

@pytest.fixture(autouse=True)
def save_capfd_fixture(request, capfd):
    def raise_on_exception_in_out():
        stdout, stderr = capfd.readouterr()
        if ('exception' in stderr.lower()
            or 'exception' in stdout.lower()):
            raise AssertionError('Suspicious output to stderr')
    request.addfinalizer(raise_on_exception_in_out)

@pytestbot

This comment has been minimized.

Copy link
Author

pytestbot commented Sep 8, 2013

Original comment by Nikolaus Rath (BitBucket: nikratio, GitHub: nikratio):


Thanks, that version works nicely enough for my usecase!

Generally, though, it seems to me that there really should be a way to make a test fail from a hook. If the problem is to distinguish between a true internal error, and an exception that was raised to make the test fail, maybe pytest could define a special exception type for that? E.g. if a hook raises PytestFailTest, don't treat it as an internal error but make the test fail?

@pytestbot

This comment has been minimized.

Copy link
Author

pytestbot commented Sep 9, 2013

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):


fix issue333: fix a case of bad unittest/pytest hook interaction.

@pytestbot

This comment has been minimized.

Copy link
Author

pytestbot commented Sep 9, 2013

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):


actually you can usually use pytest.fail or assertion errors from pytest_runtest* hooks. I just looked again, and indeed there was an edge case interaction issue here at work. Should be fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.