New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when assert fails in multiprocessing task #1984

Closed
adborden opened this Issue Oct 4, 2016 · 8 comments

Comments

Projects
None yet
4 participants
@adborden

adborden commented Oct 4, 2016

Seems that pytest crashes when trying to get the code snippet of an assertion failure when the assertion failure occurs in a multiprocessing task.

Here's a log of the pytest run https://gist.github.com/adborden/376f2dd3defae8fced33b655913e5f91#file-pytest_crash-log

Here's the test case that causes the crash
https://gist.github.com/adborden/376f2dd3defae8fced33b655913e5f91#file-test_pytest_crash-py

Here's a snippet of the traceback (see above for full stack).

INTERNALERROR>   File "/home/adborden/.pyenv/versions/3.4.5/envs/pytest-crash/lib/python3.4/site-packages/_pytest/_code/code.py", line 373, in __init__
INTERNALERROR>     self.traceback = _pytest._code.Traceback(self.tb, excinfo=self)
INTERNALERROR>   File "/home/adborden/.pyenv/versions/3.4.5/envs/pytest-crash/lib/python3.4/site-packages/_pytest/_code/code.py", line 270, in __init__
INTERNALERROR>     list.__init__(self, tb)
INTERNALERROR> TypeError: 'NoneType' object is not iterable

Using pytest-3.0.3 on Ubuntu 15.10

Virtualenv:

$ pip list
pip (8.1.1)
py (1.4.31)
pytest (3.0.3)
setuptools (20.10.1)
@RonnyPfannschmidt

This comment has been minimized.

Member

RonnyPfannschmidt commented Oct 5, 2016

indeed, py.test currently expects real tracebacks, and multiprocessing is NOT supplying that
it seems like an issue we will have to work around :(

@Daenyth

This comment has been minimized.

Daenyth commented Oct 25, 2016

This is affecting me really heavily - I'm trying to covert a large python2 app to python3 and this is preventing most of the suite from running.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Oct 26, 2016

I noticed this only happens because of the code which tries to print chained exceptions in Python 3+. As a workaround, it is possible to use pytest --tb=native:

_____________________________ test_multitask_job ______________________________
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\pytest\.tmp\test_foo.py", line 5, in process_task
    assert False
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\pytest\.tmp\test_foo.py", line 15, in test_multitask_job
    multitask_job()
  File "C:\pytest\.tmp\test_foo.py", line 11, in multitask_job
    pool.map(process_task, tasks, chunksize=1)
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "d:\programming\python35\Lib\multiprocessing\pool.py", line 608, in get
    raise self._value
AssertionError
========================== 1 failed in 0.37 seconds ===========================
@Daenyth

This comment has been minimized.

Daenyth commented Oct 26, 2016

That works, thanks!

@Daenyth

This comment has been minimized.

Daenyth commented Oct 26, 2016

Thanks, that works! I suspect that it also happens for threading, not just
multiprocessing, since I'm not actually using multiprocessing directly,
though maybe flask's test utilities do?

On Tue, Oct 25, 2016 at 8:33 PM Bruno Oliveira notifications@github.com
wrote:

I noticed this only happens because of the code which tries to print
chained exceptions in Python 3+. As a workaround, it is possible to use pytest
--tb=native:

_____________________________ test_multitask_job ______________________________
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(_args, *_kwds))
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\pytest.tmp\test_foo.py", line 5, in process_task
assert False
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\pytest.tmp\test_foo.py", line 15, in test_multitask_job
multitask_job()
File "C:\pytest.tmp\test_foo.py", line 11, in multitask_job
pool.map(process_task, tasks, chunksize=1)
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "d:\programming\python35\Lib\multiprocessing\pool.py", line 608, in get
raise self._value
AssertionError
========================== 1 failed in 0.37 seconds ===========================


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1984 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAA5NMOTab1XvsU8z-2_MlJCi5OQ84I0ks5q3p_fgaJpZM4KODNP
.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Oct 26, 2016

@Daenyth could you please post the traceback you were getting? Just want to make sure you were also having the exact same problem as the OP.

@Daenyth

This comment has been minimized.

Daenyth commented Oct 26, 2016

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 96, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 131, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 152, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 79, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 135, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/hypothesis/extra/pytestplugin.py", line 69, in pytest_runtest_makereport
INTERNALERROR>     report = (yield).get_result()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/skipping.py", line 222, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/pytest_capturelog.py", line 175, in pytest_runtest_makereport
INTERNALERROR>     report = __multicall__.execute()
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/runner.py", line 272, in pytest_runtest_makereport
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/python.py", line 587, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/python.py", line 580, in _repr_failure_py
INTERNALERROR>     style=style)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/main.py", line 433, in _repr_failure_py
INTERNALERROR>     style=style, tbfilter=tbfilter)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 423, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 635, in repr_excinfo
INTERNALERROR>     excinfo = ExceptionInfo((type(e), e, e.__traceback__))
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 373, in __init__
INTERNALERROR>     self.traceback = _pytest._code.Traceback(self.tb, excinfo=self)
INTERNALERROR>   File "/Users/daenyth/Curata/cmp/.tox/py35-unit/lib/python3.5/site-packages/_pytest/_code/code.py", line 270, in __init__
INTERNALERROR>     list.__init__(self, tb)
INTERNALERROR> TypeError: 'NoneType' object is not iterable
@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Oct 27, 2016

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment