sys.exit() in a test causes a test run to die

[voidspace]

BPO	10611
Nosy	@rbtcollins, @merwok, @voidspace, @serhiy-storchaka
Files	issue_10611.diff: patch against py3k

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/voidspace'
closed_at = <Date 2010-12-19.14:56:28.056>
created_at = <Date 2010-12-03.01:25:17.075>
labels = ['easy', 'type-bug']
title = 'sys.exit() in a test causes a test run to die'
updated_at = <Date 2021-09-17.13:18:30.937>
user = 'https://github.com/voidspace'

bugs.python.org fields:

activity = <Date 2021-09-17.13:18:30.937>
actor = 'serhiy.storchaka'
assignee = 'michael.foord'
closed = True
closed_date = <Date 2010-12-19.14:56:28.056>
closer = 'michael.foord'
components = []
creation = <Date 2010-12-03.01:25:17.075>
creator = 'michael.foord'
dependencies = []
files = ['19994']
hgrepos = []
issue_num = 10611
keywords = ['patch', 'easy']
message_count = 9.0
messages = ['123153', '123717', '123718', '123726', '124110', '124356', '124357', '230788', '230790']
nosy_count = 5.0
nosy_names = ['rbcollins', 'eric.araujo', 'michael.foord', 'ysj.ray', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10611'
versions = ['Python 2.7', 'Python 3.2']

[voidspace]

Reported by a unittest2 user.

A SystemExit (or GeneratorExit) will cause a test run to stop in 2.7 / 3.2. This would just be reported as an error in 2.6.

>>> from unittest import TestCase
>>> def test(s):
...  raise GeneratorExit
... 
>>> class T(TestCase):
...  test = test
... 
>>> t = T('test')
>>> t.run()

Above code works in Python 2.6 (the exception is caught by TestCase.run) but dies in 2.7 / 3.2

[ysjray]

Agreed. I think the "except Exception" in TestCase.run() should be "except BaseException", since BaseException could catch Exception, SystemExit, GeneratorExit, KeyboardInterrupt. The KeyboardInterrupt should be caught first. The remaining three is exactly what is needed.

Here is a patch I worked, with unittest.

[merwok]

LGTM.

[voidspace]

At the moment exception handling for setUp / tearDown / testMethod and cleanUp functions are all handled separately. They all have to call addError and as a result we have inconsistent handling of skips, expected failures (etc). There are separate issues for handling expected failures in setUp and skips in tearDown.

I'd like to fix all these issues by moving the exception handling into a single method and unifying the reporting of failure / error / expected failure / skip test. This will fix all these issues and nicely simplify the implementation.

[ysjray]

I'd like to fix all these issues by moving the exception handling into a single method and unifying the reporting of failure / error / expected failure / skip test. This will fix all these issues and nicely simplify the implementation.

That sounds good.

[voidspace]

Committed to Python 2.7 in revision 87406.

[voidspace]

Committed to py3k in revision 87390.

[rbtcollins]

Hmm, so testtools went in a different direction here - the same unification stuff, but see testing-cabal/testtools@18bc574

tl;dr - I think unittest should not prevent this causing the process to exit (but it should still fail the test and fail the test run as a whole), analgous to how KeyboardInterrupt is handled.

[voidspace]

Allowing sys.exit() to end the test run was particularly a problem for testing command line tools, where improper patching / unexpected code paths would trigger a sys.exit.

If a test framework author wants a way to end the test run I'm happy to provide that (a custom exception to raise or a flag to turn off sys.exit handling), but I don't think having sys.exit kill test runs is best for test authors.