unittest: on failure, TestCase.run() keeps a reference to the exception

[vstinner]

BPO	19880
Nosy	@pitrou, @vstinner, @ezio-melotti
Files	unittest_leak.py unittest_workaround.patch contextmanager_leak.py contextmanager_leak2.py generator_workaround.patch frame_ref_cycle.py unittest_leak.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-12-09.00:56:40.951>
created_at = <Date 2013-12-03.22:46:40.957>
labels = []
title = 'unittest: on failure, TestCase.run() keeps a reference to the exception'
updated_at = <Date 2013-12-09.00:56:40.949>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2013-12-09.00:56:40.949>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2013-12-09.00:56:40.951>
closer = 'python-dev'
components = []
creation = <Date 2013-12-03.22:46:40.957>
creator = 'vstinner'
dependencies = []
files = ['32951', '32952', '32953', '32954', '32955', '32956', '32957']
hgrepos = []
issue_num = 19880
keywords = ['patch']
message_count = 7.0
messages = ['205167', '205168', '205173', '205174', '205177', '205179', '205630']
nosy_count = 4.0
nosy_names = ['pitrou', 'vstinner', 'ezio.melotti', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue19880'
versions = ['Python 3.4']

[vstinner]

Test attached unittest_leak.py script: you will see MyException.ninstance counter increased up to 10, whereas I expect that MyException is destroyed at TestCase.run() exit.

It looks like a tricky reference cycle between:

• frames
• exc_info local variable of _Outcome.testPartExecutor() context manager
• _Outcome.errors list
• _Outclass instance

Attached unittest_workaround.patch patch works around the issue.

[vstinner]

contextmanager_leak.py: shorter script to demonstrate the issue.

Replacing "exc_info = sys.exc_info()" with "sys.exc_info()" works around the issue.

[vstinner]

contextmanager_leak2.py: even simpler example, storing a current frame in a local variable of the frame is enough.

generator_workaround.patch is another workaround: call frame.clear() when at generator exit to explicitly break the reference cycle.

[vstinner]

frame_ref_cycle.py: if I understood correctly, unittest_leak.py can be simplified to this script. A frame contains a local variable which contains the frame: reference cycle.

[vstinner]

I found this issue while working the memory limit feature of tracemalloc module (issue bpo-19817). I opened bpo-19835 to workaround an unlimited loop on PyErr_NoMemory() when Python is out of memory.

See also the issue bpo-17807 and the PEP-442 for a similar reference cycle with generators.

[vstinner]

unittest_leak.patch: Fix the the initial bug, unittest_leak.py.

I don't think that it's possible to write a generic fix for frame_ref_cycle.py. Storing sys.exc_info() to format it as a traceback later is a common pattern. Clearing a frame at function exit breaks this use case.

[python-dev]

New changeset 09658ea0b93d by Victor Stinner in branch 'default':
Close bpo-19880: Fix a reference leak in unittest.TestCase. Explicitly break
http://hg.python.org/cpython/rev/09658ea0b93d