Reference leaks in Python 3.0rc3

[tiran]

BPO	4373
Nosy	@loewis, @brettcannon, @amauryfa, @tiran
Files	issue4373_build_ext.patch pickle-leak2.patch issue4373_build_ext2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-11-25.21:21:59.908>
created_at = <Date 2008-11-21.02:28:58.494>
labels = ['release-blocker', 'performance']
title = 'Reference leaks in Python 3.0rc3'
updated_at = <Date 2008-11-25.21:21:59.906>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2008-11-25.21:21:59.906>
actor = 'christian.heimes'
assignee = 'none'
closed = True
closed_date = <Date 2008-11-25.21:21:59.908>
closer = 'christian.heimes'
components = []
creation = <Date 2008-11-21.02:28:58.494>
creator = 'christian.heimes'
dependencies = []
files = ['12086', '12090', '12092']
hgrepos = []
issue_num = 4373
keywords = ['patch']
message_count = 11.0
messages = ['76158', '76159', '76161', '76162', '76167', '76169', '76172', '76175', '76422', '76428', '76431']
nosy_count = 4.0
nosy_names = ['loewis', 'brett.cannon', 'amaury.forgeotdarc', 'christian.heimes']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue4373'
versions = ['Python 3.0']

[tiran]

A refleak test of r67311 py3k shows several leaks:

test_distutils leaked [142, 142, 142, 142] references, sum=568
test_httpservers leaked [0, 0, 0, 217] references, sum=217
test_multiprocessing leaked [0, 0, 24, 0] references, sum=24
test_pickle leaked [1, 1, 1, 1] references, sum=4
test_pickletools leaked [1, 1, 1, 1] references, sum=4
test_telnetlib leaked [-84, 84, 0, 0] references, sum=0
test_threadedtempfile leaked [94, -94, 0, 0] references, sum=0

[tiran]

Only distutils and pickle seem to have real leaks.

$ ./python Lib/test/regrtest.py -R:15: test_multiprocessing
test_multiprocessing
beginning 20 repetitions
12345678901234567890
....................
1 test OK.
[124096 refs]

$ ./python Lib/test/regrtest.py -R:15: test_distutils
test_distutils
beginning 20 repetitions
12345678901234567890
....................
test_distutils leaked [144, 144, 144, 144, 144, 144, 144, 144, 144, 144,
144, 144, 144, 144, 144] references, sum=2160
1 test OK.
[123507 refs]

$ ./python Lib/test/regrtest.py -R:15: test_httpservers
[...]
test_httpservers leaked [-217, 0, 198, 19, -35, 20, -202, 217, -217,
217, 0, 0, 0, 0, -33] references, sum=-33
1 test OK.
[96060 refs]

$ ./python Lib/test/regrtest.py -R:15: test_pickle
test_pickle
beginning 20 repetitions
12345678901234567890
....................
test_pickle leaked [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
references, sum=15
1 test OK.
[96783 refs]

[tiran]

distutils.tests.test_build_ext leaks the references. I think it's
related to the xx module and totally harmless.

[tiran]

This simple patch doesn't load the 'xx' module more than once in ref
leak tests.

The problem may also be caused by the xxmodule itself. Somebody should
give it a proper review :)

[amauryfa]

I have studied this some time ago. The xx module comes each time from a
different file (in a directory created with tempfile.mkdtemp).

Every instance of the module creates a new entry in the static
"extensions" dictionary in Python/import.c. This entry at least contains
a copy of the module dictionary, which explains the number of "leaked"
references.

I do not see any way to clear this dictionary. The proposed patch is
probably the best thing to do.

[amauryfa]

The refleak in test_pickle comes from unpickler_read(). The attached
patch corrects it.

[tiran]

Good analysis Amaury! The new patch for build ext uses a single temp dir.

[amauryfa]

The problem is that on Windows (and cygwin) you cannot unlink a .pyd
that is currently loaded in memory.

I tried to use ctypes and call FreeLibrary. Now the .pyd can be removed,
but the interpreter crashes when it comes to free the module on shutdown.
I'm afraid that until python has a real support for dlclose() on dynamic
loaded module, the best is to skip this test during the refleak hunt.

[brettcannon]

Amaury's patch for pickle looks fine to me.

[amauryfa]

Applied pickle-leak2.patch in 67381.

[tiran]

Second fix applied in r67382