Reference leaks on Windows

[zware]

BPO	29041
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @serhiy-storchaka, @zooba, @zhangyangyu

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-01.02:17:21.701>
created_at = <Date 2016-12-21.22:28:41.724>
labels = ['3.7', 'OS-windows', 'performance']
title = 'Reference leaks on Windows'
updated_at = <Date 2017-10-02.09:31:29.198>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2017-10-02.09:31:29.198>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-01.02:17:21.701>
closer = 'zach.ware'
components = ['Windows']
creation = <Date 2016-12-21.22:28:41.724>
creator = 'zach.ware'
dependencies = []
files = []
hgrepos = []
issue_num = 29041
keywords = []
message_count = 4.0
messages = ['283792', '291802', '303446', '303517']
nosy_count = 7.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'xiang.zhang']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue29041'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[zware]

Discussion in bpo-29034 inspired me to run a refleak check on Windows, with the following results from a 32-bit build of the current 3.6 branch:

C:\cpython\3.6>PCbuild\build.bat -e -d
C:\cpython\3.6>python -m test -uall -R 3:3:refleak_log.txt --timeout=7200

...

24 tests failed:
test_atexit test_capi test_concurrent_futures test_fileio
test_functools test_glob test_idle test_io test_math
test_multiprocessing_spawn test_ntpath test_os test_platform
test_posixpath test_shutil test_ssl test_sys test_tarfile
test_tempfile test_threading test_unicode_file_functions
test_unittest test_warnings test_winconsoleio

1 test altered the execution environment:
test_distutils

28 tests skipped:
test_crypt test_curses test_dbm_gnu test_dbm_ndbm test_devpoll
test_epoll test_fcntl test_fork1 test_gdb test_grp test_ioctl
test_kqueue test_nis test_openpty test_ossaudiodev test_pipes
test_poll test_posix test_pty test_pwd test_readline test_resource
test_spwd test_syslog test_threadsignals test_wait3 test_wait4
test_zipfile64

Total duration: 190 min 60 sec
Tests result: FAILURE

test_atexit leaked [12, 12, 12] references, sum=36
test_atexit leaked [4, 6, 6] memory blocks, sum=16
test_capi leaked [6, 6, 6] references, sum=18
test_capi leaked [2, 4, 4] memory blocks, sum=10
test_concurrent_futures leaked [792, 783, 765] references, sum=2340
test_concurrent_futures leaked [264, 263, 257] memory blocks, sum=784
test_fileio leaked [1, 2, 2] memory blocks, sum=5
test_functools leaked [0, 3, 2] memory blocks, sum=5
test_glob leaked [265, 266, 266] memory blocks, sum=797
test_idle leaked [471, 471, 471] references, sum=1413
test_idle leaked [209, 211, 211] memory blocks, sum=631
test_io leaked [2, 3, 3] memory blocks, sum=8
test_multiprocessing_spawn leaked [168, 117, 144] references, sum=429
test_multiprocessing_spawn leaked [55, 43, 50] memory blocks, sum=148
test_ntpath leaked [103, 104, 104] memory blocks, sum=311
test_os leaked [33, 33, 33] references, sum=99
test_os leaked [58, 60, 60] memory blocks, sum=178
test_platform leaked [12, 12, 12] references, sum=36
test_platform leaked [4, 6, 6] memory blocks, sum=16
test_posixpath leaked [3, 4, 4] memory blocks, sum=11
test_shutil leaked [5, 6, 6] memory blocks, sum=17
test_ssl leaked [3, 4, 4] memory blocks, sum=11
test_sys leaked [9, 9, 9] references, sum=27
test_sys leaked [3, 5, 5] memory blocks, sum=13
test_tarfile leaked [4, 5, 5] memory blocks, sum=14
test_tempfile leaked [82, 83, 83] memory blocks, sum=248
test_threading leaked [12, 12, 12] references, sum=36
test_threading leaked [4, 6, 6] memory blocks, sum=16
test_unicode_file_functions leaked [5, 6, 6] memory blocks, sum=17
test_winconsoleio leaked [38, 38, 38] references, sum=114
test_winconsoleio leaked [0, 1, 2] memory blocks, sum=3

[zware]

Here's an updated refleak run, on master as of last Thursday:

test_atexit leaked [12, 12, 12] references, sum=36
test_atexit leaked [4, 6, 6] memory blocks, sum=16
test_capi leaked [6, 6, 6] references, sum=18
test_capi leaked [2, 4, 4] memory blocks, sum=10
test_asyncio leaked [4, 1, 1] memory blocks, sum=6
test_concurrent_futures leaked [822, 807, 810] references, sum=2439
test_concurrent_futures leaked [274, 271, 272] memory blocks, sum=817
test_import leaked [-2, 2, 0] references, sum=0
test_import leaked [-2, 4, 2] memory blocks, sum=4
test_logging leaked [-21, 1, 1] memory blocks, sum=-19
test_os leaked [6, 7, 7] memory blocks, sum=20
test_platform leaked [12, 12, 12] references, sum=36
test_platform leaked [4, 6, 6] memory blocks, sum=16
test_sys leaked [9, 9, 9] references, sum=27
test_sys leaked [3, 5, 5] memory blocks, sum=13
test_threading leaked [12, 12, 12] references, sum=36
test_threading leaked [4, 6, 6] memory blocks, sum=16
test_xml_etree_c leaked [6, -6, 0] references, sum=0
test_xml_etree_c leaked [1, 1, 2] memory blocks, sum=4
test_multiprocessing_spawn leaked [159, 180, 162] references, sum=501
test_multiprocessing_spawn leaked [54, 62, 55] memory blocks, sum=171

I ran this one with -j0 which may have affected the results slightly, but shaved 40 minutes off the run time.

[zware]

Victor has cleaned these up, and we now have a buildbot running refleak checks on Windows every day (as long as it's not hung :)

[vstinner]

Victor has cleaned these up, and we now have a buildbot running refleak checks on Windows every day (as long as it's not hung :)

\o/ Hopefully I was alone ;-) https://haypo.github.io/contrib-cpython-2017q2-part2.html