platform.win32_ver() leaks in 2.7.12 #72119
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = 'https://github.com/zooba' closed_at = <Date 2016-09-21.16:08:56.448> created_at = <Date 2016-09-01.15:34:31.099> labels = ['3.7', 'OS-windows', 'performance'] title = 'platform.win32_ver() leaks in 2.7.12' updated_at = <Date 2017-03-31.16:36:14.564> user = 'https://bugs.python.org/OkkoWilleboordse'
activity = <Date 2017-03-31.16:36:14.564> actor = 'dstufft' assignee = 'steve.dower' closed = True closed_date = <Date 2016-09-21.16:08:56.448> closer = 'steve.dower' components = ['Windows'] creation = <Date 2016-09-01.15:34:31.099> creator = 'Okko.Willeboordse' dependencies =  files = ['44324'] hgrepos =  issue_num = 27932 keywords =  message_count = 13.0 messages = ['274144', '274145', '274146', '275520', '275521', '275672', '275673', '275715', '275744', '275753', '276859', '276861', '276862'] nosy_count = 8.0 nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'python-dev', 'Okko.Willeboordse', 'zach.ware', 'eryksun', 'steve.dower'] pr_nums = ['552'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'resource usage' url = 'https://bugs.python.org/issue27932' versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']
The text was updated successfully, but these errors were encountered:
I did a gc.get_objects() before and after platform.win32_ver() and printed
If I run platform.win32_ver() in a loop I see the memory increasing.
On 1 September 2016 at 17:35, STINNER Victor <firstname.lastname@example.org> wrote:
The leak is due the pointer-type cache that ctypes.POINTER uses. The type the pointer refers to is used as the key. In this case, VS_FIXEDFILEINFO is created each time win32_ver is called, so the pointer-type cache grows without bound. Example leak:
>>> import psutil, platform >>> proc = psutil.Process() >>> proc.memory_info().rss 14704640 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 92704768 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 168861696
Clearing the cache followed by a collect() reclaims the leaked memory for the most part:
>>> import gc, ctypes >>> gc.collect() 333 >>> proc.memory_info().rss 168849408 >>> ctypes._pointer_type_cache.clear() >>> gc.collect() 740000 >>> proc.memory_info().rss 20303872
It's a moot point, since Steve plans to re-implement this check in C, but the minimal change to fix this leak is to bypass the pointer-type cache by manually subclassing ctypes._Pointer:
class PVS_FIXEDFILEINFO(_Pointer): _type_ = VS_FIXEDFILEINFO pvi = PVS_FIXEDFILEINFO()
There's no more leak after this change:
>>> import psutil, platform >>> proc = psutil.Process() >>> proc.memory_info().rss 15450112 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 16592896 >>> for i in range(10000): ... v = platform.win32_ver() ... >>> proc.memory_info().rss 16601088
Limiting the pointer-type cache could be a problem. It's expected that POINTER() returns a reference to an existing pointer type. ctypes uses Python types to ensure compatible C data types. For example:
LP_c_int = ctypes.POINTER(ctypes.c_int) class LP_c_int_uncached(ctypes._Pointer): _type_ = ctypes.c_int
>>> arr = (LP_c_int * 2)() >>> ctypes.POINTER(ctypes.c_int) is LP_c_int True >>> arr = ctypes.POINTER(ctypes.c_int)() >>> arr = LP_c_int_uncached() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: incompatible types, LP_c_int_uncached instance instead of LP_c_int instance
The docs could warn users that the cache used by ctypes.POINTER(), and implicitly by ctypes.pointer(), will leak memory if it's called with a type that's created at function scope instead of at module or class-body scope.
Phew. Doing that in a reasonable way for all the active versions was a pain, but I think I managed it.
AFAICT the implementation is solid, but please shout if you spot anything weird/wrong.