python-2.7.5-r3: 40 bytes in 1 blocks are definitely lost

[mmokrejs]

BPO	18884
Nosy	@tim-one, @pitrou, @vstinner, @skrah
Files	valgrind_python275_without-pymalloc.txt: valgrind output python275 without pymalloc

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-08-30.16:03:30.520>
created_at = <Date 2013-08-30.13:45:47.518>
labels = ['interpreter-core', 'invalid', 'performance']
title = 'python-2.7.5-r3: 40 bytes in 1 blocks are definitely lost'
updated_at = <Date 2013-08-30.18:44:06.451>
user = 'https://bugs.python.org/mmokrejs'

bugs.python.org fields:

activity = <Date 2013-08-30.18:44:06.451>
actor = 'tim.peters'
assignee = 'none'
closed = True
closed_date = <Date 2013-08-30.16:03:30.520>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2013-08-30.13:45:47.518>
creator = 'mmokrejs'
dependencies = []
files = ['31523']
hgrepos = []
issue_num = 18884
keywords = []
message_count = 6.0
messages = ['196538', '196558', '196560', '196561', '196564', '196569']
nosy_count = 5.0
nosy_names = ['tim.peters', 'mmokrejs', 'pitrou', 'vstinner', 'skrah']
pr_nums = []
priority = 'low'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue18884'
versions = ['Python 2.7']

[mmokrejs]

It is not important why I had in this moment matplotlib not in sync with python itself whcih was configure using --with-pydebug ... but here I just want to show that maybe you do not test for memleaks using valgrind on import errors (maybe include such testcase into you tests).
Anyway, here is what I got:

==14007== Memcheck, a memory error detector
==14007== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==14007== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==14007== Command: /usr/bin/python2.7 blah.py
==14007== 
Traceback (most recent call last):
  File "blah.py", line 288, in <module>
    import pylab
  File "/usr/lib64/python2.7/site-packages/pylab.py", line 1, in <module>
    from matplotlib.pylab import *
  File "/usr/lib64/python2.7/site-packages/matplotlib/pylab.py", line 222, in <module>
    from matplotlib import mpl  # pulls in most modules
  File "/usr/lib64/python2.7/site-packages/matplotlib/mpl.py", line 1, in <module>
    from matplotlib import artist
  File "/usr/lib64/python2.7/site-packages/matplotlib/artist.py", line 7, in <module>
    from transforms import Bbox, IdentityTransform, TransformedBbox, \
  File "/usr/lib64/python2.7/site-packages/matplotlib/transforms.py", line 35, in <module>
    from matplotlib._path import (affine_transform, count_bboxes_overlapping_bbox,
ImportError: /usr/lib64/python2.7/site-packages/matplotlib/_path.so: undefined symbol: _PyMem_DebugFree
[100070 refs]
==14007== 
==14007== HEAP SUMMARY:
==14007==     in use at exit: 6,303,492 bytes in 31,921 blocks
==14007==   total heap usage: 1,266,299 allocs, 1,234,378 frees, 179,304,947 bytes allocated
==14007== 
==14007== 40 bytes in 1 blocks are definitely lost in loss record 167 of 3,515
==14007==    at 0x4C2C63B: malloc (vg_replace_malloc.c:270)
==14007==    by 0x4EF1E8C: PyMem_Malloc (object.c:2343)
==14007==    by 0x10064848: initialize_builtin_datetime_metadata (arraytypes.c.src:3953)
==14007==    by 0x100719E2: set_typeinfo (arraytypes.c.src:4047)
==14007==    by 0x1016354E: initmultiarray (multiarraymodule.c:4057)
==14007==    by 0x4FB2661: _PyImport_LoadDynamicModule (importdl.c:53)
==14007==    by 0x4FAE3A7: load_module (import.c:1915)
==14007==    by 0x4FB07C1: import_submodule (import.c:2700)
==14007==    by 0x4FAFCC6: load_next (import.c:2515)
==14007==    by 0x4FAED8B: import_module_level (import.c:2224)
==14007==    by 0x4FAF320: PyImport_ImportModuleLevel (import.c:2288)
==14007==    by 0x4F78862: builtin___import__ (bltinmodule.c:49)

[vstinner]

According to traces, the leak does not come from Python but matplotlib. Please report the issue to matplotlib bug tracker.

[mmokrejs]

Why do you think so? My point is that this happens when import fails. But python is at fault and should handle import errors.

[tim-one]

The matplotlib people won't care about this one either. matplotlib allocated the memory, and the error message at the end says it's _trying_ to call _PyMem_DebugFree (which may well be trying to release the memory), but the binaries aren't compatible: _PyMem_DebugFree doesn't exist. That's on you, not on matplotlib or on Python - you're mixing together binaries that can't possibly work together.

So if this bug is re-opened, it should be assigned to you to fix it ;-)

[mmokrejs]

Of course I did recompile matplotlib to use the recompiled python with debug ABI. I just don't understand why if something is not leaded why a memleak has to happen.

Anyway, now my application using the recompiled matplotlib baffled this through valgrind as well. An there was no ImportError: ;-)

==17489== 40 bytes in 1 blocks are definitely lost in loss record 718 of 5,487
==17489== at 0x4C2C63B: malloc (vg_replace_malloc.c:270)
==17489== by 0x4EF1E8C: PyMem_Malloc (object.c:2343)
==17489== by 0x10064848: initialize_builtin_datetime_metadata (arraytypes.c.src:3953)
==17489== by 0x100719E2: set_typeinfo (arraytypes.c.src:4047)
==17489== by 0x1016354E: initmultiarray (multiarraymodule.c:4057)
==17489== by 0x4FB2661: _PyImport_LoadDynamicModule (importdl.c:53)
==17489== by 0x4FAE3A7: load_module (import.c:1915)
==17489== by 0x4FB07C1: import_submodule (import.c:2700)
==17489== by 0x4FAFCC6: load_next (import.c:2515)
==17489== by 0x4FAED8B: import_module_level (import.c:2224)
==17489== by 0x4FAF320: PyImport_ImportModuleLevel (import.c:2288)
==17489== by 0x4F78862: builtin___import__ (bltinmodule.c:49)

This is probably because I also added --without-pymalloc as you wanted.

[tim-one]

Now you have something to show the matplotlib folks - although they're not likely to get excited about leaking 40 bytes.

There is nothing Python can do about this. matplotlib is responsible for free'ing the memory matplotlib allocates, just as Python is responsible for free'ing the memory Python allocates. If _you_ wrote a C extension module, and did

char *p = (char *)malloc(40);

and never did a free(p) later, it would be the same kind of thing.