Python cApi memory problem. Py_Initialize memory leak

[rstarostecki]

BPO	19893
Nosy	@vstinner, @skrah
Files	pytest.tgz: test code , valgrind output vgrind3.3.3vc.out vgrindNext.out

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-12-09.11:32:59.626>
created_at = <Date 2013-12-05.08:50:23.464>
labels = ['invalid', 'performance']
title = 'Python cApi memory problem. Py_Initialize memory leak'
updated_at = <Date 2013-12-09.11:46:34.566>
user = 'https://bugs.python.org/rstarostecki'

bugs.python.org fields:

activity = <Date 2013-12-09.11:46:34.566>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2013-12-09.11:32:59.626>
closer = 'rstarostecki'
components = []
creation = <Date 2013-12-05.08:50:23.464>
creator = 'rstarostecki'
dependencies = []
files = ['32983', '32986', '32989']
hgrepos = []
issue_num = 19893
keywords = []
message_count = 10.0
messages = ['205283', '205288', '205289', '205290', '205291', '205292', '205299', '205301', '205624', '205678']
nosy_count = 3.0
nosy_names = ['vstinner', 'skrah', 'rstarostecki']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue19893'
versions = ['Python 3.2', 'Python 3.3']

[rstarostecki]

I wrote small test program using Python cApi, and found some memory issues while working on it. I checked it with valgrind. (test code and valgrind output appended)
Maybe I'm doing something wrong, but most of the problem occurs durring Py_Initialize. I've checked python3.2.5 and 3.3.3 version. The latest version is much less confusing, but I can still see there something nasty.
I would be grateful if you could look at this, and tell me If I'm doing something wrong, or if I can do something to prevent this memory issues.

[skrah]

Did you use --suppressions=Misc/valgrind-python.supp?

[skrah]

Also, you have zero "definitely lost". "possibly lost" is not
particularly informative in the context of running Python:
These are almost certainly false positives.

[rstarostecki]

I didn't use --suppressions=Misc/valgrind-python.supp before . (But I've done it now). Nothing important has changed.
I understand that "possibly lost is not particularly informative". I'm rather worried about "Invalid read of size 4" etc. Isn't it potentially dangerous? I thought it is some kind of memory corruption, isn't it?

[skrah]

Did you compile Python --with-valgrind?

[rstarostecki]

I've just done it. Python 3.3.3 --with-valgrind. I can't see the difference. Output appended.

[vstinner]

You shoud configure --with-valgrind *and* use the suppression list, or
disable pymalloc using configure.

Victor

[rstarostecki]

I compiled python --with-valgrind --without-pymalloc, and used valgrind with suppressions.

valgrind --suppressions=../Misc/valgrind-python.supp --leak-check=full --show-reachable=no --show-possibly-lost=no --track-origins=yes --log-file=vgrindNext.out ./test

[vstinner]

Sorry, but I still don't understand this issue.

"Invalid read of size 4" is a known false positive. It can be worked around using ./configure --with-valgrind and the suppression list, or using ./configure --without-pymalloc. If you still get the warning, you used the wrong options or you are still using another Python binary (or shared library). Make sure that you are linked to your newly compiled shared library.

"valgrind --leak-check=full --log-file=valgrind.log ./python -c pass" shows me "possibly lost: 286,779 bytes in 654 blocks". This is also another known issue: Python doesn't release all the memory at exit, they are many "singletons" and variables initialized once but never released.

The PEP-3121 helps this issue but the PEP is not fully implemented yet, many modules should still be modified.

So what is your question? Do you think that Python leaks memory? Why do you think so?

[rstarostecki]

I've checked it one more time.
And you're right (Sorry for trouble). I left old pyconfig.h in one place, so my new python compilation was not just what I wanted.

Now I belive that everything with memory is ok.

Thank you very much for your help.