memory leak with generators

[florentx]

BPO	15635
Nosy	@ronaldoussoren, @pitrou, @ned-deily, @bitdancer, @florentx, @hynek
Files	testiterbug.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-08-17.12:25:02.924>
created_at = <Date 2012-08-13.09:25:30.706>
labels = ['interpreter-core', 'invalid', 'performance']
title = 'memory leak with generators'
updated_at = <Date 2012-08-17.12:25:02.923>
user = 'https://github.com/florentx'

bugs.python.org fields:

activity = <Date 2012-08-17.12:25:02.923>
actor = 'flox'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-17.12:25:02.924>
closer = 'flox'
components = ['Interpreter Core']
creation = <Date 2012-08-13.09:25:30.706>
creator = 'flox'
dependencies = []
files = ['26779']
hgrepos = []
issue_num = 15635
keywords = []
message_count = 10.0
messages = ['168072', '168082', '168083', '168088', '168089', '168108', '168120', '168124', '168125', '168441']
nosy_count = 6.0
nosy_names = ['ronaldoussoren', 'pitrou', 'ned.deily', 'r.david.murray', 'flox', 'hynek']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue15635'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[florentx]

Hello,

I noticed a large memory consumption in my application.
I tracked it down to be a problem with garbage collection of generator locals.

The issue was noticed in 2.6 first. Then I reproduced it in 2.7.
The test case finds some leak in 3.3 too, it seems.

[florentx]

Though, I cannot reproduce on Debian Squeeze (2.6.6 deb or 2.7 from source) or Ubuntu (2.7.2+ or 3.2).

Someone on OS X might confirm the same issue.
This is python 2.7.3 installed from source (using perlbrew) and GCC 4.2.1.

The output of the script is:

$ python testiterbug.py
2.7.3 (default, May 20 2012, 19:54:58)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

[row for row in iterit(16777216)]
Memory usage: 9.3 MB

[row for row in iterit(8388608)]
Memory usage: 266.6 MB

[florentx]

I don't mean perlbrew, but homebrew (an OS X package manager to install from source).

[ronaldoussoren]

I can reproduce this on an OSX 10.8 system, both using python 2.7 and python 3.3. The growth is significantly less using python 3.3.

What's odd is that the growth does not occur when both test_iter calls use 1<<24 as the argument (or larger values).

If I'd had to guess I'd say that the free implementation doesn't return a buffer to the system for smaller blocks and does do it for larger buffers.

[ronaldoussoren]

BTW. I don't think this is a memory leak, the amount of memory used doesn't increase when there are more calls to test_iter(1<<23).

[ronaldoussoren]

I've added calls to vmmap to the script (using os.system) to check my guesss.

The relevant bit after the call to test_iter(1<<24):

REGION TYPE VIRTUAL
=========== =======
MALLOC 405.9M see MALLOC ZONE table below
MALLOC guard page 32K
MALLOC metadata 376K
STACK GUARD 56.0M
Stack 8192K
VM_ALLOCATE 4K
__DATA 1720K
__LINKEDIT 53.4M
__TEXT 12.1M
__UNICODE 544K
shared memory 8K
=========== =======
TOTAL 538.1M

                                 VIRTUAL ALLOCATION      BYTES

MALLOC ZONE SIZE COUNT ALLOCATED % FULL
=========== ======= ========= ========= ======
DefaultMallocZone_0x100436000 404.9M 414059 390.6M 96%
DispatchContinuations_0x1007dc000 1024K 17 1056 0%
=========== ======= ========= ========= ======
TOTAL 405.9M 414076 390.6M 96%

Memory usage: 14.3 MB

And after the call to test_iter(1<<13)

REGION TYPE VIRTUAL
=========== =======
MALLOC 208.9M see MALLOC ZONE table below
MALLOC freed, no zone 456.0M
MALLOC guard page 32K
MALLOC metadata 392K
STACK GUARD 56.0M
Stack 8192K
VM_ALLOCATE 4K
__DATA 1720K
__LINKEDIT 53.4M
__TEXT 12.1M
__UNICODE 544K
shared memory 8K
=========== =======
TOTAL 797.1M

                                 VIRTUAL ALLOCATION      BYTES

MALLOC ZONE SIZE COUNT ALLOCATED % FULL
=========== ======= ========= ========= ======
DefaultMallocZone_0x100436000 207.9M 209467 197.0M 94%
DispatchContinuations_0x1007dc000 1024K 17 1056 0%
=========== ======= ========= ========= ======
TOTAL 208.9M 209484 197.0M 94%

Memory usage: 462.6 MB

The total amount of "memory usage" is grown by about 450 MBytes (according to psutils). This is explained by the change in MALLOC memory, before:

MALLOC 405.9M see MALLOC ZONE table below
MALLOC guard page 32K
MALLOC metadata 376K

Total: about 406 MByte

After:

MALLOC 208.9M see MALLOC ZONE table below
MALLOC freed, no zone 456.0M
MALLOC guard page 32K
MALLOC metadata 392K

Total: about 665 MByte.

The difference is about 260 MBytes, which is significant but less than the difference according to psutil. But: the psutil difference as calculated by the script is the difference in RSS, not VSIZE (which vmmap shows).

If I change the _mem function to use the VSIZE the before and after values are:

Memory usage: 2390.4 MB
Memory usage: 2846.4 MB

Difference: about 456 MByte, which is still higher than the output of vmmap.

Note that the same behavior can be seen by a much simpler script:

print "Large"
x = list(range(1<<24))
print _mem()
del x
print _mem()

print

print "Small"
x = list(range(1<<23))
print _mem()
del x
print _mem()

(where _mem is the same function as in testiterbug.py).

The output is:

Large
Memory usage: 527.3 MB
Memory usage: 19.3 MB

Small
Memory usage: 330.6 MB
Memory usage: 147.6 MB

With this script is see the same pattern with vmmap: with the "small" list a large amount of memory is in regions of type (MALLOC freed, no zone).

All of this clearly points towards malloc not releasing allocated memory to the system. This is perfectly fine and not a bug in either Python or the system.

[pitrou]

The test case finds some leak in 3.3 too, it seems.

3.3 should be much better since it uses mmap() and munmap() to manage the object allocator's arenas (unless this is disabled under OS X for some reason, that is).

Here under Linux:

$ python3.3 testiterbug.py 
3.3.0b1 (default:1d811e1097ed, Jul 29 2012, 20:20:37) 
[GCC 4.5.2]

[row for row in iterit(16777216)]
Memory usage: 7.8 MB

[row for row in iterit(8388608)]
Memory usage: 7.8 MB

[florentx]

All of this clearly points towards malloc not releasing allocated memory to the system.
This is perfectly fine and not a bug in either Python or the system.

So it means there's no reliable way to measure the memory consumption of the program.
Sorry for the wrong report, if this is confirmed.

[pitrou]

> All of this clearly points towards malloc not releasing allocated memory to the system.
> This is perfectly fine and not a bug in either Python or the system.

So it means there's no reliable way to measure the memory consumption of the program.

What it means is that "memory consumption" itself is a bit ill-defined.
"ps", "top" and friends will tell you the amount of memory (physical or
virtual) currently assigned by the OS to the process. It doesn't tell
you whether the memory is "actually" in use by the program.

Note that, in Python 3.3, sys._debugmallocstats() gives you
fined-grained information about the object allocator. This excludes any
blocks bigger than 512 bytes, though, since those are requested directly
using malloc() and free().

[florentx]

Thank you for digging into this. I close the issue.

I discover now that this kind of problem is quite common in the Mac world.

Other references:
http://news.ycombinator.com/item?id=3879194
http://www.markvanda.net/apple/mac-os-x-memory-issues/