Python memory allocator: Free memory

[vulturex]

BPO	1123430
Nosy	@tim-one
Files	obmalloc-free-arenas.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tim-one'
closed_at = <Date 2006-06-30.06:17:06.000>
created_at = <Date 2005-02-15.21:27:11.000>
labels = ['interpreter-core']
title = 'Python memory allocator: Free memory'
updated_at = <Date 2006-06-30.06:17:06.000>
user = 'https://bugs.python.org/vulturex'

bugs.python.org fields:

activity = <Date 2006-06-30.06:17:06.000>
actor = 'tim.peters'
assignee = 'tim.peters'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2005-02-15.21:27:11.000>
creator = 'vulturex'
dependencies = []
files = ['6502']
hgrepos = []
issue_num = 1123430
keywords = ['patch']
message_count = 11.0
messages = ['47788', '47789', '47790', '47791', '47792', '47793', '47794', '47795', '47796', '47797', '47798']
nosy_count = 3.0
nosy_names = ['tim.peters', 'illume', 'vulturex']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1123430'
versions = ['Python 2.5']

[vulturex]

This is the second version of my Python memory allocator patch.
The first version was discussed on the python-dev mailing list
here:

http://mail.python.org/pipermail/python-dev/2005-January/
051255.html

This patch enables Python to actually return memory to the
system. The current version's memory usage will only grow. This
version maintains the same backwards compatability guarantees
as the previous version: Calling PyObject_Free with a pointer that
was returned by malloc() while NOT holding the GIL will work, and
will not corrupt the state of the memory allocator.

The patch modifies obmalloc.c. If it is accepted, other
modifications to that file are required. In particular, I have not yet
updated the WITH_MEMORY_LIMITS implementation, nor have I
looked closely at the PYMALLOC_DEBUG code to see what changes
(if any) are required.

[vulturex]

Logged In: YES
user_id=539295

Please ignore this patch for the moment: I'm in the process of making
some fixes.

[vulturex]

Logged In: YES
user_id=539295

As per the discussion on python-dev, I've removed the concurrency hack.
The routines in obmalloc.c now *must* be called while holding the GIL,
even if the pointer was allocated with malloc(). I also finally fixed the
PYMALLOC_DEBUG routines, so I believe this patch is now "complete."

[vulturex]

Logged In: YES
user_id=539295

Whoops! I uploaded a "fixed" version a while ago, but I guess I didn't
update the comments. The patch currently attached to this is the most up-
to-date version. Sorry about that.

[tim-one]

Logged In: YES
user_id=31435

Assigned to me.

[tim-one]

Logged In: YES
user_id=31435

The patch here is out of date, but that's OK. I created
branch tim-obmalloc, with a working version of the patch,
extensively reformatted to Python's C style, and with some
minor changes to squash compiler warnings. I plan to finish
this during PyCon.

[vulturex]

Logged In: YES
user_id=539295

Great news! If you need any assistance, I would be more than
happy to help.

[tim-one]

Logged In: YES
user_id=31435

The tim-obmalloc branch was merged to the trunk (for Python
2.5a1) in revision 43059. Thank you again for your hard
work and patience, Evan!

[illume]

Logged In: YES
user_id=2042

Note, that this patch doesn't fix all memory leaks of this
type. For example, this code below doesn't release all the
memory after it is run. It starts at about 3MB, goes up to
about 56MB and then only drops to 50MB.

AFAIK restarting python processes is still needed to reduce
memory usage for certain types of processes.

<pre>

import random
import os

def fill(map):
    random.seed(0)
    for i in xrange(300000):
        k1 = (random.randrange(100),
              random.randrange(100),
              random.randrange(100))
        k2 = random.randrange(100)
        if k1 in map:
            d = map[k1]
        else:
            d = dict()
        d[k2] = d.get(k2, 0) + 1
        map[k1] = d

if __name__ == "__main__":
    os.system('ps v')
    d = dict()
    fill(d)
    print "-"* 50
    print "\n" * 3
    os.system('ps v')
    del d

    import gc
    gc.collect()

print "-"* 50
print "\\n" * 3
os.system('ps v')

</pre>

[illume]

Logged In: YES
user_id=2042

I have done some more tests... and it seems that
dictionaries do not release as much memory as lists do.

Here is a modification of the last example posted.

If you only let fill2() run almost all the memory is freed.
fill2() uses lists. However if you let the others which
use dicts run not all of the memory is freed. The processes
are still 38MB when the data is del'ed.

It is still much better than python2.4, however I think
something fishy must be going on with dicts.

<pre>

import random
import os

def fill_this_one_doesnot_free_much_at_all(map):
    random.seed(0)
    for i in xrange(300000):
        k1 = (random.randrange(100),
              random.randrange(100),
              random.randrange(100))
        k2 = random.randrange(100)
        if k1 in map:
            d = map[k1]
        else:
            d = dict()
        d[k2] = d.get(k2, 0) + 1
        map[k1] = d


def fill(map):
    random.seed(0)
    for i in xrange(3000000):
        map[i] = "asdf"


class Obj:
    def __init__( self ):
        self.dumb = "hello"


def fill2(map):
    a = []
    for i in xrange(300000):
        o = Obj()
        a.append(o)
    return a


if __name__ == "__main__":
    import time
    import gc

    os.system('ps v | grep memtest')
    d = dict()
    a = fill2(d)
    #a2 = fill(d)
    a2 = fill_this_one_doesnot_free_much_at_all(d)
    print "-"* 50
    print "\n" * 3
    os.system('ps v | grep memtest')
    del d
    del a

    gc.collect()

    time.sleep(2)
    for x in xrange(100000):
        pass
    print "-"* 50
    print "\n" * 3
    os.system('ps v | grep memtest')
</pre>

[tim-one]

Logged In: YES
user_id=31435

As the NEWS entry says,

"""
Patch bpo-1123430: Python's small-object allocator now returns
an arena to the system free() when all memory within an
arena becomes unused again. Prior to Python 2.5, arenas
(256KB chunks of memory) were never freed. Some
applications will see a drop in virtual memory size now,
especially long-running applications that, from time to
time, temporarily use a large number of small objects. Note
that when Python returns an arena to the platform C's
free(), there's no guarantee that the platform C library
will in turn return that memory to the operating system.
The effect of the patch is to stop making that impossible,
and in tests it appears to be effective at least on
Microsoft C and gcc-based systems.
"""

An instrumented debug build of current trunk showed that
Python eventually returned 472 of 544 allocated arenas to
the platform free() in your program, leaving 72 still
allocated when Python shut down. The latter is due to
fragmentation, and there will never be "a cure" for that
since CPython guarantees never to move objects. The arena
highwater mark was 240 arenas, which is what Python would
have held on to forever before the patch.

A large part of the relatively disappointing result we see
here is due to that the tuple implementation maintains its
own free lists too: it never returns a few thousand of the
tuple objects to obmalloc before Python shuts down, and that
in turn keeps all the arenas from which those tuple objects
were originally obtained alive until Python shuts down.
There's nothing obmalloc or gcmodule can do about that, and
the tuple free lists are _likely_ an important optimization
in real applications.