Memory leak in load_map/load_map_from_string

[artemp]

It looks like both load_map and load_map_from_string functions have memory leaks. I've attached the script that reproduces the problem (should run fine on any system with ps available). The script itself simply creates a new map, loads XML file and then del's map object in a cycle. Below is the output for 10 iterations:
mishok 11031 0.0 0.6 54536 13440 pts/5 R+ 16:23 0:00 python memleak.py
mishok 11031 86.9 3.5 112948 73736 pts/5 R+ 16:23 0:30 python memleak.py
As you can see memory usage goes from 13MB to almost 74MB. Running valgrind points to some "possible leaks" in libxml2, but I'm not sure if that's relevant.
This script was tested with Mapnik 0.6.0, 0.6.1 and recent checkout of 0.7 branch and the results are the same for all versions.
System: Ubuntu 9.10, Boost version:" 1.38, Python: 2.6, libxml2: 2.7.5.

[artemp]

[mishok13] Sometimes I hate myself for forgetting about existence of preview button. Here's the version with correct formatting:

It looks like both load_map and load_map_from_string functions have memory leaks. I've attached the script that reproduces the problem (should run fine on any system with ps available). The script itself simply creates a new map, loads XML file and then del's map object in a cycle.

Below is the output for 10 iterations:

{{{
mishok 11031 0.0 0.6 54536 13440 pts/5 R+ 16:23 0:00 python memleak.py
mishok 11031 86.9 3.5 112948 73736 pts/5 R+ 16:23 0:30 python memleak.py
}}}

As you can see memory usage goes from 13MB to almost 74MB. Running valgrind points to some "possible leaks" in libxml2, but I'm not sure if that's relevant.

This script was tested with Mapnik 0.6.0, 0.6.1 and recent checkout of 0.7 branch and the results are the same for all versions.

System: Ubuntu 9.10, Boost version: 1.38, Python: 2.6, libxml2: 2.7.5.

[artemp]

[mishok13] Uhm, it looks like libxml2 (or mapnik?) doesn't release memory after parsing the XML file. I guess that's related to the fact that xmlFreeParserCtxt call doesn't release memory acquired ctx_.myDoc (according to doc). I'll try to release that one also, this might resolve the issue.

[artemp]

[mishok13] Nope, calling xmlFreeDoc doesn't help. Calling malloc_trim(0) in libxml2_loader destructor didn't help either.

[artemp]

[springmeyer] mishok13, I see you cc'ed yourself - are you not getting emails as the 'reporter'?

Perhaps your email is not filled out in the trac prefs? http://trac.mapnik.org/prefs

[artemp]

[springmeyer] hmm, calling xmlFreeDoc does work for me.

I can replicate, and fix, on Mac 10.5 with:

{{{
#!diff

Index: src/libxml2_loader.cpp

--- src/libxml2_loader.cpp (revision 1505)
+++ src/libxml2_loader.cpp (working copy)
@@ -151,6 +151,7 @@
}

             populate_tree( root, pt );

•            xmlFreeDoc(doc);
       }
  
   private:
  

  }}}

Here is the output of the testing script before and after:

Current trunk on osx 10.5:

{{{
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
spring 17017 52.0 0.6 115956 12904 s001 S+ 2:27PM 0:00.31 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 89.5 0.9 121124 19608 s001 S+ 2:27PM 0:00.88 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 89.4 1.0 123188 21748 s001 S+ 2:27PM 0:01.44 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 91.3 1.1 125252 23888 s001 S+ 2:27PM 0:02.01 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 91.6 1.2 127316 26024 s001 S+ 2:27PM 0:02.58 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 90.6 1.3 129380 28160 s001 S+ 2:27PM 0:03.14 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 91.9 1.4 131444 30304 s001 S+ 2:27PM 0:03.71 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 92.3 1.5 133508 32448 s001 S+ 2:27PM 0:04.28 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 92.8 1.6 135572 34584 s001 S+ 2:27PM 0:04.85 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 93.5 1.8 137636 36720 s001 S+ 2:27PM 0:05.43 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17017 94.8 1.9 139700 38864 s001 S+ 2:27PM 0:06.00 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
}}}

With xmlFreeDoc()
{{{
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
spring 17035 67.9 0.6 127236 13592 s001 S+ 2:28PM 0:00.33 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 88.4 0.9 131364 19336 s001 S+ 2:28PM 0:00.92 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 95.1 0.9 131364 19344 s001 S+ 2:28PM 0:01.50 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 90.5 0.9 131364 19344 s001 S+ 2:28PM 0:02.09 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 91.6 0.9 131364 19344 s001 S+ 2:28PM 0:02.67 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 92.5 0.9 131364 19344 s001 S+ 2:28PM 0:03.25 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 92.8 0.9 131364 19344 s001 S+ 2:28PM 0:03.83 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 96.9 0.9 131364 19344 s001 S+ 2:28PM 0:04.41 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 93.4 0.9 131364 19344 s001 S+ 2:28PM 0:04.99 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 91.5 0.9 131364 19344 s001 S+ 2:28PM 0:05.57 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
spring 17035 92.4 0.9 131364 19344 s001 S+ 2:28PM 0:06.15 /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python ./memleak
}}}

[artemp]

[springmeyer] so, this should be fixed in trunk in r1507 and the 0.7 series branch in r1506.

Feel free to re-open if this does not fix for you.

[artemp]

[mishok13] Damn it, I knew something went wrong with cc. :)
I'll check your fix when I get back from vacation and see if this works. Thanks for fixing this without my input.