mapnik2 uses a lot more memory than before

[artemp]

When I render the following tile with mod_tile's renderd it requires significantly more memory when I use the trunk mapnik2 code instead of the mapnik-0.7.x branch.

http://tile.osm.org/10/511/340.png
-- this meta tile covers all of central London and hence it is a pretty difficult test case.

With mapnik-0.7.x the peak virtual memory usage of the renderd process is around 1GB

renderd[28969]: DEBUG: Got command RenderPrio fd(9) xml(default), z(10), x(511), y(340)
renderd[28969]: DEBUG: Connection 0, fd 9 closed, now 0 left
renderd[28969]: DEBUG: DONE TILE default 10 504-511 336-343
renderd[28969]: DEBUG: DONE TILE default 10 504-511 336-343 in 26.177 seconds
$ grep VmPeak /proc/`pidof renderd`/status
VmPeak:  1028044 kB

With mapnik2 the peak usage is over 8GB:

renderd[29347]: DEBUG: Got command RenderPrio fd(67) xml(default), z(10), x(511), y(340)
renderd[29347]: DEBUG: Connection 0, fd 67 closed, now 0 left
renderd[29347]: DEBUG: DONE TILE default 10 504-511 336-343
renderd[29347]: DEBUG: DONE TILE default 10 504-511 336-343 in 182.653 seconds
$ grep VmPeak /proc/`pidof renderd`/status
VmPeak:  8434556 kB

The rendering time is a lot slower, probably due to the swapping caused by this extra memory usage.

I have not dug any deeper to figure out where the memory is being used. I'll probably start by turning off the new featureset caching because it looks like a likely culprit.

[artemp]

[springmeyer] Hmm, possibilities are the new feature caching (#624)

[artemp]

[jburgess777] If I force cache_features=false in apply_to_layer() then the very high peak memory usage no longer occurs.

I'll try to delve into the caching code a little to see if there are excessive objects being cached or if the objects themselves appear bloated.

[artemp]

[springmeyer] artem on skype mentioned with the mq style he only got ~180 MB max, so this style must not read nearly the same amount of data.

I ask him for the bbox and tile so I can test on a system without mod_tile:

box2d(-313086.0678560819,6574807.424977722,0,6887893.492833804)

10/0/0/17/245/128.meta

[artemp]

[springmeyer] assigning to mapnik2 release, milestone, as this needs further addressing before release. See original ticket at #624

[artemp]

[springmeyer] r2636 disables feature caching, but enables it to be turned on as an option on a layer. So, experienced users that want to test th potential io optimization for rendering layers with multiple styles (that do not pull enormous amounts of data) can turn on feature caching.