reduce compile memory usage peaks

[springmeyer]

xml_tree.cpp and json/feature_collection_parser.cpp have consistently been causing build failures on launchpad. This is new - we've been doing nighly builds on launchpad for over a year and have only recently seen out of memory problems due to these two files. It happens in 1 of every 10-15 builds, likely when the build is dispatched to a machine with less memory.

My hunch is these files are peaking well over 2 GB of memory. Ideally we could try to split up the translation units somehow.

[herm]

xml_tree.cpp is actually a very simple file. Splitting it wouldn't improve anything. The problem is that is depends on nearly all gramars which are very expensive to compile.
Moving the gramar definitions (and all boost spritit includes) from .hpp to .cpp would solve this problem.

[springmeyer]

Yep, I agree @herm on the solution for xml_tree.cpp. Here is the output of the json problem:

g++ -o src/json/feature_collection_parser.os -c -g -O3 -DHAVE_JPEG -DBOOST_REGEX_HAS_ICU -ansi -Wall -pthread -ftemplate-depth-300 -DLINUX -DBOOST_SPIRIT_THREADSAFE -DMAPNIK_THREADSAFE -O3 -finline-functions -Wno-inline -Wno-parentheses -Wno-char-subscripts -DNDEBUG -DHAVE_CAIRO -fPIC -I/usr/include/cairomm-1.0 -I/usr/lib/x86_64-linux-gnu/cairomm-1.0/include -I/usr/include/cairo -I/usr/include/sigc++-2.0 -I/usr/lib/x86_64-linux-gnu/sigc++-2.0/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/pixman-1 -I/usr/include/libpng12 -Ideps/agg/include -I. -Iinclude -I/usr/local/include -I/usr/include -I/usr/include/freetype2 -I/usr/include/libxml2 -I/usr/include/gdal -I/usr/include/postgresql src/json/feature_collection_parser.cpp

Session terminated, terminating shell...make: *** [build-arch] Terminated
 ...terminated.
scons: *** [src/json/feature_collection_parser.os] Build interrupted.
scons: Build interrupted.
scons: building terminated because of errors.
scons: writing .sconsign file.
make[1]: *** [mapnik] Error 2
Build killed with signal 15 after 150 minutes of inactivity

[herm]

I just had a look at the json file: It also include two grammars, so the solution would be the same.

[herm]

Are there any ideas how to reduce the size of these grammars in general? Currently the linker takes several minutes and uses about 1.5GB of RAM because the input files are that large.

9 of the 10 largest .os files contain grammars. Together they take up more than 600MB. Somehow that feels wrong considering that the grammars are not very complicated.

[novldp]

I can now compile xml_tree.cpp where before it consistently segfaulted cc1plus (gcc 4.5.3).

[springmeyer]

compile time of xml_node.cpp is now ~8 seconds for me after finishing moving the color and path_expression grammars to cpp files. closing. will re-open if I see any future failures on launchpad or from user reports.

[kapouer]

Reading this, it seems there is still a little room for improving memory usage during compilation.
On debian mipsel build servers, it fails most of the time:
https://buildd.debian.org/status/fetch.php?pkg=mapnik&arch=mipsel&ver=2.2.0%2Bds1-5&stamp=1385948694
I tried some gcc build flags (like lower optimizations) already without success.
I have no clue how to solve this and any help would be very welcome.

[springmeyer]

@kapouer - tracking further work at #2130. Is this still blocking? Is mipsel important? What is it?