minetestmapper + postgresql = oom?

[BluebirdGreycoat]

So, some time ago my server was using the default SQLite database for the map. minetestmapper was working fine, on a virtual system that then had 2 GB of memory. I even used it to generate huge map images that were about 6224x6224 pixels, unscaled. I still have the successfully generated images.

I soon decided to upgrade the map database to a PostgreSQL database stored separately from the virtual OS itself. Since that time minetestmapper has consistently failed to generate images for even TINY portions of the map, while using huge amounts of server memory (available memory is now 4 GB).

Obviously something is wrong with how one or more of my processes is configured, because Minetest has no issues with the database (it is now 30+ GB in size), and the client can use the sent chunks to generate the image for the minimap. And unless I'm mistaken, I think that PostgreSQL is one of the supported backends for minetestmapper.

I tried compiling a debug version of minetestmapper but was unable to obtain a stacktrace. I'm pretty sure the OOM killer is whacking it.

Here is how I'm trying to execute a test run (which fails shortly after using 51% of server memory):

minetestmapper --geometry "0:0+16+16" -i "/home/minetest/server/world" -o "/home/minetest/test.png"

Here is a snippet of output from top shortly before the crash:

top - 15:39:45 up 8 days, 18:43,  2 users,  load average: 2.61, 2.12, 1.12
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us, 59.9 sy,  0.0 ni,  0.0 id, 38.8 wa,  0.0 hi,  0.2 si,  0.5 st
KiB Mem :  4046376 total,   106972 free,  3247760 used,   691644 buff/cache
KiB Swap:        0 total,        0 free,        0 used.    19892 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
12743 minetest  20   0 2232412 1.959g   1684 S   0.0 50.8   0:07.42 minetestmapper

Here is my world.mt file:

gameid = musttest
backend = postgresql
creative_mode = false
enable_damage = true
pgsql_connection = dbname=survivalworld
player_backend = sqlite3

[BluebirdGreycoat]

Here's a backtrace obtained by running the program under gdb and interrupting it from time to time. All traces obtained this way were the same, leading me to think that the program is stalled at this point.

Program received signal SIGINT, Interrupt.
0x00007ffff6be5730 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:84
84      in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0  0x00007ffff6be5730 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1  0x00007ffff78e673b in ?? () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#2  0x00007ffff78e68c8 in ?? () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#3  0x00007ffff78e48a1 in PQgetResult () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#4  0x00007ffff78e4bab in ?? () from /usr/lib/x86_64-linux-gnu/libpq.so.5
#5  0x00000000004381f7 in DBPostgreSQL::execPrepared (this=0x64c810, stmtName=0x43900c "get_block_pos", paramsNumber=0, params=0x0, paramsLengths=0x0,
    paramsFormats=0x0, clear=false, nobinary=false) at /home/me/source/git/mapper/minetestmapper/db-postgresql.cpp:144
#6  0x0000000000437ba1 in DBPostgreSQL::getBlockPos (this=0x64c810) at /home/me/source/git/mapper/minetestmapper/db-postgresql.cpp:59
#7  0x000000000042877c in TileGenerator::loadBlocks (this=0x7fffffffdd90) at /home/me/source/git/mapper/minetestmapper/TileGenerator.cpp:336
#8  0x0000000000427e96 in TileGenerator::generate (this=0x7fffffffdd90, input="/home/minetest/server/world", output="/home/minetest/test.png")
    at /home/me/source/git/mapper/minetestmapper/TileGenerator.cpp:245
#9  0x0000000000434c08 in main (argc=7, argv=0x7fffffffe578) at /home/me/source/git/mapper/minetestmapper/mapper.cpp:223
(gdb)

[sfan5]

Fixed by 5b264fd.
Previously minetestmapper tried to load the position of every MapBlock into memory.