Added a serial-threads options to have control over the memory used in the load-data part #26

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
4 participants

In some cases of postgres-configuration the different threads in the load-data part require much working memory and when it comes to swap, the process becomes endless slow. With the serial-threads option you can scale the amount of RAM that will be used at the same time.

Added serial-threads as option
In some cases of postgres-configuration the different threads in the load-data part require much working memory and when it comes to swap, the process becomes endless slow. With the serial-threads option you can scale the amount of RAM that will be used at the same time.
Owner

twain47 commented Aug 28, 2012

This sounds like you either need to use --threads to limit the number of threads or reconfigure postgresql - if it is using more working memory than you have in the box then it is missconfigured.

I worked with lonvia (very good support by the way. Thanks for that!) for about 3 weeks on the postgresql-config and it doesn't matter how low I put the values (including threads), it did not get better. I got 24GB on RAM so it's not that much lower than the recommendations in the wiki. This serial-threads-hack worked for me, but I admit it only works around the problem that postgres is using too much memory.

matt-42 commented Mar 1, 2013

I encountered the same problem on my configuration (4GB of RAM +2 GB of swap). I use postgres 9.2.3 and postgis 2.0.2 . The steps load data and index fail after filling all RAM and swap available, even with tunning the postgres memory parameters to the minimum, and --thread 1:

shared_buffers = 128kB
temp_buffers = 1MB
work_mem = 64kB
fsync = off
full_page_writes = off

checkpoint_segments = 1
checkpoint_timeout = 30s
checkpoint_completion_target = 0.1

I am trying to load just an 268MB bz2 osm file (http://download.geofabrik.de/openstreetmap/europe/france/ile-de-france.osm.bz2)

I got the following error in the index step:
Done 39650 in 1544 @ 25.680052 per second - Rank 30 ETA (seconds): 10224.979492
Done 39650 in 1545 @ 25.663431 per second - Rank 30 ETA (seconds): 10231.601562
Done 39650 in 1546 @ 25.646830 per second - Rank 30 ETA (seconds): 10238.224609
Done 39650 in 1546 @ 25.646830 per second - Rank 30 ETA (seconds): 10238.224609
index_placex: UPDATE failed: ERROR: Error executing external command:
[...]/Nominatim/nominatim/nominatim -i -d nominatim -P 5432 -t 1 -r 26

During the load-data step, I get a pgsql connection error: "the database system is in recovery mode"

I use the lastest nominatim git version (commit f201371)

Thanks,
Matthieu

Member

lonvia commented Mar 1, 2013

I suspect that this is somehow related to issue #31. Could you try out the suggestion in this comment and replace the regexp_replace(..) function simply with null.

Member

lonvia commented Apr 10, 2013

Issue should be solved with 23d3031.

@lonvia lonvia closed this Apr 10, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment