Rank 30 ETA Error

[SanyaBeer]

On rank 30 I get error on the same place.

Done 39072755 in 156167 @ 250.198517 per second - Rank 30 ETA (seconds): 173854.015625
index_placex: UPDATE failed: ERROR:  buffer 151937 is not owned by resource owner Portal
CONTEXT:  SQL statement "INSERT INTO search_name_36 values (in_place_id, in_rank_search, in_rank_address,
        in_name_vector, in_geometry)"
PL/pgSQL function insertsearchname(integer,bigint,character varying,integer[],integer[],integer,integer,double precision,geometry,geometry) line 301 at SQL statement
PL/pgSQL function placex_update() line 359 at assignment
ERROR: Error executing external command: /root/Nominatim/nominatim/nominatim -i -d nominatim -P 5432 -t 23 -r 26
Error executing external command: /root/Nominatim/nominatim/nominatim -i -d nominatim -P 5432 -t 23 -r 26

There is some technical info:
OS:

Operating System: Debian GNU/Linux 8 (jessie)
Kernel: Linux 3.16.0-4-amd64

Postgres v.9.4 settings:

shared_buffers = 4GB                # min 128kB
work_mem = 50MB                     # min 64kB
maintenance_work_mem = 16GB         # min 1MB
synchronous_commit = off            # synchronization level;
fsync = off                         # turns forced synchronization on or off
full_page_writes = off              # recover from partial page writes
effective_cache_size = 24GB         
checkpoint_segments = 100           # in logfile segments, min 1, 16MB each
checkpoint_timeout = 10min          # range 30s-1h
checkpoint_completion_target = 0.9  # checkpoint target duration, 0.0 - 1.0

PostGis ("select PostGIS_full_version();"):

POSTGIS="2.1.4 r12966" GEOS="3.4.2-CAPI-1.8.2 r3921" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10.1, released 2013/08/26" LIBXML="2.9.1" LIBJSON="UNKNOWN" RASTER

Nominatim .local settings ("Osm2pgsql_Index in other tablespace on SSD"):

// Paths
@define('CONST_Postgresql_Version', '9.4');
@define('CONST_Postgis_Version', '2.1');
// osm2pgsql settings
@define('CONST_Osm2pgsql_Flatnode_File', '/var/nominatim/flatnode.file');
// tablespace settings
// osm2pgsql caching tables (aka slim mode tables) - update only
@define('CONST_Tablespace_Osm2pgsql_Data', 'nominatim_general');
@define('CONST_Tablespace_Osm2pgsql_Index', 'nominatim_ind');
// osm2pgsql output tables (aka main table) - update only
@define('CONST_Tablespace_Place_Data', 'nominatim_general');
@define('CONST_Tablespace_Place_Index', 'nominatim_general');
// address computation tables - update only
@define('CONST_Tablespace_Address_Data', 'nominatim_general');
@define('CONST_Tablespace_Address_Index', 'nominatim_general');
// search tables - needed for lookups
@define('CONST_Tablespace_Search_Data', 'nominatim_general');
@define('CONST_Tablespace_Search_Index', 'nominatim_general');
// additional data, e.g. TIGER data, type searches - needed for lookups
@define('CONST_Tablespace_Aux_Data', 'nominatim_general');
@define('CONST_Tablespace_Aux_Index', 'nominatim_general');

Hardware:
RAM 64GB, 150GB swap space;
CPU 24 core;

There are two questions:

1. Why indexing stops and how it fix?
2. If tablespace location and settings is optimal? (only nominatim_ind stores on SSD, nominatim_general on the HDD without RAID). On indexing we have load-avg about 18-20; total dsk read/write 4MB/10MB.

Thanks for help!

[mtmail]

Thanks for the detailed report.

It looks like the postgresql server crashed.

Do you see any errors in the postgresql log file (/var/log/postgresql or similar)? Is there enough space left on the harddrives? Do you have monitoring on the RAM usage?

The tablespaces, load-avg, dsk read/write look fine. I've run a full planet import just a couple of days ago (with 64GB RAM) so I doubt it's anything data related.

[SanyaBeer]

1. About log files. There are some errors about:
   ERROR: permission denied for relation new_query_log
   and
   ERROR: permission denied for relation query_log
   but looks like it's not what we look, but some specific thing.. After crash indexing in log:

ERROR:  buffer 284997 is not owned by resource owner Portal
CONTEXT:  SQL statement "INSERT INTO search_name_139 values (in_place_id, in_rank_search, in_rank_address,
            in_name_vector, in_geometry)"
    PL/pgSQL function insertsearchname(integer,bigint,character varying,integer[],integer[],integer,integer,double precision,geometry,geometry) line 1714 at SQL statement
    PL/pgSQL function placex_update() line 699 at assignment
STATEMENT:  update placex set indexed_status = 0 where place_id = $1
LOG:  could not send data to client: Connection reset by peer
STATEMENT:  select place_id from placex where rank_search = $1 and geometry_sector = $2 and indexed_status > 0
FATAL:  connection to client lost
STATEMENT:  select place_id from placex where rank_search = $1 and geometry_sector = $2 and indexed_status > 0
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
LOG:  could not send data to client: Broken pipe
........more Broken pipe.........

and second crash:

ERROR:  buffer 151937 is not owned by resource owner Portal
CONTEXT:  SQL statement "INSERT INTO search_name_36 values (in_place_id, in_rank_search, in_rank_address,
            in_name_vector, in_geometry)"
    PL/pgSQL function insertsearchname(integer,bigint,character varying,integer[],integer[],integer,integer,double precision,geometry,geometry) line 301 at SQL statement
    PL/pgSQL function placex_update() line 359 at assignment
STATEMENT:  update placex set indexed_status = 0 where place_id = $1
LOG:  could not receive data from client: Connection reset by peer
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
........more Broken pipe.........

It seems that connection lost. But what is the reason?

2. On drive we have 250GB free space on SSD, tablespace "nominatim_ind" and around 500GB "nominatim_general";

3. RAM usage about 5GB VIRT/RES for postgre and nominatim, and 59GB in cache. Swap space absolutely free, about 50Mb;

[lonvia]

I've now seen this error on my test instance as well (Debian stretch). Looks actually like an internal postgres problem where the different threads run each other into a deadlock. As a workaround the indexing can be safely continued with the following command:

./utils/setup.php --index --create-search-indices

[SanyaBeer]

@lonvia In this way indexing continues within the level? Or starts at the beginning of the unfinished rank?

[lonvia]

It will continue within the level exactly where it left off. (You'll see it scanning again through all rank levels but that is just because unindexed rows are specially marked and it searches for the next marked ones.)

[lonvia]

Closing as I suspect it is more of a Postgres issue than a Nominatim issue. I haven't seen this error in a while.