memory issue

[jeremy-rutman]

I hit an out-of-memory on a 64Gb ram aws machine during 'prepare all' for planet . The process continues so I am not sure how that impacts the geocoder, if at all :

+ pelias prepare all
  Creating extract at /data/placeholder/wof.extract
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                                                                                 /data/openstreetmap/planet-latest.osm.pbf is very large.
                                                                                                                                         You will likely experience memory issues working with large extracts like this.
           We strongly recommend using Valhalla to produce extracts for large PBF extracts.
                                                                                           see: https://github.com/pelias/polylines#download!data
                                                                                                                                                 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                     converting /data/openstreetmap/planet-latest.osm.pbf to /data/polylines/extract.0sv
                                                                                                        2019/03/29 17:13:17 found 0 refs for way 23687328
2019/03/29 17:16:04 found 0 refs for way 37090295
2019/03/29 17:16:04 found 0 refs for way 37147040
                                              2019/03/29 17:17:02 found 0 refs for way 47725798
                                                                                               2019/03/29 17:43:00 found 0 refs for way 314692067
Done!
import...
2019/03/29 17:54:32 found 0 refs for way 384162726
                                                  populate fts...
2019/03/29 17:58:00 found 0 refs for way 405118396
                                                  optimize...
close...
Done!
2019/03/29 18:01:56 found 0 refs for way 435366836
                                                  2019/03/29 18:07:10 found 0 refs for way 479715347
                                                                                                    2019/03/29 18:27:13 found 0 refs for way 620069036
                                                                                                                                                      2019/03/29 18:27:13 found 0 refs for way 620069065                                                                                                                                                            2019/03/29 18:27:13 found 0 refs for way 620069094
2019/03/29 18:27:13 found 0 refs for way 620069190
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0x8577c0, 0x16)
/usr/local/go/src/runtime/panic.go:619 +0x81
runtime.sysMap(0xd196ae0000, 0x100000, 0x11fb800, 0xc87c78)
/usr/local/go/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0xc6eae0, 0x100000, 0x7fa4cb174698)       
/usr/local/go/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0xc6eae0, 0x2, 0x0)
 /usr/local/go/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0xc6eae0, 0x2, 0xc87c88, 0x7fa4cb174600)
/usr/local/go/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0xc6eae0, 0x2, 0xc60049, 0xc7dbfb8fe8)
/usr/local/go/src/runtime/mheap.go:686 +0x118      
runtime.(*mheap).alloc.func1()                                                                                                                                                                   /usr/local/go/src/runtime/mheap.go:753 +0x4d 
runtime.(*mheap).alloc(0xc6eae0, 0x2, 0x7fa4cb010049, 0x7fa4cb174600)
/usr/local/go/src/runtime/mheap.go:752 +0x8a     
runtime.(*mcentral).grow(0xc71070, 0x0)
/usr/local/go/src/runtime/mcentral.go:232 +0x94    
runtime.(*mcentral).cacheSpan(0xc71070, 0xffffffffffffffff)
/usr/local/go/src/runtime/mcentral.go:106 +0x2e4 
runtime.(*mcache).refill(0x7fa580eb3000, 0xc7dbfb8f49)                    
/usr/local/go/src/runtime/mcache.go:123 +0x9c

btw prepare all consistently manages to somehow munge up the terminal display , preventing newlines from going to the begninning of the next line - that's happened on 2 debian jessie machines and one ubuntu 18 machine . most of the output above had a bunch of spaces everywhere, i left the first few lines as is

In any case I suppose the issue is either

• how much memory is needed at minimum for planet
• do these oom errors cause problems down the line

[missinglink]

As the author of the code in question I can tell you that it's unlikely that the algorithm I used will work for the planet regardless of how much RAM you throw at it.

You should consider these errors as fatal and follow the instructions in the warning message I left.

[missinglink]

Please don't use the issue tracker for general support, this case was clearly documented in the error message you provided along with a solution.

[jeremy-rutman]

I thought that valhalla was already being used by the docker install , having seen the ~/pelias/images/valhalla dir and dockerfile.
Sorry , my mistake .
I had added virtual memory which avoided this error but iiuc , valhalla should be used in any case.

[jeremy-rutman]

I tried using a valhalla-generated file (the one conveniently linked to here) , putting

 "imports": { 
....
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "road_network.polylines" ]
    },
....

into ~/pelias/projects/planet/polylines/pelias.json after putting the unzipped/renamed polylines file at /tmp/pelias/planet/data/polylines/road_network.polylines (my DATA_DIR being /tmp/pelias/planet/data), ran pelias elastic stop , nuked the directory /tmp/pelias/planet/data/elasticsearch , and tried the install from start, wait, create, prepare but after prepare I still get the valhalla warning and see

converting /data/openstreetmap/planet-latest.osm.pbf to /data/polylines/extract.0sv

which was the previously-used polylines file . Is this expected ?

Trying again with pelias compose down does remove the elasticsearch index , preventing an 'index exists' message on pelias elastic start (I since realize that pelias elastic remove will take care of removing a partially-built index) but otherwise the same - the valhalla msg appears and an extract.0sv starts getting built. Those msgs are generated by polylines:docker_extract.sh, called by prepare_polylines() (in turn called by pelias prepare all) for any pbf over 1Gb so I hope I am still on the right track - conceivably, one should only run the prepare steps other than prepare_polylines() if a polylines file is being supplied?

[jeremy-rutman]

ok it looks like you DON'T run pelias prepare polylines if you're using valhalla - and pelias prepare all includes that (the pelias commands are all defined here) - so you;d do pelias prepare placeholder and pelias prepare interpolation , skipping the pelias prepare polylines that comes first in the definition of prepare_all

[deanshelton913]

Suggestion: Why not roll valhalla into pelias as a dependency, and when a file is too large, pelias' prepare script could execute the valhalla scripts for you.

[missinglink]

FYI I just added this 3 days ago https://github.com/pelias/docker-valhalla-baseimage