How much resource do I need to build planet tiles?

[junior-yoda]

valhalla_build_tiles is killed and it seems it's because of high RAM usage, Machine which tiles are build on contains 32GB of RAM. Is this a bug or it really needs more RAM?
How much RAM do I need?

[kevinkreiser]

@junior-yoda it shouldnt need that much ram but it will use as much ram as it can (it memory maps files that are many 10s of gigabytes). memory mapping can make it appear as if you are ram limitted but its not always the case. can you tell us what message you see when its killed? what the return code is? what stage of the data processing it has reached?

there are been a decent amount of changes as late and im fairly certain we havent been very strict with our memory requirements so they may have grown. @gknisely what kind of ram usage do you see in your latest runs?

[gknisely]

@junior-yoda all of our conversions are running on tasks that have 16GB of RAM.

[junior-yoda]

@kevinkreiser There is no special return code or error message but here is full log, which may help.
I also retried and same thing happend.

[kevinkreiser]

Killed does indeed sound like out of memory. @junior-yoda are you running master? Looks like its in the enhancer phase as well.. We did have some recent changes in there wrt speeds ups. Can you try a commit before this one: #2349 or perhaps even earlier to see if its related?

[dnesbitt61]

was running with 24 threads I believe. Could that be the issue?

[kevinkreiser]

@dnesbitt61 good suggestion. have you tried with less parallelism @junior-yoda? also please note, you can resume your run from the enhancer stage by using the checkpointing feature of valhalla_build_tiles. doing this can save you time in debugging.

[junior-yoda]

@dnesbitt61 @kevinkreiser I'm using release tag 3.0.8. You mean this amount of parallelism could be the reason of high RAM usage? I guess it seems promising, I will try less parallelism and let you know the result.
I didn't know there is a checkpoint feature, it can be really helpful. Thanks.

[junior-yoda]

@kevinkreiser Limiting build process to 4 cores of CPU solved the problem. Thanks.