Unnecessary flat-nodes with enough RAM

[pnorman]

It's been said that you could almost use /dev/null for a flat-nodes file if you have a large enough -C value and don't plan to update the database. An option that allows you to use no file at all for flat nodes would be useful for when someone doesn't plan to update diffs and has enough memory. It would save on sequential writes and save on disk space. The latter is useful when using SSDs.

[lonvia]

How about literally allowing /dev/null instead of adding yet another option? Or would that be too obscure?

We'd have to disable ALLOW_LOSSY on the ram cache, so it fails if the cache is too small and chain get request right through. Not very difficult to do.

[pnorman]

#668 may fix this if it makes --cache unnecessary because it's a mmapped file

[pnorman]

@lonvia, did we ever test this? Perhaps there's a doc change needed.

[lonvia]

If you mean /dev/null, that does not work. If you mean the mmapped file, you probably still need the space on the disk but it might never be written to disk if there is enough memory and we delete it early. I can't test that on my setup at the moment.

[pnorman]

I was thinking of --slim --flat-nodes nodes.bin --cache 0, which then relies on OS caching for the mmapped file. But I guess there still is a use for a /dev/null equivalent

[lonvia]

The cache is more compact for extracts, so yes. Flat nodes on the other side cannot accidentally run out of memory.

[lonvia]

We should go about this in a different way: if --drop is given and no floatnode file, then osm2pgsql should only save the node locations in the DB that do not fit into the RAM cache. This has the same effect as requested in the original issue with the added bonus that it works even when the RAM cache is a little bit too small to keep all node locations.

[lonvia]

We now have a very efficient RAM middle, which allows to import the whole planet on a machine with 128GB RAM. That's what should be used on machines with enough RAM to import a non-updateable database.

Can we close this issue as not relevant anymore?