Multi-core optimization

[band-p]

I have an 18G oms.pbf file.
When I execute the following command using docker.

docker run -v /tmp:/app osmtw/tilemaker \
    --output /app/highways.mbtiles \
    --input /app/highways.osm.pbf \
    --config resources/config-openmaptiles.json \
    --process resources/process-openmaptiles.lua
    --combine

cpu uses only one core, and the execution process is very slow.

can you optimize the use of multi-core.

[systemed]

Is that single-core throughout the process?

.pbf reading (the first half of the run) is single-threaded but tile generation (the second half) should be multi-threaded.

[band-p]

The docker container has been running for an hour,I am not sure if he has finished reading

[kleunen]

It is still reading, at a certain moment you will see how many shapes are loaded:
Stored 9639463 nodes, 1192436 ways, 7462 relations
Shape points: 0, lines: 0, polygons: 19
Generated points: 628203, lines: 365421, polygons: 977403

From this point on, it is multi-core. But an 18G file will take a long time to load.

[kleunen]

You can use one of the prebuilt executables from CI:
https://github.com/systemed/tilemaker/suites/2667959237/artifacts/58763551

And I would recommend using the store to reduce memory:
./tilemaker --store osm_store.dat
--output /app/highways.mbtiles
--input /app/highways.osm.pbf
--config resources/config-openmaptiles.json
--process resources/process-openmaptiles.lua

You have 64G memory available, but maybe add an additional 64G of swap. Because even with the store, it takes a lot of memory.

[band-p]

Thanks for your answer

[band-p]

I really have 64G memory on this computer, not swap

[kleunen]

Yes, i understand. But actually 64G may not be enough to process 18G osm.pbf! So maybe add an additional 64G. It depends on how many layers you have enabled.

[band-p]

I use I use prebuilt executables execute this command.

./tilemaker --store osm_store.dat
--output /app/highways.mbtiles
--input /app/highways.osm.pbf
--config resources/config-openmaptiles.json
--process resources/process-openmaptiles.lua

Fifteen hours passed.


May I know if that's normal?

[kleunen]

Yes. That is normal. You are nearing the point where it starts generating. This also will take hours

[kleunen]

But it would be a good idea to at least include the water polygons:

From the main page:
If you want to include sea tiles, then create a directory called coastline in the same place you're running tilemaker from, and then save the files from https://osmdata.openstreetmap.de/download/water-polygons-split-4326.zip in it, such that tilemaker can find a file at coastline/water_polygons.shp.

Because otherwise the map looks quite odd, without seas and oceans.

[band-p]

highways.osm.pbf is worldwide highway vector data.
I downloaded the full amount of data from OpenStreetMap and use osmium-too to split the higways data.

https://github.com/osmcode/osmium-tool

May not need ocean data.

[kleunen]

ok, just have a look. Usually it is better to start testing at country level to see if everything is OK.

[band-p]

You are right, 64G memory is not enough to process 18G osm.pbf!
So I increased the memory to 128G and split osm.pbf by continent.
Successfully made mbtiles.
Slice generation is multi-threaded.

[kleunen]

Excellent

[kleunen]

@systemed Maybe it is good to open the discussion section on tilemaker github, it is possible to have a discussion area on the project. Discussion like this is good to keep for reference.