Whole planet build

[rpmcoding]

systemed you mention that if you want a full planet build (as we do) you should consider an alternative solution.

What other solutions would you suggest?

Thanks so much.

[systemed]

You can do a whole planet build with tilemaker now, if you have a lot of RAM and SSD. #315 has some comments. Note that it's best to use the bounding box in that example (-60° to 75° latitude), i.e. excluding the polar regions - tilemaker doesn't cope very well with the polar regions at present.

You could look at https://github.com/onthegomap/planetiler which has lower memory and disk requirements - if you just want to create an OpenMapTiles-compatible planet then this might be a good way to go.

[rpmcoding]

@systemed thanks, really appreciate that.

[sujithkris]

Whether anyone tried the whole planet with tilemaker and benchmarked the minimum system requirement?
I tried in an AWS instance with 64CPU/128GB/1900GBSSD. The full planet conversion is crashing the process.
Tried with coastline/water_ploygons and natural earth urban areas.

Any help?

[systemed]

Have you excluded the polar regions - i.e. by specifying a bounding box restricted to -60° to 75° latitude? That will make a big difference.

[sujithkris]

Essentially the reason to exclude the polar regions is because the coastline/landcover creates a lot of tiles here. Because of the way the Mercator projection works, tilemaker ends up creating vast numbers of objects for the coastline/landcover at the poles, using about 27GB extra RAM.

But if you don't have any objects in these areas, or you exclude these areas automatically, the objects won't get created.

#293 (comment) refers. #314 and #323 were attempts to fix this but both had problems.

Alright. Thanks for the clarification.
Newbie question - For the tile request for z15-z19, I can return the same z14 tiles from my tile server? My idea is to place an indoor map of the building on top of this planet map, hence zoom levels of 15+ are widely used. Primarily trying to migrate from Mapbox which has planet tiles till z22.

[systemed]

Yes, you can just use the z14 tiles. The vector data in these can be magnified for higher zoom levels.

Because tilemaker pre-generates vector tiles (rather than serving them on demand), generating everything up to z19 would be massively slow and memory-hungry.

If you're planning to go up to high zoom levels, you might want to set the high_resolution parameter in your JSON config (see #384). This doesn't affect RAM consumption and only makes the tiles slightly bigger, but will result in greater accuracy when you're zoomed in.

[sujithkris]

Yes, you can just use the z14 tiles. The vector data in these can be magnified for higher zoom levels.

Because tilemaker pre-generates vector tiles (rather than serving them on demand), generating everything up to z19 would be massively slow and memory-hungry.

If you're planning to go up to high zoom levels, you might want to set the high_resolution parameter in your JSON config (see #384). This doesn't affect RAM consumption and only makes the tiles slightly bigger, but will result in greater accuracy when you're zoomed in.

Highly insightful. Thanks

I am generating tiles using tilemaker, pushing tiles to CDN, and then serving from CDN

So we need a system with min 150GB memory to accommodate coastline and landcover/ne_urban_areas, right ? For me process crashed in the system of 128GB. but successfully generated 0-14 tiles with the default config without the above two shapes.

[systemed]

Interesting. I've run the planet successfully with 144GB (a few months ago, and obviously the data is always getting bigger). I have a long-term idea about reducing memory usage but that's going to take some heavy lifting; I wonder if there's anything more that could be done in the short term to reduce the coastline data overhead.

One thing you can do to save on memory is run the coastline/landcover and OSM-derived data separately. In other words, first run tilemaker without specifying a .pbf, and with a full bounding box (within the limits of the Mercator projection, so roughly -85° to 85° latitude). This will generate an .mbtiles containing the shapefile-derived data only. (It's worthwhile saving a copy of this somewhere.)

Then edit out the shapefile layers from the .json file, and rerun tilemaker with planet-latest.osm.pbf, this time using the --merge option with your newly created .mbtiles. It will overlay the OSM-derived data onto the already-created tiles with coastline etc.

[systemed]

#479 is a revival of #323 which I hope may be more efficient. Still testing.

[sujithkris]

@systemed : Not getting the (state/country boundaries) for planet osm pbf. Is this because I am not including the coastline/landcover shapes?

Tilemaker

Mapbox

[systemed]

tilemaker's OMT-compatible config does write boundaries (https://github.com/systemed/tilemaker/blob/master/resources/process-openmaptiles.lua#L249-L285) so they should be in the vector tiles. Maybe a stylesheet issue?

[sujithkris]

@systemed - I directly applied the styling from Maptiler, which again failed to draw the boundaries with tilemaker tiles. I am using tilemaker master branch.
My tilemaker-config.json

Also, very few ocean tiles are available (yes, I am using coastline shape also). However, almost all lakes are present.

This screenshot is with zoom 6.3 and center {lng: 79.34599067516137, lat: 8.080629437792709}

Do you recommend any stylesheet to check whether features are generated properly by tilemaker? ( to isolate any stylesheet issues)

[systemed]

The best way to see what's in a vector tile is to use an Inspect plugin: https://github.com/acalcutt/maplibre-gl-inspect

It's also sometimes helpful to use vt2geojson to fetch and decompile a tile into GeoJSON, which you can then inspect in QGIS or whatever your favoured editor is!

[sujithkris]

inspect helped!

"filter": [
        "all",
        [
          "in",
          "admin_level",
          3,
          4
        ],
        [
          "==",
          "maritime",
          0
        ]
      ],

I could see tilemaker generated tiles don't have maritime : 0 and my boundary styles not applied.
I took the styles as is from https://github.com/openmaptiles/openmaptiles/blob/master/layers/boundary/style.json

[sujithkris]

@systemed: Do we have a log file to keep track of progress? For me, the process looks like in _idle/sleep_. Generated store files (last modified is 30hrs before) but yet to generate tiles.

System
256.0 GiB | 16 vCPUs | 950 GB

Process
tilemaker --input /data/osm/planet-latest.osm.pbf --output /data/tiles --store /data/store --config 16_config.json --bbox -180,-60,180,75 --compact

I am running for minzoom 15 and maxzoom 16.

"settings": {
		"minzoom": 15,
		"maxzoom": 16,
		"basezoom": 16,
		"include_ids": false,
		"combine_below": 14,
		"name": "Tilemaker to OpenMapTiles schema",
		"version": "3.0",
		"description": "Tile config based on OpenMapTiles schema",
		"compress": "gzip",
		"filemetadata": {
			"tilejson": "2.0.0", 
			"scheme": "xyz", 
			"type": "baselayer", 
			"format": "pbf", 
            "tiles": ["https://example.com/liechtenstein/{z}/{x}/{y}.pbf"]
		}
	}

> top

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                     

42875 root      20   0  565.8g 228.3g  39.5g S 100.0  91.8   3904:39 tilemaker

> free -h

                    total        used        free      shared  buff/cache   available
Mem:            248Gi      190Gi      3.2Gi    0.0Ki     55Gi             56Gi
Swap:           0B            0B           0B

[systemed]

Any logging output goes to console (stdout). You can get more logging by running with --verbose but I doubt it would be relevant in this case.

There's two possibilities here - either it's slowly deallocating the node/way/relation store which is no longer needed at this stage of the process (void_mmap_allocator::shutdown(); in tilemaker.cpp), or it's taking a very long time to assemble the low-zoom tiles from the z16 constituents.

Out of interest, what's the reason you're generating tiles at z15/z16?

[sujithkris]

Out of interest, what's the reason you're generating tiles at z15/z16?

As my final plan is to place a building map on this from another source, z14 is too far and the building looks tiny. I may need support to zoom the full map till z20 at least to see the details of the rooms in my building map.

Well, I remember you hinted about reusing z14 pbfs for higher levels. Currently, for me, 15/x/y.pbf is requested when the map is zoomed to z15, (and so on) which results in a blank tile as z15 tiles are not present. Whether setting a maxzoom:14 in style json is sufficient to make the maplibre make requests to 14/x/y.pbf even at higher zoom levels?

I could see mapbox-gl-js is requesting 16/x/y.pbf even when the map is zoomed to 20.

[systemed]

Yes – if you set maxzoom for that source to 14 in the stylesheet, MBGL won't make requests any higher than that, and it'll just "overzoom" from the data in the z14 tiles. If you set the "high_resolution": true parameter in tilemaker's JSON config, then tilemaker will use extra resolution at z14 so the geometries stay sharp when you're zoomed in.

If you're planning on overlaying a building map, I'd suggest you generate the planet just up to z14 with the high_resolution parameter set. Then generate your building map separately.

I might put a warning into tilemaker to say "You really don't need to do this" when people generate tiles above z14/15 because no-one ever understands this about vector tiles ;)

[dieterdreist]

sent from a phone

On 29 Mar 2023, at 13:46, Richard Fairhurst ***@***.***> wrote: Yes – if you set maxZoom to 14 in the stylesheet, MBGL won't make requests any higher than that, and it'll just "overzoom" from the data in the z14 tiles. If you set the "high_resolution": true parameter in tilemaker's JSON config, then tilemaker will use extra resolution at z14 so the geometries stay sharp when you're zoomed in.

will it use the original input data resolution or will it still simplify but with a higher resolution?

[systemed]

Entirely up to you – you can specify simplification per-layer in tilemaker's JSON config.

[dieterdreist]

sent from a phone

On 29 Mar 2023, at 17:46, Richard Fairhurst ***@***.***> wrote: Entirely up to you – you can specify simplification per-layer in tilemaker's JSON config.

yes I know and am using it for one layer, just wanted to know what exactly the high detail flag did (skip simplification or change the threshold)

[systemed]

Ah, ok! It increases the vector tile resolution from 4096 to 8192 at the highest zoom level.

[sujithkris]

@systemed:

1. Just wondering about the impact of the --compact flag? I could not see any documentation w.r.t that.

2. Whether the coastline/water_polygons.shp is sufficient to address the below difference between openmaptiles+tilemaker and mapbox ?

i) Pier boundaries are not precise, seeing background layer at edges

Mapbox

Tilemaker

ii) Ocean Missing in left hand side ( however landlocked water bodies / ocean available)

Mapbox

Tilemaker

[sujithkris]

@systemed : I generated waterbodies separately without source ( ---input). This time I am getting full water bodies.
But my concern is how I can merge the tiles I generated now with the existing tiles of planet osm pbf.
I am not generating the single .mbtiles file, but rather direct .pbf files.

Question: Whether the --merge work for .pbf files generation? Or limited to .mbtiles ?

[systemed]

--merge is mbtiles-only I'm afraid.

--compact is used if you've renumbered your .pbf beforehand (with osmium renumber). It saves memory by using contiguous storage for nodes - basically it means tilemaker can use an array/vector rather than a hash/map.

[FunMiles]

I found this project from the README of planetiler. I thought (and still think) that it could allow me to play with custom profiles more easily than planetiler allows.
On planetiler, I had no problem doing the whole planet within a few hours on my laptop with 64GB. What is the reason tilemaker needs more memory?

[systemed]

Different internal storage basically. In particular tilemaker currently keeps all generated geometries in memory. I'm working on a refactoring in #499 that should enable the memory footprint to be substantially reduced.