tilemaker on the whole planet: from 2 h 10 min to under 24 minutes, and how you can help land it

[dktue]

Hi everyone 👋

In mid-October, building vector tiles for the whole planet with tilemaker took 2 hours 10 minutes on our test server. Today, with the changes that are merged plus the ones in review, the same build takes 23 minutes 34 seconds, uses 6.7× less CPU time and 109 GB less memory. The map is the same, except that some parks and lakes that master drops are back (more on that below).

Update (Oct 31): @systemed has merged #943, #945, #949, #951, #953, #956 and #964, and on Nov 1 #961. Thank you! 🎉

We wanted to share where that came from, show what's already in master and what's still waiting in open PRs, and invite you to try it on your own data.

The headline

Full planet (89 GB .osm.pbf)	Before #939	master today	master + open PRs
Wall time	2 h 10 min	43 min 06 s	23 min 34 s (5.5× faster)
CPU time (user + system)	190 h	54.5 h	28.4 h (6.7× less)
Peak memory (RSS)	270 GB	175 GB	161 GB (−109 GB)
Output size	75.47 GB	75.44 GB	75.48 GB

Same machine, same planet file, same command for all three: OpenMapTiles profile with coastline and landcover shapefiles, .pmtiles output, everything in memory, default thread count. AMD EPYC 9454P (48 cores / 96 threads), 503 GB RAM, Ubuntu 22.04, GCC 11.4. "Open PRs" means all open performance PRs together: #941, #943, #945, #949, #959, #961, #966, #968 and #970–#976. One run each; "Before #939" is the baseline run from #939 on the same planet file, the other two were run back to back today.

Not just for big servers

Most people don't run tilemaker on a 96-thread machine, so here is Germany limited to 8 threads, roughly a normal desktop:

Germany (4.9 GB .osm.pbf), 8 threads	Before #939	master today	master + open PRs
Wall time	5 min 31 s	2 min 54 s	2 min 05 s (2.6× faster)
Peak memory	14.2 GB	8.2 GB	8.3 GB (−42%)

(Average of 2 interleaved rounds.)

And if your machine doesn't have much RAM: with --store, Germany under an 8 GB memory limit went from "stopped after 69 minutes, unfinished" to 5 min 36 s (#943).

Where the time went

Phase	master	master + open PRs
Reading OSM data and shapefiles	6.7 min	5.4 min	−19%
Relations	2.1 min	41 s	−68%
Preparing tiles	26 s	8 s	−69%
Writing tiles	33.5 min	16.9 min	−50%

The flat stretches where master waits with 1–5 busy cores (shapefile loading, then a handful of giant relations) are gone, and tile writing takes half the time.

Reading, relations, preparing and writing all got faster, not just one phase. The biggest jumps came from no longer doing expensive work that wasn't needed: proving that already-valid polygons are valid, clipping polygons with a general-purpose algorithm when a rectangle clip is exact, and letting single huge relations (Lake Huron, Lake Nasser, Guri Reservoir…) hold up the threads waiting behind them.

The biggest single wins

Each line shows the gain in the part the change targets, and what it did to the whole planet build.

Change	What it speeds up	Before → after	Whole planet	Status
#939	everything (memory, Lua, stores, tiles)	2 h 10 → 38 min (Oct 14)	3.4× faster	✅ merged
#949 rectangle clip	clipping valid polygons to tiles	planet --store 65.7 → 49.1 min	−25% wall, −31% CPU	✅ merged
#961 validity precheck	proving polygons valid	planet --store 49.9 → 36.4 min	−27% wall, −33% CPU	✅ merged
#970 hole index	loading ocean shapefiles	95 s → 21 s (−78%)	−3.9%	🔍 in review
#974 + #975 relation geometry	building multipolygons	relations phase 105 s → 61 s (−42%); Lake Nasser 54 s → 3 s (−94%)	−2.4%	🔍 in review
#972 parallel finalizing	steps between reading and writing	70.5 s → 24.2 s (−66%)	−1.0%	🔍 in review
#971 malloc top pad	memory allocation	sys CPU −5.3%	−3.6%	🔍 in review
#973 store teardown	exit after --store runs	30 s → 8 s (−73%)	−1.3%	🔍 in review
#976 ring combining	repairing self-intersecting giants	Lake Huron 103 s → 25 s (−76%)	relations phase −15%	🔍 in review

And some that matter a lot for specific setups:

• perf: don't scan every tile coordinate when filtering z15+ tiles #959: tiles above zoom 14 no longer scan every possible tile coordinate. Monaco at maxzoom 17: 239 s → 0.47 s.
• perf: much faster combine_polygons_below (#868) #945 ✅: combine_polygons_below is about 4× faster (Luxembourg with OpenTopoMap 14.5 s → 5.8 s; the slowest single merge 9.9 s → 1.2 s).
• fix/perf(store): output matches in-memory, 12x faster when RAM is tight, 67x less disk writing, 3.2% less CPU #943 ✅: --store when RAM is tight: over 12× faster, 67× less disk writing, and the output now matches in-memory runs.
• perf: cap the mbtiles write buffer and hand out tile batches on demand #966 and perf: store generated points in an AppendVector #968: less memory for queued tile batches and generated points (planet anonymous memory 167 GB → 153 GB in the runs above).

Beyond speed

• Deterministic output. Since perf: 3.4x faster planet build (2h10 -> 39 min), less RAM, deterministic output #939, the tiles no longer depend on the thread count or on scheduling: the same input gives the same tiles on every run.
• MapLibre Tiles (MLT). Since Add MapLibre Tile (MLT) output #938, tilemaker can write MLT with "tile_format": "mlt", about 12% smaller than MVT.
• Fewer crashes, clearer errors. A batch of fix PRs (fix: bbox from pbf header west/south of 0; vector_layers metadata for integer fields and write_to layers #944, fix: drop polygon rings that collapse to zero area on the tile grid (#864) #946–fix: report JSON config syntax errors as line/column instead of byte offset (#867) #948, fix(mbtiles): keep widest zoom range and merge vector_layers when using --merge #950–fix: write settings.metadata.attribution to directory metadata.json (#781) #958, fix: hang when attribute_function returns an integer (C++20 regression from #938) #964, fix: relation_postscan_function erases tags of non-accepted child relations #965, fix: keep clipped slices of large polygons when filtering by area (#763) #967) turns std::terminate into readable error messages, fixes musl/BSD/NetBSD builds, keeps roundabouts that used to disappear when simplified, and more.

How we made sure the tiles are still right

Speed is worthless if the map changes. Every PR was checked with:

• Tile-by-tile comparison of whole extracts (area and length per layer, zoom and tag group) against master. The pure speed changes give identical tiles.
• Cross-checks against the original code on the whole planet. Every fast path that skips a Boost call was run side by side with that call: e.g. 65,637 shapefile polygons and 63 million OSM polygons for the validity precheck, 273,813 ring tests for the multipolygon index. 0 mismatches.
• Unit tests for the new code, several of which fail on the old code.
• A final check of all open PRs combined (the build measured above) on Germany: lines, points and labels are identical to master. Polygons only come back: 13,526 polygon groups that master drops from z14 tiles (whole parks, lakes and landuse areas) return with @TimSC's Fix dropped polygons in scaleRing; skip distant shapefile records... #941 (scaleRing dropped rings starting with a duplicated vertex), and about 800 more with perf: exact rectangle clip for valid polygons (planet 25% faster, 31% less CPU) #949 (Boost's clip failed when a vertex lay exactly on the tile edge). The only losses are tiny simplification differences at z10–12, under one millionth of the polygon area.

Try it, and help us land it

All of this is open source and ready for review. The fastest way to get it into a release is real-world testing:

1. Pick a PR that matters to you (or all of them), build it, and run it on your own extract and profile.
2. Compare it with master: time, memory, and whether your tiles look right.
3. Post your results in the PR, good or bad. A "works on my Lua profile with X" helps a lot.

A big thank you to @systemed for maintaining tilemaker and for reviewing and merging #939, to @TimSC for #941, which the rectangle clip builds on, and to everyone who reported the issues these PRs fix. We know a pile of PRs takes time to review; there's no rush, and we're happy to split, rebase or explain anything.

What's next

• Keeping all 96 threads busy while writing tiles: the chart above shows the tile phase still dipping to 45–90 busy cores.
• MLT output speed: MLT encoding is currently ~8% slower than MVT; we're profiling it now.
• Sharded stores by default: on big-RAM machines, --store without sharding is 16% faster; on small ones sharding wins. We'd like to discuss choosing this automatically.
• Your ideas: what's slow for you? Which setup should we measure next?

Happy tiling! 🗺️

_{The profiling, benchmarking and much of the code were done with the help of Claude Code; every number above comes from real runs on the hardware described.}

[cldellow]

Very nice work! Is there a branch with all the PRs on it, or should I manually apply them for testing?

I maintain a hobby project that renders North America using GitHub Actions. To fit within their runtime budget for free projects, I had to do some contortions -- sounds like those may all be non-issues now. (Lake Huron and some of the forests of the USA stand out in my memory as causes of grief, for example.)

[cldellow]

the run needed about 110 GB of free disk at its peak

Interesting. The old ubuntu-24 runners split the disk across two mount points. After using an action that tries to delete non-essential files from the image, the old runner would have a maximum of about 70 GB available on a single mount, if memory serves.

The new ubuntu-26 runners have only a single mount point, and have 122 GB after removing non-essential files. Very promising (but tight!) for a single pass -- I'll try that in the next few days, hopefully.

the bounding box in the header of Geofabrik's North America file spans all longitudes

I suspect I must have ran into that, too - looks like I clamp it to something more suitable.

Before/after

This project has been sitting on a shelf for a couple years, so my first order of business was to see if the pipeline still worked. Some things needed updating, e.g. the ubuntu-24 vs ubuntu-26 runner disk layouts had changed.

My before baseline was forked from 13b841d (~Oct 19, 2024), with an extra Lua function to support emitting GeoJSON files and another geometry-related commit (cldellow#4).

Both the before and after runs ran on a similarly old north-america.pbf -- a GeoFabrik export from Oct 3, 2024.

export workflow

This one is not yet a clear win - the before/after jobs took about the same amount of wall clock.

• before: https://github.com/hikeratlas/basemap/actions/runs/41251579411/job/124686042289
• after: https://github.com/hikeratlas/basemap/actions/runs/41267100768/job/124732069928

Digging in shows some interesting things, but requires a bit of context. The export workflow is an atypical use of tilemaker.

I wanted to produce an autosuggest database, with entries for POIs like:

{"name":"Tonquin Ridge", "park":"Mount Robson PP", "qrank":1, "country":"Canada", "kind":"peak", "state":"British Columbia", "osm_id":"3705066738", "lat":52.73483909351, "lon":-118.4769808}

There are probably purpose-built tools to transform a PBF into such a database. But alas, I'm not a GIS person. I knew tilemaker existed, so I reached for it and wrote a zindex profile that produced such a file.

The zindex profile needed some shapefiles in order to call FindIntersecting to decorate the output with metadata like the containing country, province, park, etc.

There are probably places a person could go online to get such shapefiles. But alas... out came tilemaker again and the city-parks, countries, parks and states profiles.

So, this workflow invokes tilemaker 5 times. The first 4 runs (city-parks, countries, parks, states) each produce a GeoJSON file. Those files are used as input to the final run (zindex), which also produces a JSON file. No tiles are written by any of these invocations.

In theory, the first 4 runs could be parallelized in the GH infrastructure, I just hadn't got around to it yet. This turns out to be useful, because it means all of these times are from the same machine:

Very nice speed up on the first four. Unfortunately, the gains are given back by the slowdown of zindex.

Comparing the relations section for the old zindex:

2026-10-31T23:42:26.1600435Z Block 221/235 
2026-10-31T23:42:26.5160731Z Block 223/235 
2026-10-31T23:42:28.1330678Z Block 225/235 
2026-10-31T23:42:28.8628979Z Block 227/235 
2026-10-31T23:42:29.7169717Z Block 229/235 
2026-10-31T23:42:29.8917285Z Block 231/235 
2026-10-31T23:46:30.5850320Z Block 233/235 (540388 ms)

vs the new one:

2026-11-01T03:16:24.8151452Z Block 221/235 
2026-11-01T03:16:26.0212008Z Block 223/235 
2026-11-01T03:16:28.2482168Z Block 225/235 
2026-11-01T03:16:29.4331450Z Block 227/235 
2026-11-01T03:16:30.6490258Z Block 229/235 
2026-11-01T03:16:51.2668486Z Block 231/235 
2026-11-01T03:29:03.8844793Z Block 233/235 (988550 ms)

it makes me think some straggler relation is acting as a long pole: the final blocks take 12 min (vs 4 min).

The zindex profile is the only one that calls Centroid('polylabel') and FindIntersecting, which might hint at what's going on there if it's reproducible. I'll see if it's reproducible and if I can add a bit of logging to identify where the time is spent

layers workflow

This is a clear win: from 46m 11s to 27m 36s. Some of that time isn't tilemaker time, so this understates the benefit.

• before: https://github.com/hikeratlas/basemap/actions/runs/41256621808
• after: https://github.com/hikeratlas/basemap/actions/runs/41269522414

This is a much more typical use of tilemaker. 9 Lua profiles go in, 9 mbtiles come out.

Most profiles are much faster. Most importantly, these do run in parallel in the GH infrastructure, and the long pole went from being the water profile (38 minutes) to now being the ways profile (20 minutes).

I suspect the buildings slowdown is just runner noise. GH runners are not very good. But they're free, so they're also very good.

The parks slowdown might be real. I'll kick off another run to see if it's reproducible. Claude thinks they slowed down particularly when writing tiles in Alaska/Yukon/BC. These are areas that have massive forest/national park polygons like this chonky boy. Although hm, that polygon is meant to be ignored by the parks profile. Still, Claude thinks the clip cache change in #939 might be relevant here.

Even allowing for the zindex export question and parks layer question, all of your work looks pretty awesome! tilemaker is now fast enough that the other parts of the pipeline (fetching the PBF from R2, merging the 9 mbtiles into a single mbtile, converting that file to pmtiles) are starting to be the long poles.

[dktue]

Thanks a lot for testing this so thoroughly on real pipelines! 46 min → 27.5 min for the layers workflow is great to hear, and the disk numbers for the ubuntu-26 runners are really useful.

zindex relations tail (4 → 12 min). That looks like one or a few relations getting much slower, so the quickest way in is to find out which. I've put up a debug branch that logs every relation taking at least 0.5 s: debug/relslow. It prints lines like

RELSLOW <shard> <relation id> <ms> <outer ways> <inner ways>

to stderr. If you could run zindex with it and post the slowest few, we can extract just those relations (osmium getid -r) and profile them here. That's how we found the Lake Huron problem. Centroid('polylabel') and FindIntersecting are good suspects, since none of our benchmarks use them.

A fresher branch. perf/all-open-prs is still based on master from Oct 28, so it lacks what @systemed merged since, including #964 (a hang in C++20 builds when an attribute_function returns an integer). perf/open-prs-on-master is today's master plus the remaining open performance PRs; debug/relslow sits on top of it. For further tests I'd use that one. perf/all-open-prs stays as it is, since the numbers in the post above refer to it.

parks. We'll try to reproduce that one ourselves with your parks profile and a current North America extract, compare a build from before #939 with today's, and report back here. Your theory about the clip cache is plausible: since #939 each batch of tiles clips a huge polygon from scratch instead of reusing another batch's clip. If it is that, it should show up clearly in a profile.

[dktue]

We reproduced the parks slowdown: you were right, and it's two separate steps.

Your parks slice on a current North America extract, set up like your job (--bbox -170,20,-48,75, --store --fast, .mbtiles, 4 CPUs, 16 GB; qrank stubbed, since the per-feature lookup costs the same in every build):

build	total	tile writing
before #939	517 s	~357 s
master right after #939	1,437 s	~1,342 s
current master + open perf PRs	2,309 s	~2,228 s

A bisect on Alaska/BC/Yukon (16 threads, 2 rounds) put the second step at #949, the rectangle clip, which made this profile another 52% slower. Its fast path re-read and re-clipped the whole polygon for every tile it covers, which hurts exactly the kind of huge forest and park polygons your slice is made of. #978 fixes that: 24% faster on Alaska/BC/Yukon with identical output, recovering about 70% of what #949 cost.

The first step is #939's clip cache, as you suspected: since #939, each batch of tiles clips a huge polygon from scratch instead of reusing a clip from another batch, which made the output independent of scheduling. Getting that speed back without losing determinism needs a shared cache of clips at a fixed zoom; that's the next, bigger change on our list.

Thanks for flagging it. Without your test this would have stayed hidden, since our OpenMapTiles benchmarks barely contain such polygons.

[cldellow]

zindex. The slowdown seems reproducible. A re-run with relslow is available here: https://github.com/hikeratlas/basemap/actions/runs/41316837801/job/124878566026

The culprit relation is indeed the Tongass National Forest:

  RELSLOW 0 6535292 1368460 29059 4

The parks profile skips this relation, but the zindex profile doesn't, which is why it only cropped up here. As a practical matter, I'm happy just skipping it, but it might be of interest to you if you want to do more analysis.

[dktue]

Thanks for running relslow, that pinned it down quickly.

Tongass isn't slow because of its geometry, it's a malloc setting. #939 calls mallopt(M_TRIM_THRESHOLD, 4 MB) at startup, and setting any malloc parameter freezes glibc's mmap threshold at 128 KB. Building Tongass runs Boost's is_valid on a polygon with 1.9 million points, which grows large temporary vectors over and over; with the threshold at 128 KB, every one of those becomes a fresh mmap whose pages the kernel has to zero first.

Tongass alone, relation phase on our server (4 threads):

	time
before #939	236 s
after #939	677 s
with the mmap threshold at 8 MB	232 s

polylabel turned out to be innocent: with or without Centroid('polylabel') the times are the same. We didn't test FindIntersecting, but the 3× already shows up with a profile that only writes the relation's area, so it isn't needed to explain your 12 minutes.

The fix is now in #971 (malloc settings): it sets the mmap threshold to 8 MB next to the top pad. Besides Tongass, that makes the planet another 2.6% faster for 0.85 GB more heap, and a planet on 4 CPUs with 16 GB is unaffected. Until it's merged, you can get the same effect without rebuilding by setting
GLIBC_TUNABLES=glibc.malloc.top_pad=16777216:glibc.malloc.mmap_threshold=8388608
for the tilemaker step (tilemaker then leaves malloc as configured).

[dktue]

Update: the planet on ordinary machines

The numbers above came from a 96-thread server. To see what this means for everyone else, we ran the full planet again on the same server, limited to what smaller machines have: CPUs pinned with taskset, RAM as a hard limit without swap (cgroup), --store on NVMe, .pmtiles output, OpenMapTiles profile with coastline and landcover. Build: current master plus the open performance PRs (perf/open-prs-on-master).

Machine	Planet build time	tilemaker's own memory (heap peak)	Disk needed at peak
Workstation: 16 cores, 64 GB RAM	54 min	12.7 GB	209 GB + 89 GB input
Desktop: 8 cores, 32 GB RAM	1 h 52 min	12.2 GB	209 GB + 89 GB input
Small VM: 4 cores, 16 GB RAM	5 h 15 min	12.0 GB	209 GB + 89 GB input

What this shows:

• A desktop PC builds the whole planet in under two hours, and a 16-core workstation in under an hour.
• The work scales with cores: 8 and 16 cores both needed about 13 CPU-hours in total, so twice the cores means half the time.
• About 16 GB of RAM is enough. tilemaker itself needs about 12–13 GB at most; everything above that is cache for the --store files, which makes it faster but isn't required. 8 GB is not enough: in a test with 8 cores and 8 GB the machine was busy paging the store in and out (more kernel time than tilemaker time) and reached only a quarter of the desktop's reading speed while the heap was still under 3 GB, so we stopped it.
• Disk is the main requirement: about 300 GB free in total (89 GB input plus 209 GB for the store files and the 76 GB output at the end).

With 16 GB, only about 4 GB is left to cache the store next to tilemaker's 12 GB, so the 4-core machine reads a lot from disk and is slower than its core count alone would suggest (17 instead of 13 CPU-hours). It works, but 32 GB is where it gets comfortable.