Use fixed polygon clip cache level

[Symmetricity]

Use fixed polygon clip cache level

This PR was AI generated.

Summary

This makes multipolygon clip-cache reuse deterministic by caching polygon clips
at tilemaker's existing z6 object-clustering level, then writing z0..z6 tiles
before z7+ tiles.

Higher zoom polygon tiles previously reused whichever lower-zoom cached clip
happened to be present. In a multithreaded run, that ancestor could depend on
worker scheduling. Clipping an already clipped polygon through different
ancestor paths can produce different encoded polygon geometry, even when the
source data and command line are unchanged.

Linestring clip-cache behavior is unchanged.

Background

The existing clip cache is intentional and performance-sensitive. It was added
as part of #612, which extracted ClipCache and reduced memory use by lazily
reconstructing way-backed geometries. PR #607 explains the intended cache model:
when writing higher zooms, tilemaker prefers clipping a cached lower-zoom
geometry over clipping the original geometry again. That avoids repeated work
for very large polygons.

This PR keeps that optimization but makes the parent level deterministic for
polygons. Rather than searching z-1, z-2, and so on until any cached ancestor is
found, multipolygons now use the z6 ancestor. z6 is already tilemaker's object
clustering level, so the change follows an existing locality boundary instead
of adding a new one.

Investigation

This was found while investigating generated-tile semantic differences. A
separate fix exposed a repeatability issue where normal multithreaded runs could
emit different polygon coordinates for the same tile. The first observed local
case was:

• tile: z11/1077/1329
• layer: water
• tags: class=lake
• source: OSM relation 9404214, a large Rhine water multipolygon with inner
  rings
• symptom: matching feature counts and tags, but different polygon coordinates
  between repeated runs

Additional checks:

• running with --threads=1 was stable
• temporarily disabling polygon ClipCache reads made repeated normal
  multithreaded runs semantically match again
• revalidating clipped polygons after spike removal did not fix the scheduling
  sensitivity
• a full zoom-barrier scheduler made local repeats deterministic, but was too
  expensive on a medium fixture

The narrowed root cause is shared parent-tile clip-cache reuse. A higher zoom
tile could reuse a z10 parent in one run and a z6 parent in another run,
depending on which worker populated the cache first.

Implementation

This PR adds an optional fixed-zoom mode to ClipCache.

For multipolygons only:

• cache entries are written only at min(basezoom, CLUSTER_ZOOM)
• cache lookups for higher zooms map directly to that fixed parent tile
• z0..z6 tiles are written before z7+ tiles so the fixed parent cache entries
  are available before child tiles are processed

The existing ancestor-search behavior remains available and is still used for
linestrings.

Results

Small fixture / CI

The fix was tested stacked on the currently submitted PR changes in fork CI run
25960612652.

After rerunning a known intermittent Windows crash in tile generation, the run
passed native generated-tile verification:

• macOS 14 Makefile
• macOS latest Makefile
• Ubuntu 22.04 CMake
• Ubuntu 22.04 Makefile
• Windows CMake

Repeat outputs and cross-runner outputs matched semantically. Raw MBTiles and
PMTiles bytes still differed, but decoded MVT content matched.

The clean, isolated branch was also checked locally against the Liechtenstein
fixture using upstream resources/config-openmaptiles.json and
resources/process-openmaptiles.lua.

That isolated check is useful, but it also shows why this PR should be read as
one part of the broader determinism work:

• upstream master repeat output differed in 15 decoded MBTiles layers:
  1 water geometry difference and 14 poi differences
• the clean fixed-z6 branch removed the observed z11 water repeat difference,
  but still showed unrelated poi differences because the isolated branch does
  not include the Lua POI-order determinism fix
• upstream-vs-fixed output differed in polygon geometry as expected:
  204 decoded MBTiles layer differences, mostly landcover, landuse,
  water, and park

Two visual checks were prepared from that fixture:

• Rhine water polygon, OSM relation 9404214:
  https://www.openstreetmap.org/relation/9404214#map=15/47.16020/9.49300
  • upstream repeat output was not stable for z11/1077/718 XYZ
  • the fixed-z6 output matched the shorter 19-point outline seen in one
    upstream repeat, instead of the upstream run that added an extra clipped
    corner point
• high-zoom water clipping near Triesen:
  https://www.openstreetmap.org/#map=17/47.07210/9.48670
  • fixed-z6 changes the clipped edge geometry compared with upstream dynamic
    parent reuse
  • the change is an expected output change from making the parent clip source
    deterministic

Larger local fixture

A local Austria fixture was used for a larger timing check:

• input: Austria extract, 754 MB
• profile: resources/config-openmaptiles.json and
  resources/process-openmaptiles.lua
• output: PMTiles
• command output redirected to log files for both runs
• source PBF effectively warm in the page cache for both runs
• semantic verification was not run for the timing comparison

Results:

Build	Wall time	User time	System time	CPU	Max RSS
upstream master	3:53.30	540.47s	466.40s	431%	3,537,824 KB
fixed z6 cache	3:33.61	552.21s	398.39s	445%	3,461,776 KB

On this fixture, the fixed-z6 branch was about 8.4% faster wall-clock, used
about 5.6% less total CPU time, and used about 2.1% less peak RSS. Treat these
as fixture-specific results rather than a guarantee.

An earlier larger PMTiles repeat test on the same Austria fixture produced
47,742 addressed tiles. The fixed-z6 outputs differed at the raw tile-byte
level, but the semantic verifier checked 43,124 raw-different PMTiles tile
pairs and confirmed matching decoded content.

Planet Estimate

This has not been measured on a planet build.

PR #612 reported a planet build at 67m51s and 42 GB RAM on a 48-core Hetzner
CCX63. If the Austria wall-time ratio transferred directly, that would estimate
roughly 62 minutes for the same planet build. Applying the Austria RSS ratio
directly would estimate roughly 41 GB RAM.

Those numbers are only directional:

• planet output has a different mix of very large polygons and high-zoom tiles
• memory is dominated by node/way/object stores, not only the clip cache
• the z6 phase boundary may help or hurt depending on workload balance
• the local Austria run used a different machine and a much smaller input

The safer expectation is that this should be approximately performance-neutral
to modestly faster on large polygon-heavy outputs, with memory roughly neutral.

Expected Improvement

The main improvement is deterministic decoded tile content for multipolygon
outputs that previously depended on worker scheduling.

This should make repeated tilemaker runs less sensitive to thread timing and
platform scheduling. It also keeps the monster-polygon clip cache rather than
disabling it.

Possible Regressions

This can change polygon geometry compared with previous output because clipping
is not perfectly associative:

clip(original, z6 parent), then clip(child)

can differ slightly from:

clip(original, z8 parent), then clip(child)

Expected visual differences, if any, should be localized to polygon edges,
especially water, landuse, landcover, and building polygons that cross tile or
z6-cluster boundaries.

This does not make generated archives byte-for-byte deterministic. Archive
ordering, compression, and raw tile bytes can still differ while decoded MVT
content matches.

There is also a scheduling tradeoff: z0..z6 tiles now complete before z7+ tiles
are posted. The tested Austria fixture did not show a slowdown, but a workload
with unusually expensive low-zoom/z6 tiles could see reduced parallelism during
that phase.

Related Issues and PRs

• some memory and concurrency improvements #612 introduced the current ClipCache structure and explains the memory and
  concurrency goals behind the surrounding code.
• Clipping monster polygons 2 #607 introduced the monster-polygon clipping cache behavior this PR preserves.
• Retry with boost::geometry::intersection if fast_clip fails #597 and Use faster bbox-specialised clipping algorithm #482 provide relevant clipping-performance and clipping-correctness
  context.
• Invalid polygons after clipping at tile boundaries #697 reports invalid polygons after clipping at tile boundaries. This PR is
  related to polygon clipping, but it should not be claimed as a fix for Invalid polygons after clipping at tile boundaries #697
  without targeted reproduction against that issue's fixture.

No directly matching upstream issue or PR was found for the scheduling-sensitive
deterministic-output bug.

Testing

• cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo
• cmake --build build -j2
• fork CI run 25960612652, stacked test branch, native generated-tile
  semantic verification passed after rerunning the known intermittent Windows
  tile-generation crash
• local Austria upstream/fixed-z6 timing comparison, PMTiles output, no
  semantic verification for the timing run

[systemed]

Thanks!

Oddly I get a big slowdown running this on a Europe extract. Current master:

User time (seconds): 71641.03
System time (seconds): 1164.42
Percent of CPU this job got: 1462%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22:58

With this applied:

User time (seconds): 108798.76
System time (seconds): 1287.37
Percent of CPU this job got: 1487%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:03:19

HP DL380 G7, 144GB RAM, running with --store on SSD. I'll try it again at some point to check this wasn't just a one-off.

[Symmetricity]

I suspect it's probably because I wasn't testing with a lot of low zoom level fixtures (nor did I include the extra ne polys). Biggest I did was Austria, so wouldn't have had to deal with 0-6 much.

This code gates on needing all of 0-6 to be generated before proceeding to higher zoom levels, so the larger fixtures take longer to generate at those low zoom levels and so it spends a lot of time waiting.

I'll have a think about how to resolve this. Might be this fix isn't the right way to achieve what I'm trying to achieve (determinism).

[Symmetricity]

So essentially this PR made the cache worthless, which explains the slow down:

Variant	Wall	User CPU	RSS	Polygon cache hits	Fallback calls	Fallback time
Upstream-style dynamic cache	0:54.86	352.28s	4,482,948 KB	2,435,236 / 8,511,699 (28.61%)	110,688	15,281.8 ms
PR #892 fixed z6 cache	0:58.57	375.56s	4,279,580 KB	30,115 / 8,510,953 (0.35%)	167,994	42,417.7 ms

I'm investigating another candidate looks only slightly worse than the current cache and preserves determinism. If you'd like me to close this PR and submit a new one, happy to do that, since it will be a significantly different solution. If you're also not happy with a slight performance hit to guarantee determinism, let me know.

[Symmetricity]

I'm going to cancel this PR. The current method doesn't work.
I might pivot to accept the slight semantic differences during real runtime, but add an argument that can be called to essentially disable the cache for the purposes of the semantic tests only.