Gaps and overlaps in tiles on high zoom with -zg

[jdeboer-geoplan]

Hi, I've got a small problem with the -zg option, there might be a flag option I'm missing that would sort it out. When I run with -zg it guesses that I want to go down to z8

Choosing a maxzoom of -z6 for features about 5147 feet (1569 meters) apart
Choosing a maxzoom of -z8 for resolution of about 1281 feet (390 meters) within features

However when I zoom in to level 14 it has lots of holes and overlaps, if I generate it to level 14 it looks fine all the way to 20.

Output

If I use -ab the results are much better but still not perfect:
Output

I'd hoped -pS might help but although the tile file is slightly bigger I can't see any difference in output.

I can provide the geoJSON but I feel this must be a known problem and I'm just failing to find the right flag combo to sort it out. Obviously I can get my tiles by generating all the way to z14 but I'm attracted to the much faster creation and smaller tile file size of only going to z8 if that's possible with my data.

Thanks

[flippmoke]

This is due to simplification more then likely. If two polygons share an edge and then are simplified in a very slightly different way there is a good change you get output like what you see above. You might try playing with these settings: https://github.com/mapbox/tippecanoe#line-and-polygon-simplification

[jdeboer-geoplan]

thanks @flippmoke I tried -ps and pS but it didn't help, still get the gaps. What value could I use in the --simplification= to have a better scale?

[pathmapper]

@jdeboer-geoplan did you try the following?

-ab or --detect-shared-borders

From the docs:

"... detect borders that are shared between multiple polygons and simplify them identically in each polygon."

[jdeboer-geoplan]

Hi @pathmapper yup, that dramatically improved things as you can see in the difference between pic1 and pic2. But there are still gaps

[pathmapper]

@jdeboer-geoplan ahh, sorry, I missed that.

[e-n-f]

Can you provide a copy of the GeoJSON so I can try to figure out what is going wrong? If you are using --detect-shared-borders or especially --no-line-simplification, the discrepancies might be being introduced in post-simplification polygon cleanup.

[jdeboer-geoplan]

Absolutely. I'll attache a really cut down copy of my geoJson. One thing I've noticed is that -zg goes higher with less data. e.g
Whole uk - Choosing a maxzoom of -z8 for resolution of about 1281 feet (390 meters) within features
Small part of london - Choosing a maxzoom of -z9 for features about 509 feet (156 meters) apart
two streets - Choosing a maxzoom of -z10 for resolution of about 475 feet (145 meters) within features

[jdeboer-geoplan]

1998_UK_GEO_GSSP_cutdown.zip

[e-n-f]

-zg works from whatever the geographic density is in the input data, so if your smaller sample has denser or more detailed geometries than the full data, it will choose a higher zoom level based on that. (London in particular may also throw off the calculation, since the Prime Meridian cuts through it and complicates the tile math.)

Thanks for the data. I'll take a look now.

[e-n-f]

It does look like the gaps are being introduced during polygon cleanup with Wagyu. I don't see them with cleanup turned off.

I think during snap rounding, line segments that pass within a pixel of another edge node are being snapped to that node, which doesn't happen in the other polygon that also shares that line segment as an edge, because the other node is not part of that polygon.

[e-n-f]

So maybe the answer is a global snap-rounding pass in which all polygon edges get snapped to the nodes of any other features that they come close to.

[jdeboer-geoplan]

Hi @ericfischer thanks for taking a look, on the -zg thing the geometry was the same, just I'd removed a bunch of them, is that worth raising as a separate issue to these holes from polygon cleanup?

[e-n-f]

-zg is based on the average precision of the features, not the highest precision, so it is not necessarily surprising that it changes when you change the sample. But if you think it is doing a bad job of choosing, please do file another ticket with a data file and what you think it should have chosen for that data.

[e-n-f]

However what this suggests is that what you are looking for is for -zg to choose a zoom level high enough that no node within a feature is closer than one tile unit from any line segment within the same feature.

[e-n-f]

Alternately, an --extend-zooms-if-still-simplifying would probably be more practical than trying to predict up front what will get snap-rounded.

[e-n-f]

Hmm, that doesn't really work, because Wagyu is still doing some amount of insertion of snap-rounded points into your geometries all the way to zoom level 14 (but apparently not in zooms 11 and 12, which is even stranger).

[e-n-f]

Even just checking for snap rounding that happens within the tile proper (as opposed to in the buffer), continuing until there is no snapping keeps tiling far too deeply in cases like the Natural Earth countries, which are still doing some snapping at z11.

[jdeboer-geoplan]

wow, this is a can of worms. What snap rounding algo is wagyu using? I assume it's not idempotent? I like the idea of --extend-zooms-if-still-simplifying

[e-n-f]

It uses a "hot pixels" approach where two segments that cross through the same pixel get snapped together. It unfortunately is not idempotent, so rerunning it on its own output can produce additional changes.

I feel like --extend-zooms-if-still-simplifying is producing too many false positives to be practical as a general approach, and that a global snap-rounding pass is more likely to be useful.

[jdeboer-geoplan]

Hi, well I'm learning loads reading up about the exciting array of different snap rounding algos out there, good stuff but not necessarily of any use here. I think that the global snap rounding pass sounds good, that will presumably mean that there maybe more nodes on each polygon and so they might be a little bigger?

[e-n-f]

Yes, they'll get bigger because there will be more nodes.

I'm actually much more worried about time and memory than about tile size, especially in the future (mapbox/vector-tile-spec#104) when features can be rejoined across tile boundaries, in which case a large polygon will need to be snapped against every feature that it touches in every tile, not just in one.

[Plantain]

I think I'm seeing the same behaviour, and I have a JSON file that reproduces it quite nicely with tippecanoe -P --coalesce-smallest-as-needed --detect-shared-borders --extend-zooms-if-still-dropping -z8 -pw


1700.zip

tippecanoe is multiple orders of magnitude faster than our previous tile generator, so time and memory isn't a problem for us yet! :) :) :)

[Plantain]

"It does look like the gaps are being introduced during polygon cleanup with Wagyu. I don't see them with cleanup turned off."

@ericfischer, is the Wagyu polygon cleanup controlled by a flag? Otherwise can you share how you disabled it so we can work around this in the meantime?

[e-n-f]

It is not controlled by a flag because the vector tile spec requires that polygon geometries be valid, and I don't have a way to guarantee that without the use of Wagyu. But if you want to live dangerously, you can comment out the

geoms[g] = clean_or_clip_poly(geoms[g], 0, 0, false);

line here:

tippecanoe/tile.cpp

Line 496 in d64ac19

geoms[g] = clean_or_clip_poly(geoms[g], 0, 0, false);

[e-n-f]

And please note that --coalesce-smallest-as-needed will only be useful with the use of Wagyu, since its entire value relies on being able to union together adjacent polygons, which happens in the course of the clean-or-clip call to Wagyu.

[Plantain]

I think (at least in my example) it is unrelated to polygon cleanup then, as disabling it produces almost no difference. Left: without cleanup, Right: with cleanup.

./tippecanoe/tippecanoe --detect-shared-borders -z8 -pw
1800.gz

[Plantain]

For comparison, mapshaper shows a lot of the whitespace is 'real', but the weird overlapped artifacts aren't there.

[e-n-f]

Interesting. Can you share a sample of your data so I can investigate?

[Plantain]

Attached above - 1800.gz.

[Plantain]

The zoom in particular is from this location:

[e-n-f]

For my reference: the area in question is near 9.5/43.9681/-112.0949.

I'm pretty sure the geometric errors are in your original data, which is only given to 3 decimal digits of precision (about 350 feet). If I convert your polygon rings to individual LineStrings and tile them considerably deeper than the display resolution, it is clear that the contours do not match up in many places.

If you are not seeing overlaps in mapshaper, it is because it is drawing the shapes opaquely rather than with partial transparency, which hides the overlaps.

I think whatever process is creating these polygons is doing its own line simplification, and that to get the results you want you will need to stop it from simplifying or get it to use a higher precision during simplification.

[Plantain]

You were right - there was some line simplification going on in the geojson library generating it.

[stonetip]

In many cases, simplifying adjacent features leads to overlaps and gaps if the topology isn't good in the first place, i.e. adjacent features don't share vertices. This needs to be corrected in your source data before exporting to GeoJSON and tiles. For example, you can use tools like v.clean in QGIS to try to automatically clean up topology.

[briantran37]

So ive done what @ericfischer suggested but im getting a "found unexpected character" error when trying to merge the two mbtiles

Running Command: ~/bin/tippecanoe -r1 -z6 -o low_zoom.mbtiles --force --detect-shared-borders --extend-zooms-if-still-dropping --drop-fraction-as-needed --drop-densest-as-needed geo_json.geojson
converting low zoom mbtiles 
For layer 0, using name "low_zoom"
28749 features, 1401338 bytes of geometry, 8 bytes of separate metadata, 167 bytes of string pool
  99.9%  6/19/23  
Running Command: ~/bin/tippecanoe -r1 -Z7 -o high_zoom.mbtiles --force --detect-shared-borders --extend-zooms-if-still-dropping --drop-fraction-as-needed --drop-densest-as-needed geo_json.geojson
converting high zoom mbtiles 
For layer 0, using name "high_zoom"
28749 features, 1682167 bytes of geometry, 13809 bytes of separate metadata, 167 bytes of string pool
  99.9%  14/4949/5999  
Running Command: ~/bin/tile-join -r1 -o product_vector_tile.mbtiles --force --layer=product_vector_tile low_zoom.mbtiles high_zoom.mbtiles
joining mbtiles 
low_zoom.mbtiles:1: Found unexpected character
high_zoom.mbtiles:1: Found unexpected character
0 features, 10 bytes of geometry, 8 bytes of separate metadata, 0 bytes of string pool
Did not read any valid geometries

My vector tile at high zoom level is pixalated and wanting to merge two vector tiles to remedy that. What do you suggest to resolve the error?