Question on performance

[liebrand]

Hi there, I had a qq on performance. I have four geojson files, which are all really quite small (1.5MB, 7.4MB, 3.4MB, 1.5MB). But I am trying to use tippecanoe on them, and it's been going for almost two hours now (granted it's running on my macbook, but still)?

The output tells me the four layers contain:
3295 features, 4149968 bytes of geometry, 945005 bytes of metadata

Note: I did run tippecanoe with -z17 but it's only at 82.8% at the moment, and according to the output it's still on zoom level 14. ie even if I hadn't changed the zoom level, I dont think it would be done by now.

The mbtiles it's producing so far is already 2GB. Which is rather impressive since the input in total is just under 15mb :-)

Is this expected??

[mourner]

What type of GeoJSON is this? Are there polygons that cover a huge area?

[liebrand]

They're converted S-57 nautical charts, and yes some of them cover a vast area in the pacific. After raising the issue I figured that might be the case, as I guess it has to generate a lot of tiles.

So I guess I have a follow up / different question in that case. Is there any way to parallelize this work over multiple machines? Perhaps by restricting each to produce the tiles for just a specific area. Then once all workers are done, use something like tile-join to combine the results?

[e-n-f]

Sorry I missed this before. Tippecanoe will use multiple CPUs on a single machine, but can't currently parallelize across a cluster.

What you must be running into is that the polygons that cover the ocean need to generate a clipped polygon in every single tile across the entire ocean, so you are generating thousands and thousands of copies of the same tile that contains just a square clipped from that polygon.

The only thing that can help is just to use a lower max zoom level. Zoom level 17 is tremendously detailed even for something that's not trying to cover an entire ocean.

[e-n-f]

You also have a pretty old version of tippecanoe, since the status message doesn't mention the string pool. If you upgrade it will at least take advantage of multiple CPUs on the same machine.

[liebrand]

Thanks for the info, I'll update tonight and see if that helps. But in theory, should I be able to run 17 different instances of tippecanoe for each zoom level and then join the resulting tiles together? I know it's not really clustering, but it might help a little.

Note: zoom level 17 is too detailed for the pacific, but for harbors it's needed. Since those come from different original nautical charts, I could change the max zoom per chart as well, and run tippecanoe for each.

But that leaves me with the same result: multiple mbtiles. Can tile-join join those together?

[mourner]

Note: zoom level 17 is too detailed for the pacific, but for harbors it's needed. Since those come from different original nautical charts, I could change the max zoom per chart as well, and run tippecanoe for each.

That would help, and you could also try doing a smaller max zoom but preserving the z17 detail (-d switch).

[e-n-f]

@liebrand You should be able to merge multiple non-conflicting tilesets with tile-join after the changes that I just merged into master.

You really need 3-inch resolution in the harbors? Wow, that is some very good data. I agree with @mourner that if you really need the detail, -d will be much more efficient than making lots of tiny tiles. The only water polygons I've tried are the coarse ones from Natural Earth, but there you can even get away with

tippecanoe -o water.mbtiles -z5 -d21 ne_10m_ocean.json

for a very few, very high res, tiles.

[liebrand]

Hmmmm 3-inch?? Maybe I've been misunderstanding the zoom levels, or perhaps when mapbox requests tiles, but I'm seeing it request z17 level tiles much sooner than that!

Either which way, the -d option is going to really help me. Most of my polygons are simple enough that they will scale up very well anyway, so it should be fine to reduce the max zoom level drastically.

As for "non-confilcting" tilesets; i'll do some testing. But will it error if there are conflicts? or will it produce odd results? As long as I run the instances at different zoom levels, then there shouldn't be conflicts, right?

[liebrand]

ps. I just tried to upgrade but seems like I already have the latest?
1 [18:46:15] $ brew upgrade tippecanoe
Error: tippecanoe 1.2.0 already installed

[liebrand]

Ok, I'm clearly not understanding something... when I run with -z5 and -d21 I get a lot of warnings about some tiles being > 500k. More importantly though, the map just fails to render anything above zoom level 5. Which is like half of california? Once I zoom in further, it request tiles for zoom level 6 (which 404) and thus it draws nothing at that point?

[mourner]

@liebrand are you using Mapbox GL and Studio? Try explicitly setting max zoom to 5. It will then overzoom it properly.

[liebrand]

I am using Mapbox GL but not studio.... I have a hunch I know whats going on. I made my tileserver respond with an empty response for missing tiles (to avoid a gazillion 404 errors in the client console). I wonder if that is making mapbox GL think that there is a tile there, its just an empty tile. Thus when I zoom in beyond level5 it draws the empty level 6 tile and thus there is no overzooming...

Lemme fiddle with this a bit tonight.

Having looked a little closer, I think the max zoom that i'm interested in is level 16. So I could probably create tiles up to level 10 or whatever and then overzoom. (One other reason to go higher than level 5 is to keep the tile size smaller, which might work better for mobile)

[e-n-f]

When I say zoom level 17 gets you 3-inch resolution, what I mean is that a zoom level 17 tile with a detail level of 12 (4096 units per tile) means that the 360 degrees around the equator are divided 2^(17 + 12) times. That means one unit in the tile is 0.00000067 degrees. One foot is approximately 0.00000274 degrees, so a tile unit is 0.00000067 / 0.00000274 feet, which is about 0.2445 feet, which is about 3 inches. There is probably a more sensible way to do this math but that's the one that's wired into my brain by now.

[e-n-f]

And it sounds like your data is too detailed to work at -z5, which is not hard to believe, since Natural Earth's geometries are pretty coarse. But it still seems like you can probably get away with a lot less than 17.

I should warn you not to use high -d with GL, though, since anything higher than -d13 will make the tile coordinates big enough that they will overflow when rendering and draw wrong.

I still urge you to figure out what the actual resolution of your data is, and to use the lowest -z that will get you that resolution with overzooming.

[e-n-f]

The current tippecanoe in homebrew is 1.3.1, I think. You should probably brew update and then upgrade tippecanoe. I should make a new brew target for 1.4 since there are a few additional improvements in there.

[liebrand]

Ok, just an update to this.
1- indeed a brew update worked; although I've noticed with other mapbox tools that using pip is probably smarter since the brew repo tends to be out of date

2- if I change my tile server (running tilelive) to produce 404 for higher zoom levels, then mapboxGL correctly overzooms using the lower level zoom tiles. So that does indeed allow me to get away with using a max zoom level of 12. But this produces a rather crappy experience for the zoom levels between 12 and 16. Mainly:

a- if the user zooms in from 12 to 16, then it correctly overzooms. But if you now start to pan around, you inevitably will get to a tile for which the level 12 was not loaded. And thus it produces missing tiles

b- this problem is exasperated by the fact that some tiles seem to get unloaded as you pan. So as you pan around, you get to areas for which it doesn't have tiles. But then panning back to where you were now also produces missing tiles.

c- it "feels" like theres quite a bit of UI jank for all the 404s. Bizarrely it doesn't pan scroll as smoothly as it did when it did retrieve tiles. This is kinda odd, because you'd think it'd be doing less. But I haven't profiled the fps yet; it's just a feeling

All in all, although it produces a LOT of tiles, I think producing them up to max zoom level 16 gives the smoothest experience. I just need to find a good way to parallelize the work to produce and upload those tiles so that I can automate that nicely...

[mourner]

@liebrand thanks for the follow-up! I think it's worth reporting the tile loading issues along with minimal JSFiddle test cases to https://github.com/mapbox/mapbox-gl-js.

[e-n-f]

Closing this because I don't think there's any actionable tippecanoe work here.