Low zoom waterbody rendering and move to water polygons

[imagico]

This issue is meant for planning and discussing implementation of low zoom waterbody rendering based on the recently introduced preprocessed data and combined with that the move to water polygons for the coastline.

I assume there might be reservations regarding the technical approach used but please consider this in terms of alternatives - there are not too many other ways to get waterbodies rendered at low zoom levels.

The preprocessed data is produced for z0-z6. There are two possibilities for the coastline beyond that:

• continue with the simplified polygons for z7-z9 and use the full detail data for z10+, that would require a decision w.r.t. Low-zoom ocean shapefile not rendered correctly on some setups #2101.
• use the full detail data for z7+ already - this would require assessment of the performance implications. Visually this looks fine in kosmtik although some more elaborate testing might not be a bad idea.

Apart from that if we go with this approach a decision would be necessary whether to use the raster or the polygon version. A demo implementation using the raster version is available on

https://github.com/imagico/openstreetmap-carto/tree/water-lowzoom-raster

This also implements the three color scheme for water but this is not meant to suggest these changes need to be connected - it just made sense for testing.

Overall this would solve/replace #1604, #1982, #2066, #2101, #2293

and would facilitate a solution for #1547 and #1781.

[nebulon42]

Regarding reservations against using preprocessed data I personally think that's ok if the data comes from OSM itself and can be periodically updated. Both is the case for the data at openstreetmapdata.com.

[pnorman]

This issue is meant for planning and discussing implementation of low zoom waterbody rendering based on the recently introduced preprocessed data and combined with that the move to water polygons for the coastline.

I'd keep them split. We already have #1982 for switching to ocean polygons, but we've never tracked down the bugs reported in #2101. If we had those solved we could revert the switch from ocean to land polygons.

For the reduced waterbodies, I'm not sure they're suitable for two reasons

1. How to generate the data isn't fully documented

2. It's not clear that this would work with different resolutions, e.g. retina tiles

[imagico]

We already have #1982 for switching to ocean polygons, but we've never tracked down the bugs reported in #2101.

I explained the connection - if we use the full detail water polygons for z7+ we don't have to use the simplified water polygons which IIRC is what caused troubles for some users. If we have already moved to water polygons when we implement this it is no problem either.

What we cannot really do is use this approach together with land polygons but i don't think anyone would want that.

How to generate the data isn't fully documented

Details are explained on

http://www.imagico.de/map/polygon_rendering.php

Tools used for that are all available:

https://github.com/imagico/osmium_waterareas
https://github.com/imagico/gray2vec
http://gdal.org/gdal_rasterize.html
http://gdal.org/gdalwarp.html
http://gdal.org/gdal_calc.html

The rest is just shell scripts putting this together. I might write a practical howto about this in the future but this does not have high priority at the moment.

In a nutshell:

• you extract water area from the planet using osmium_waterareas.
• you rasterize them at four zoom levels above your target resolution (i.e. 16 times the linear resolution) with gdal_rasterize.
• you resample to the target resolution (for raster output) or twice the target resolution (for vector output) using gdalwarp -r average.
• you generate the vector version using gray2vec.
• when you process multiple different water layers you do a shared pixel compensation analog to
  https://github.com/imagico/gray2vec/blob/master/Gray2Vec_Grid.cpp#L279 for the raster version using gdal_calc.py.

It's not clear that this would work with different resolutions, e.g. retina tiles

You need to have and use the data source for your target resolution. For double resolution tiles this is just one zoom level offset. The lack of possibilities to do this automatically for any data sources other than PostGIS makes this somewhat awkward of course. If you use the polygon version and import this in PostGIS (which is not really an option for us here) you can select this automatically based on the z attribute and !pixel_width!/!pixel_height!.

[pnorman]

Details are explained on

http://www.imagico.de/map/polygon_rendering.php

I had another read and the only tool mentioned from that list is gray2vec. I'll have a look at what you've described in this ticket to see if that provides enough information for someone else to go from a planet file to the same output.

You need to have and use the data source for your target resolution. For double resolution tiles this is just one zoom level offset. The lack of possibilities to do this automatically for any data sources other than PostGIS makes this somewhat awkward of course. If you use the polygon version and import this in PostGIS (which is not really an option for us here) you can select this automatically based on the z attribute and !pixel_width!/!pixel_height!.

It needs to be automatic, i.e. we don't know what resolutions will be generated when generating the Mapnik XML, so the same XML has to work for normal tiles, retina tiles, print, and arbitrary Mapnik scale factors.

[imagico]

It needs to be automatic, i.e. we don't know what resolutions will be generated when generating the Mapnik XML, so the same XML has to work for normal tiles, retina tiles, print, and arbitrary Mapnik scale factors.

Then i predict we won't get low zoom waterbodies because it does not seem likely Mapnik will be extended in that regard.

Note though that even using the z6 version for all z0 to z6 would - while it is not the intended use - probably still look better than any other workaround that has been suggested for this problem in the past.

[kocio-pl]

With all waterbodies rendered now on low zoom (as a result of merging #2873), is there still a reason to switch to water polygons?

[pnorman]

With all waterbodies rendered now on low zoom (as a result of merging #2873), is there still a reason to switch to water polygons?

Yes - it's needed for some interactions between wetlands and water, as well as generally being preferred.

In fact, the reasons remain exactly the same. This was never about great lakes and other non-ocean waterbodies.

[kocio-pl]

This was never about great lakes and other non-ocean waterbodies.

Maybe it was not supposed to be initially, but - as far as I understand - it's proposed by @imagico as a superior solution for low zoom inland water rendering, currently in the context of #2952:

http://www.openstreetmap.org/user/imagico/diary/42975#comment40536

but, honestly, it's hard for me to believe it's happening:

Then i predict we won't get low zoom waterbodies because it does not seem likely Mapnik will be extended in that regard.

Was there any try to contact Mapnik team, like ticket for example? What could we do instead?

Note though that even using the z6 version for all z0 to z6 would - while it is not the intended use - probably still look better than any other workaround that has been suggested for this problem in the past.

It was proposed year ago - what do you plan to implement it and what do you need from the others to make it real?

[jeisenbe]

using the z6 version for all z0 to z6 would - while it is not the intended use - probably still look better than any other workaround that has been suggested for this problem in the past.

@imagico, would this work for different resolutions, eg exporting at 4x from Kosmtik, or producing PDFs for printing?

My understanding is that the preprocessed data will look better and render faster than the current rendering.

Perhaps it won't look perfect when rendered at higher resolutions than intended, but won't it still be an improvement on the current map?

[imagico]

@imagico, would this work for different resolutions, eg exporting at 4x from Kosmtik, or producing PDFs for printing?

No - at least not any better than using a raster image would.

Note however that rendering maps for print at z0 to z6 is not really a practically relevant use case. Still of course that it is not possible to dynamically select a data source based on rendering resolution with layer types other than PostGIS is a serious problem.

Note technically the reduced waterbody data is kind of superseded by the technique used for the low zoom demo: http://blog.imagico.de/on-basic-small-scale-landcover-rendering/ - this offers both better efficiency (and therefore additional zoom levels) and allows use for both waterbodies and landcover rendering. Unfortunately this is only in a demo state and so far no one showed interest in investing in extending that to a practically usable framework. The difficult integration into normal map rendering does not help with that of course.

Back in 2016 when this issue was opened the direction for low zoom styling was fairly clear and this issue was to discuss a practical option for this. In the meantime practical changes have moved the low zoom styling into a completely different direction - but there is not really a clear path visible where this is supposed to go (at least not to me). Right now it more or less looks to me like we have to fill the tiles at these zoom levels somehow but don't really care. So i am not really sure if pursuing the technical ideas discussed here is still of relevance.

It would be great if someone would step up and really take interest in low zoom level design here again - but as you can see the technical difficulties are anything but trivial. Apart from the waterbodies and landcover the boundaries are a big issue as well.

[jeisenbe]

I probably should have clarified that I'm mainly trying to find a solution that will allow us to switch to ocean polygons, since that is a necessary prerequisite for a number of issues.

rendering maps for print at z0 to z6 is not really a practically relevant use case

True! The current rendering looks quite poor when exported at these levels

Apart from the waterbodies and landcover the boundaries are a big issue as well.

Yes, I've noticed that the very low zoom border from Natural Earth do not match well with our coastlines, and they are rather jagged for z3, and for exports at higher resolutions from z1 and z2. I would be great to have generalized or simplified admin_level 2 and 4 borders for low zoom.

the reduced waterbody data is kind of superseded by the technique used for the low zoom demo

But this data is not available, is it? I'd like a solution that we can use to switch to ocean polygons, soon. It looks like all the the maintainers agree that ocean polygons are the right solution.

I've considered trying the generalized water polygons for z0 to z8, water-polygons-generalized-3857
from http://openstreetmapdata.com/data/generalized-coastlines

This gets us down to a zoom level where it is reasonable to use water-polygons-split-3857, thus there is no need to use simplified-water-polygons-complete-3857 which is the cause of issue #2101 (if I understand correctly).

It helps that I love the way the generalized coastlines look!

I've tried exporting the coastlines at 2x and 4x resolution, and it works fine.

Indonesia at z4
With water-polygons-generalized for ocean polygons and line for coastline

Without blue line for coastline

Current rendering z4

Maluku (Spice Islands) z9
New rendering with water-polygons-split-3857:
(More detail, but still renders quickly, even on my 2012 laptop)

Current rendering (with simplified-land-polygons-complete-3857:
(Looks choppy, like a vector map, esp. the coast of the small, round islands on the west side)

Can we consider the use of the generalized coastlines at low zoom levels? The current rendering with the smoothing and outline has a similar effect, but it takes much longer to render and does not work as well.

[jeisenbe]

Wales at z8 with fully generalized coastline from water-polygons-generalized:

With simplified polygons (as in current rendering at z8):

Versus with full data (water-polygons-split-3857), currently used at z10 and higher only:

[jeisenbe]

I've tested the reduced water polygons now. It's true that these do not render properly at 2x resolution (eg for retina tiles) when used for z0 to z6 as intended:

z2 Current rendering with simplified ocean polygons (and only very large lake polygons), and outline for coastline, screenshot from openstreetmap.org on 9-Feb-2019:

z2 Rendering with polygons-reduced-z2 shapefiles for ocean, lakes and rivers:

z1 exported at 2x (double resolution; eg retina tiles) from polygons-reduced-z1 shapefiles:

z0 exported at 4x resolution, rendered from polygons-reduced-z0 shapefiles:

This shows how the "half-toning" style screen of polygons becomes visible when rendered at much higher resolution than intended

But we can fix the exports by using the shapefiles at 1/2 resolution, that is, use the shape file designed for z1 at z0 and so on. This allows exporting or rendering z0 to z5 at 2x resolution with perfect quality, and the standard resolution rendering is still 90% the same:
z2 Rendering with polygons-reduced-z3 shapefiles for ocean, lakes and rivers: (Looks almost the same as the same area above, rendered from the "proper" z2 shape file)

**Comparison, zoomed-in, of Labrador, Canada at z4, rendered with z4 shapefiles (left) and with z5 shapefiles (right):

z1 exported at 2x resolution now looks like this: (identical to z2 rendering above)

It's even possible to use the z2 shape file for z0, and so on until z4. Rendering at standard resolution is still quite fast and the quality is similar (pixels with very small lakes or narrow rivers tend to be less blue than they would be), and now it is possible to export at 2x, 3x, 4x and even 5x resolution from Kosmtik with good results:

z2 Rendering with polygons-reduced-z4 shapefiles

z0 exported at 4x resolution, from z2 shapefiles: This is high enough resolution for printing

z4 rendering comparison of lakes and river in Finland and Russia, with z4 shapefiles vs z6 shape-files. Very similar, except a few pixels are less blue:

It's also possible to export at 3x and even 5x resolution with similar results, using this technique, so I believe it would address pnorman's concern.

[imagico]

The simplified ocean polygons sample does not seem right - the ocean polygons are split, i see no way you can create the rendering shown based on these.

Regarding use of the reduced files at zoom levels below the intended ones - while that works in principle you very soon get errors larger than the ones you'd get when rendering the original data in mapnik - and this effect will get stronger with increasing zoom level. And as said you have no way to automatically select the shapefile that is best to use.

[jeisenbe]

The simplified ocean polygons sample does not seem right - the ocean polygons are split, i see no way you can create the rendering shown based on these.

Are you referring to the z8 rendering of Wales from this comment a few days ago?

I may have uploaded the wrong file. I've rendered it again and replaced it. Wales z8 with simplified-water-polygons-complete:

Regarding use of the reduced files at zoom levels below the intended ones - while that works in principle you very soon get errors larger than the ones you'd get when rendering the original data in mapnik - and this effect will get stronger with increasing zoom level. And as said you have no way to automatically select the shapefile that is best to use.

In these tests I used 5 shapefiles (z2, z3, z4, z5, and z6) to render the 5 zoom levels from z0 to z4. I wouldn't want to use them at more than 2 zoom levels below where they were intended. The side-by-side comparisons show that most pixels are very similar even when off by 1 or 2 zoom levels. This is much better than what we see when rendering landcover at z5 and z6 from full data in Mapnik, where areas made up of many small polygons disappear entirely. And the large lakes are the same.

you have no way to automatically select the shapefile that is best to use.

This is a shame, it's true. Would you have time to open an issue on the Mapnik repository to discuss this? I'm not sure how to describe the technical requirements. If this were fixed we could use the raster files for water (and landcover, a la French style), and this would be both fast and small file size.

But the reasonable use case for this style at z0 to z5 is for standard resolution and retina tiles. Being able to render two resolutions, standard and 2x, is enough, and the reduced polygons work well for this.

[jeisenbe]

I have an idea of a way that the reduced water polygons could be modified so that they would work perfectly when rendered at two zoom levels, eg z1 and z0, which allows rendering standard and retina tiles with the same shape file.

The small "half toning" polygons should be aligned so each set of 4 pixels (2 high by 2 wide) have one shared polygon at the center, aligned so that each pixel in the lower zoom level will also have the proper percentage covered by water. This sometimes happens with the current algorithm, but not always.

Here are some examples (from the current z0 lakes file). It should look like the one in the very center:

And there are two here:

I think this could be done with only a slight increase in size of the file

[imagico]

Yes, that would be possible - would of course require someone to actually implement it.

Note though this would only be of advantage if you render on these particular grids - for print use you usually can't rely on a specific grid alignment.

What i have not tried so far is how it looks like of you re-sample the raster version to a coarser grid. Mapnik does not support average resampling - and even if it did that might be distorted by gamma correction. But none the less testing with what is available might be worth trying. Technically this is the simpler and more straight away approach.

[jeisenbe]

What i have not tried so far is how it looks like of you re-sample the raster version to a coarser grid. Mapnik does not support average resampling - and even if it did that might be distorted by gamma correction. But none the less testing with what is available might be worth trying. Technically this is the simpler and more straight away approach.

I tried it, and the results are not perfect. It gives a somewhat pixelated appearance:

z1 Oceans from raster files

z1 Oceans with raster files, used 1 zoom level lower (that is, this is rendered from the z2 raster file):

Zoomed in; from original raster file: (UPDATED)

Zoomed in, from second image, with z2 raster file used at z1: