Lake Ontario does not render at zoom 5 or below

[andrewpmk]

Lake Ontario does not render at zoom 5 or below. It looks fine at zoom 6 or above. I think that this is a bug in the OSM map style because I think that the data is fine.

[imagico]

This is nothing special about Lake Ontario - no lakes are rendered at z<6. This is not a bug but still it would be good to also show lakes at the lower zooms of course. It is difficult to do that in an efficient and good looking way though. See also #754.

[andrewpmk]

Lake Erie, Huron, Superior and Michigan render properly. Why doesn't Lake Ontario?

[imagico]

Those are mapped as coastline - see

http://wiki.openstreetmap.org/wiki/Tag:natural%3Dcoastline#What_about_lakes.3F
http://www.openstreetmap.org/changeset/28625595

[andrewpmk]

Maybe a short term workaround would be to render lakes at a somewhat lower
zoom e.g. at zoom 4 or higher. At zoom 3 the "Toronto" label covers where
Lake Ontario should be so the lack of Lake Ontario isn't that noticeable.
At zoom 5 currently Lake Ontario looks absolutely awful.

On Mon, Jun 8, 2015 at 12:38 PM, imagico notifications@github.com wrote:

Those are mapped as coastline - see

http://wiki.openstreetmap.org/wiki/Tag:natural%3Dcoastline#What_about_lakes.3F
http://www.openstreetmap.org/changeset/28625595

—
Reply to this email directly or view it on GitHub
#1604 (comment)
.

[matkoniecz]

link: http://www.openstreetmap.org/?mlat=44.024&mlon=-78.926#map=6/44.024/-78.926

@pnorman - Would it be a significant performance impact to render lakes also on z5, maybe also on z4?

[pnorman]

@pnorman - Would it be a significant performance impact to render lakes also on z5, maybe also on z4?

Probably. It'd mean another sequential scan, and I'm not sure how big the set of results would be. That would depend on pixel size and way area filtering.

[StyXman]

Why not simply make a shape file with generalized versions of big lakes and use that for rendering in low zooms? They could easily be extracted from OSM data, the shapefile would not be so big, and you already depend on external shapfiles anyways.

[imagico]

If you only show the larger lakes there is little gain compared to a plain way_area threshold - both performance wise and quality wise. And a high size cutoff is a bad idea leading to ugly results, see #754.

If you however want to produce a generalized rendering based on all water areas, even the small ones, you are dealing with a huge volume of data at the start, even if it boils down to a small file in the end (data size ratio is in the order of 1:100). This is not something you can do as a background job during the night.

Shameless plug: @joto and myself have been considering this as something that would be nice to add to openstreetmapdata.com. This is surely something that would be useful for a lot of applications but if this is going to happen depends on financing development work and processing resources.

[pnorman]

I'm against adding dependencies on additional preprocessed sources. In most stylesheets, I'd have no problems, but then again, in most stylesheets I'd probably use natural earth at zoom 5.

[imagico]

Fair enough - note however ultimately this would mean that you'd never get a really good quality rendering at the lower zooms. AFAIK @gravitystorm has repeatedly stated that one of his aims is to improve the look of the map in the starting configuration when you newly visit openstreetmap.org - which happens to be z=5 at the moment:

http://www.openstreetmap.org/#map=5/51.500/-0.100

Improving this depends a lot on use of preprocessed data - not so much for the lakes but quite definitely for boundaries and placenames.

[utack]

FYI: Even more great lakes have been changed from coastline to lakes in the dataset recently, which caused some confusion:
https://www.reddit.com/r/openstreetmap/comments/3z7w9d/where_are_the_great_lakes/

[fche]

Bummer ... I spent a few days sketching coastline ways around Lake Ontario, but now we've moved backwards.

[redsteakraw]

Well I have a hack that may work, just render the name= tags for those lakes as a lake. You keep the performance but render the important lakes.

[planemad]

The Great Lakes are too large to ignore at lower zoom levels. It seems really odd for such huge lakes to appear out of nowhere at z6.

[imagico]

IMO this is not any worse than the Greenland and Antarctic ice appearing out of nowhere at the same zoom level.

As discussed previously a good solution to this problem will require a preprocessing approach. @pnorman has voiced his opinion against such approach here. In #754 the poor and lazy solution of size based filtering has been (IMO rightfully) rejected because of its arbitrariness.

Even if a practically usable preprocessing solution is currently not available - and i don't know if and when i will have the time to work on such - see above in #1604 (comment) - it would probably be helpful if the maintainers here can agree on how they see this in principle, that means either closing this as wontfix, reopening #754 or keeping this open with the aim of solving it with preprocessed data.

[fche]

In #754 the poor and lazy solution of size based filtering has been (IMO rightfully)
rejected because of its arbitrariness.

This could be one of those cases where the perfect is the enemy of the good. There will always be arbitrary thresholds and heuristics, even in the preprocessing solutions. That's not inherently bad.

[planemad]

this is not any worse than the Greenland and Antarctic ice appearing out of nowhere at the same zoom level.

@imagico land/water relationship is more fundamental to how we view the face of this world than ice. This is more akin to showing the Caspian Sea as land when zoomed out, it causes a very basic discrepancy on the map.

[imagico]

The problems of size based filtering have been discussed in depth, no real need to repeat this here. The advantages of a preprocessing approach less so but you can take my word on it that when done well this is much less arbitrary - if you want to you can essentially try to mimic the appearence of rendering the full detail data minus the AGG rendering artefacts that would otherwise be dominating at these scales.

Probably obvious but the likeliness of a good but somewhat complex solution being developed of course significantly diminishes if a bad solution is accepted as good enough - so using your words: good enough is the enemy of great.

Also in this style you should always keep in mind the effect such decisions have on mapping activities. way_area filtering with a larger than sub-pixel threshold would put significant incentive on mapping for the renderer by merging waterbodies to push the area to get your pet lake show up at a lower zoom level.

@planemad - that is obviously a question of subjective preferences, lets focus on the issue itself and not on if it is more important than other things.

[StyXman]

... not to mention Rio de la Plata's wide mouth 'becoming' dry land between Argentina and Uruguay...

[gravitystorm]

I'd be happy to consider a pre-processing step for water areas at low zoom. But if we're going to do it, then it needs to be done properly, namely:

• Dissolving adjacent polygons (no simple way_area check). As @imagico says, way_area filtering incentivises poor mapping
• Handles all water body types. I don't like people feeling obliged to tag large lakes as coastline purely because of rendering problems
• Is frequently updated
• Is open source

I'd also like if the approach can be easily adapted to other low zoom features, like forests. These suffer from way_area based filtering when mapped in detail. When a large forest is mapped (correctly) as many polygons with small road-width-sized gaps between them, naive way_area filtering makes the forest disappear. But the main reason for easy adaptation is to ensure we don't end up with X shapefile created though method A, Y shapefile by method B, Z by C etc, with different software or update frequencies or whatnot.

[redsteakraw]

I may have a fix, lets tag all large inland lakes with lake=great_lake this would include lake Victoria and lake Baikal. The criteria for great_lake is that it is visible from space, is large enough for ferries and fishing boats and is large enough to impact the weather. All of these lakes meet the criteria and this would make it easier to match this tag to render while giving a useful semantic differentiation. Openstreetmap-carto can simply render lake=great_lake as a lake at the lower zoom levels.

[pnorman]

I may have a fix, lets tag all large inland lakes

Please discuss this on a tagging list or forum, not the style issue tracker.

[kocio-pl]

Do we have any standard tools to simplify? BTW - the problem with memory didn't occur in other places on Earth. I wonder also if partial index for water is working for all the zoom levels?

I'm still puzzled why z3 does not show me the water.

[matthijsmelissen]

I'm still puzzled why z3 does not show me the water.

Did you adapt the minzoom in the .mml, in this line?

[kocio-pl]

Thanks! I've made more mistakes than just this one while drafting, but it helped me to find the solution.

It seems that the big hidden problem is the subpixel accuracy in our SQL code. I don't know what the factor of 0.01 is needed for, but with 1 the rendering is similar, if not the same. This change however allows us to skip all the artificial limits and show every water area from z0 without any other optimizations (like shape simplification). My system just stopped exhausting the memory and became responsive even in the worst cases.

If you agree that getting rid of sub-pixel accuracy makes sense, I propose to:

1. Prepare a PR from my testing code and replace the Start showing water labels earlier #2864.
2. Prepare a PR fixing it with other layers too (namely: landcover-low-zoom, landcover, landcover-area-symbols, buildings, buildings-major and nature-reserve-boundaries), because it's probably performance bottleneck on all the medium and high zoom levels.

[matthijsmelissen]

A demo-rendering would be very welcome for that.

[kocio-pl]

I had some strange problem with exporting from Kosmtik and I have enough of investigating for today, so I will try later. @rrzefox It would be great if you could test the performance with my current code and update the testing layer, so we could compare the results.

[kocio-pl]

Just a screenshot of a z6 near Great Lakes with a lot of small water areas (I'm using water tags extract to keep database import time manageable) - click to see the full scale:

[rrzefox]

Which code branch did you use?

That was simply 4.3.0 with changed zoomlevel 6->3 in water.mss and adapted zoomlevel 4->3 in project.mml

@rrzefox It would be great if you could test the performance with my current code and update the testing layer, so we could compare the results.

This has been deployed now (for reference: this has 1 as the pixel factor).
With that, Z 0-3 each took 400 seconds to render. On Z 4, the worst tile was 490 seconds, on Z 5 256 seconds (but most are very fast).
Note that only Z 0-7 have been rerendered fully, on other zoom levels you may need to dirty. All Zoom levels should show the new rendering by now (30.09. 11:50)

[kocio-pl]

Thanks a lot!

Direct comparison is available (as always) here:
http://bl.ocks.org/math1985/raw/af7a602c222dbf1ff1a2c0d84ed755b7/#3.00/18.42/1.85

Water labels are different due to #2845, which is now dropped. Some new areas lack the label because of a previous deployment, so they should be rerendered.

I hope we will have a Mapnik release with a label placement fix soon (mapnik/mapnik#3771 (comment)) and some other improvements are possible (mapnik/mapnik#3550), so we would get even better water areas.

[matthijsmelissen]

I see some differences in Northern Poland on z6 between the 0,01 and 1 pixel rendering. I prefer the (new) 1 pixel rendering. In other areas or zoom levels I don't notice differences immediately.

So it would be great if we can have a performance improvement and rendering improvement in one.

I would be very interested to see what would be the effect of applying the 1 pixel change to other landuse too.

[rrzefox]

Water labels are different due to #2845, which is now dropped.

This should not be visible on any rerendered tiles anymore, I removed that patch.

[matthijsmelissen]

Would it not be easier to remove the old filecache, then? Especially when spotting small differences like 0.01 vs 1px on the high zoom levels, it is really hard to see if files are old or new.

[rrzefox]

This is a tileserver that is actually used by people, so I cannot just remove all cached tiles, because that would make it unusable :) But it does already rerender tiles in the background (currently at Z8) and will continue to do so, but that'll simply take a long time. So you can already look at Z0-7 now, and if you need a specific tile in another zoomlevel, feel free to dirty that - otherwise just wait, it will have rerendered all tiles eventually.

[kocio-pl]

This is a tileserver that is actually used by people

Great! Could you share some real-life statistics then to compare the old and new rendering time, memory and disc IO usage?

Tweaking other layers should be straightforward and even more interesting, because it deals with landcovers and buildings (+nature reserves) and we have a lot of them!

[kocio-pl]

I see some differences in Northern Poland on z6 between the 0,01 and 1 pixel rendering. '

When I zoom Firefox to 300% (max available) I see some smaller lakes missing in this area (south of Koszalin):

http://bl.ocks.org/math1985/raw/af7a602c222dbf1ff1a2c0d84ed755b7/#9.00/53.7320/16.6433

[kocio-pl]

BTW - @rrzefox you have some problems with fonts used, especially here in China:

http://bl.ocks.org/math1985/raw/af7a602c222dbf1ff1a2c0d84ed755b7/#7.00/25.414/99.502

[kocio-pl]

The main issue is performance. Perhaps now we have the requirement that all rendered features should at least be 0,1 pixel (if I remember correctly), performance is not so much of an issue anymore. This should be tested.

I guess adding this limit was an improvement comparing to lack of any area measuring before, and 0.01 was probably used just in case to not break anything.

I think that we could test the database performance change alone (real life statistics are more complex), but how can we do it? Directly from Postgres by some script maybe for given location or via Mapnik? I would like to check it on my system.

[kocio-pl]

I have another performance related question: is it possible to make Postgres read from 2 instances of a "gis" database in parallel? Would it require any changes to the osm-carto code and what speedup I could expect?

[rrzefox]

I get the feeling this discussion is somewhat drifting off-topic, so I'll try to keep it short. Feel free to skip reading everything but the last sentence.

Great! Could you share some real-life statistics then to compare the old and new rendering time, memory and disc IO usage?

That's the problem with a production server: Tile rendering times differ a lot, depending on things like: how many and which other rendering requests run at the same time, or even what happens to be in the disk cache. The only thing you can really see is if a change alters the rendering times significantly, e.g. as in this case, where the rendering time for Z0 increases by a factor of >100 (without this change, the one Z0 tile that exists for the planet rendered in 0.5 seconds!). The same goes for memory usage and all that. If you want numbers that are more precise than "it's in the same order of magnitude", it's better to measure that on an extra instance only used for benchmarking.

BTW - @rrzefox you have some problems with fonts used, especially here in China

This is running the standard Ubuntu 16.04 versions of the fonts with noto-emoji installed manually. My best guess is that the OSM tileservers use a newer version of the fonts, or these are renderd<->tirex differences. As I don't care about china much (that is not where this tileserver is usually used), I prefer to stick with the fonts packaged by Ubuntu, even if they may be outdated.

Last but not least: all tiles on the "demo" should now have rerendered, so compare away.

[rrzefox]

There are lots of differences visible with the glaciers at the border around here: http://bl.ocks.org/math1985/raw/af7a602c222dbf1ff1a2c0d84ed755b7/#8.00/43.000/42.387 (also in lower zoom levels). However the differences disappear as expected when you zoom in, so it's really just getting rid of one-pixel-noise in the lower zoom levels and I don't think that's a bad thing.

[rrzefox]

I was surprised to learn where the Caspian Sea is located, according to the current rendering.

[matkoniecz]

@rrzefox This is probably result of #1465

[kocio-pl]

I was surprised to learn where the Caspian Sea is located, according to the current rendering.

Strange - this is error on z5+, but on z4 it's placed properly:

http://bl.ocks.org/math1985/raw/af7a602c222dbf1ff1a2c0d84ed755b7/#4.00/40.49/-277.64

Anyway, some future Mapnik release should fix it soon.