Option needed to automatically move tags from outer way to multipolygon relation (Load OSM File tool)

[mboeringa]

Hi,

This is a problem that really needs to be addressed and have some kind of solution if my OSM Renderer for ArcGIS is really to have value.

Currently, the Load OSM File tool treads OSM multipolygons without any tags beside type=multipolygon as a sort of invalid objects, and sets osmSupportingElement = _yes_ for these, and hence makes them disappear from the map, as the Definition Queries exclude them.

In case the related outer ways of the same multipolygons do have tags, the outer ways get set to osmSupportingElement = _no_, and will display, but won't properly represent the multipolygon geometry they are actually part of, as, for example, the inner ways are not "cookie-cut" from the outer way polygons as in the multipolygon set to osmSupportingElement=yes (which does have the right geometry, just no useful tags).

Actually this type of multipolygon relation, with tags on its outer way and not on the multipolygon relation, is outdated, and it is now recommended practice to tag the multipolygon instead. However, there are still innumerable features, including many prominent ones like museum buildings, tagged this way.

Thus, a lot of features are excluded from the map, they "disappear"...

While this functionality probably closely follows the requirements for editing the data in ArcGIS, this is a major headache for rendering the data. One partial solution I have come up with and currently use in my renderer, as the loss of especially many buildings with inner courts and waterways let to pretty much unusable render results, is to allow building and polygon waterway features with osmSupportingElement=yes to be drawn.

But this is nothing more than a very rudimentary solution, as the outer ways do not have the same geometry, as they are lacking the inner ways or holes that exist in the multipolygon definition, which causes other issues, like features in inner courts of building complexes not showing up, as covered by the false building polygon. And there are still features missing, as I do not want to implement this rudimentary solution for all classes of features, I just implemented it for buildings and waterways, as the rendering results where otherwise unacceptable.

It is something I have started to realize over the past year, but of course the standard renderings done on the OpenStreetMap website, need to deal with the same issue. I didn't see this properly documented up to now, but it became clear that osm2pqsql, actually must be moving tags from the outer ways to their corresponding multipolygon relation. There is no other way the default renderings can display what they display.

I recently saw this confirmed on the osm2pgsql issue tracker on Github, where one of the users talks about this option, see below link and the multiple remarks related to ways, relations and multipolygons and especially the "superseded" remark. Actually, in the link below the OP desires to do the opposite, add relation tags to ways, but in this context, it is also described how the relations are supplied with way tags:

osm2pgsql-dev/osm2pgsql#230

I really think the Load OSM File tool needs to be enhanced with a similar - user selectable - option for moving these tags automatically from the outer ways to the multipolygon relation during the load process and the creation of the geodatabase, and removing the outer ways or setting them to osmSupportingElement=yes to hide them, something similar seems to be done in the osm2pgsql load process based on the comment in the linked issue above.

It will seriously improve render results, and make them comparable to the main OSM website slippy layers. In addition, potential users of the toolbox, will expect similar render results to OSM, and not be satisfied with the current "make-shift" solution.

What are ESRI's thoughts on all of this?

As I see it, there is no other way to get proper rendering of many multipolygon features, instead of waiting an undefined amount of time until everything gets fixed manually in OSM by painstakingly moving tags from outer ways to the corresponding multipolygon relation for each and every feature having this problem. Clearly, the OSM style and osm2pgsql maintainers, did not have such patience... so we probably shouldn't too?

Last note:

Moving of outer ways should only happen under the following two conditions:

• All of the outer ways share exactly the same tags (multipolygons can of course potentially have multiple outer ways, that may be tagged differently in OSM). Naturally, the multipolygon relation can only have one set of tags, so this requirement should be checked. If the outer ways have different tags, than these multipolygons will unfortunately need to be ignored, as there is no automatic way to deal with this and resolve the problem.
• The multipolygon relation must be of type=multipolygon or any other type that is supported as polygon, e.g. boundary, and may have no additional tags (or at least no conflicting tags that differ from the outer ways)

Marco

[ThomasEmge]

Marco,

if you could provide a sample relation where you see this issue I'll take a closer look.

Here is the quick rundown:
Tags like 'inner' and 'outer' are not considered as they are optional and non-conclusive. The challenge for relations is that they are a mixed bag. It can be a mixture lines and closed lines, an array of lines that in aggregate close on themselves, or an array of relations that contain relations that will eventually form a closed line, (did I mention that I really would like polygons as an explicit geometry type ;).... Most super-relations will be ignored and go into the relation table. For relations tagged with 'multipolygon' the code is making a reasonable effort in constructing a polygon, based on the geometry and then in attempting to harmonize the attributes.

• Thomas

[mboeringa]

if you could provide a sample relation where you see this issue I'll take a closer look.

Please see the multipolygon that I also referenced in the other issue:

Wooded area near "Schloss Nymphenburg", Munich, Germany, it is this one:

http://www.openstreetmap.org/relation/335052

Tags like 'inner' and 'outer' are not considered as they are optional and non-conclusive.

I don't really think these are considered "optional" for multipolygons. The most important OSM editors, JOSM and ID, both have rather distinct support for these tags and defining multipolygons relations based on this information.

In fact, JOSM even shows clear "donut holes" for this wooded area in its editor windows, signifying it is considering inner and outer way roles in multipolygon relations. Remember, JOSM is not using an osm2pgsql "render" database extract stored in PostGIS, where the multipolygons are already dealt with through the import process, it is dealing with tagging database more or less directly (hstore in Postgresql?) through the APIs developed for OSM.

At least, that is my basic understanding of it based upon all I read up to now...

By the way, here is an image of the area in JOSM:

[mboeringa]

@ThomasEmge

You may find this also an interesting read. It is about another render pipeline pgmapcss, that also refers to the different types of multipolygons, and how tags may be transfered from the outer ways to multipolygon relations:

https://github.com/plepe/pgmapcss/blob/master/doc/database.md

Especially see the section about multipolygon support...

[mboeringa]

@ThomasEmge ,

I have tested the latest dlls now, and it seems almost OK now, the Kronprinzengarten and other holes are correctly cut from the relation, but you need to set the outer way:

http://www.openstreetmap.org/way/16580198

of the relation with ID:

http://www.openstreetmap.org/relation/335052

to osmSupportingElement=yes. This needs to happen in all these cases, the outer way shouldn't be shown if the relation is added to the polygon table. Because now I have two overlapping wooded areas: one for the correctly build up relation, and one for its outer way, both set to osmSupportingElement=no

[mboeringa]

Thomas,

The latest dlls seem fine. I have attached the latest rendering results, both in my ArcGIS renderer, and the default openstreetmap-carto rendering. This looks really good now, but I want to start testing on other areas now. I am especially interested in issues with buildings and waterways, that I have seen popping up frequently in some areas. These issues should largely be solved now.

[ThomasEmge]

Looks very good, keep us posted.

[mboeringa]

Looks very good, keep us posted.

I will, may take some time though, as I intend to render bigger regions now. My longest render session took up to a month... (entire Netherlands, 23.6 GB *.osm file, but I won't start with that one).

[mboeringa]

Thomas,

I did encounter one more issue I think you can easily solve:

If there is a multipolygon which has outer ways, where the outer ways are tagged themselves, but share exactly the same tag set as the multipolygon, except for the multipolygon having the obligatory extra tag type=multipolygon, than all of the outer ways sharing the same tags as the multipolygon should be set to osmSupportingElement=yes.

E.g.

Multipolygon:
type=multipolygon
landuse=forest

Outer way 1:
landuse=forest

Outer way 2:
landuse=forest

Set the Outer way 1 and 2 to osmSupportingElement=yes in these cases. This prevents outer ways from overlapping the multipolygon needlessly.

[mboeringa]

By the way, here is another screenshot showing the dramatic effect of the better Multipolygon support in the toolbox. Left is the "old" rendering, right is from the latest toolbox. Note the large stretches of forest and agricultural land that are missing in the left image, but are now properly displayed in the right one.

[ThomasEmge]

Looks much better...not sure why no one had noticed the issue before this fall. Yes, I think your proposed outer way behavior should be doable. If you have a testing relation for verification I would greatly appreciate it.

[mboeringa]

Looks much better...not sure why no one had noticed the issue before this fall.

I certainly noticed it from the moment I really seriously started to look at the OpenStreetMap project and started to work on my renderer at the end of 2013. I think there may have been two reasons:

• Most people involved in OpenStreetMap may be focusing on the road system and routing, and simple Points Of Interest, so polygons are of lesser concern, while they are of utmost importance for a "topographic" style renderer as the one I have developed.
• The OpenStreetMap data model complexity (especially the concept of Multipolygon "relations"), and basically being a line based datasource like "old-style" CAD drawings, combined with unfamiliarity of seasoned ArcGIS users with OSM, may simply have let people to unjustly concluding that the data they got out of the Load OSM File tool was "it", so nothing better could be obtained using ArcGIS to read and process this data, and they would have to "make do with it".

And maybe third, I am also quite notorious for sniffing out bugs I think... ;)

Yes, I think your proposed outer way behavior should be doable. If you have a testing relation for
verification I would greatly appreciate it.

This multipolygon relation should help. Both the relation itself, and 3 of the 5 outer ways are tagged landuse=forest:

Relation 52270
http://www.openstreetmap.org/relation/52270#map=11/52.5716/12.1653

E.g. outer way 252613409
http://www.openstreetmap.org/way/252613409

Please note this area is actually a mess... I discovered looking at the data on the OSM website, that there is a second multipolygon relation stacked on top of the above one. It is this one, and uses partly the same, but also slightly different, inner and outer member ways:

Relation 3344305
http://www.openstreetmap.org/relation/3344305#map=11/52.6233/12.1852

[mboeringa]

Thomas,

Here is also a screenshot of relation 52270. As you can see, it renders properly with the natural=heath as inner ways (purple).

I did need to remove the relation 3344305 for showing this though, by setting a Definition Query to exclude it, as it overlaps the the other relation. This is not an issue with the Load OSM File tool however, as I already wrote that this area is a mess and badly tagged without proper multipolygons and having overlapping features. But this last reported problem of outer way and multipolygon double tagging seems OK now.

[mboeringa]

@ThomasEmge ,

Looking at some of my latest rendering results, I noticed a problem with a multipolygon not being rendered in the Belgium dataset.

Looking at the data, it seems pretty "normal". There is a multipolygon with tags for landuse=forest and deprecated wood=mixed, and there are 4 outer ways, 2 of which have been tagged with _exactly_ the same two tags as the multipolygon, the other 2 carry no tags.

Of course, in case a multipolygon has tags from itself, any tags on its outer ways should be ignored and not influence the tags for the multipolygon itself (although there is the corner case of all outer ways sharing one or more identical tags, which in that case might be transferred to the multipolygon if they don't exist on the multipolygon relation yet). Outer ways with tags, should probably be set to osmSupportingElement=yes if they have _exactly_ the same set of tags as the multipolygon or no tags at all, which is the case here.

I don't know if any of my rambling in the paragraph above is of relevance to this issue. Anyway, in the first image below, the multipolygon is visible on the OpenStreetMap website. The second image has a cross where the forest area is missing.

xxx EDIT xxx:
Some more observations:

• In the Polygon feature class as created by the Load OSM File tool, when selecting based on OSMID, I do see all 4 outer ways as records in the table, with a setting of osmSupportingElement=yes.
• However, if i search for the multipolygon relation ID (163334) in the OSMID column, nothing is found, so it appears no multipolygon was generated.

[ThomasEmge]

Hmmm, here is what I am seeing after loading the feature.

Can you please check if ID 163334 does exist in the relation table?

[mboeringa]

Hi Thomas,

That looks good!

Really weird. I now checked, but I can't find OSMID 163334 in the Relation table, nor in the Polygon or even the Polyline table. A simple query based on the ID returns nothing (as the rendering already showed).

As I wrote in the previous post, I _DO_ see all the members of the relation in the Polygon table. All are set to osmSupportingElement=yes, suggesting the multipolygon relation was properly processed.

I will attempt a second load of the geodatabase using the Load OSM File tool and see if the issue persists.

*** EDIT ***
One vague feeling that I have, is that, because I reloaded the database already one time using your new DLLs last week, I may have forgotten to clean up the relation and revision table, but maybe cleaned up only the Feature Classes. Anyway, for the new attempt, just to be sure, I will delete the entire geodatabase and create a new one.

[mboeringa]

@ThomasEmge : a small update: I reloaded the database from my existing Geofabrik 'Belgium' extract. I still do not see the multipolygon '163334' in my Polygon feature class. In the image below, you see the results of an attribute query:

OSMID IN ('163334','36451508','234746235','89997075','89796417','36918339')

Notice that all inner and outer ways are found, and set to osmSupportingElement=yes (as they should be).

This is weird. The extract is from the 2nd of December, the multipolygon itself hasn't been touched for two years in OSM, as witness-able by the images posted in my first report here in this issue.

One last thing I can do is to download a new extract from Geofabrik, which I will do, and load it again. In the mean time, could you also try to download the Belgium dataset, and do the same, and report your results?

[mboeringa]

@ThomasEmge . I now loaded a newly downloaded extract of Geofabrik as well. Unfortunately, the results are the same: no ID 163334 anywhere in the tables, while the five member ways are in the database as supporting elements.

[ThomasEmge]

I know - it is very strange, I stepped through the loading process in the debugger and everything looked good. I'll do some more investigation but it should be there, and it is when loaded as the only feature (see the screen capture above).

[mboeringa]

I'll do some more investigation but it should be there, and it is when loaded as the only feature (see the screen capture above).

Yes, that was clear and looked good. The only thing I can think of that is that one of the member ways is involved in other relations, and that that somehow causes the multipolygon to not being added. Something like that would at least explain why importing not just this feature, but a bigger extract with "context" causes the feature to dissappear.

But anyway, this is just a wild guess, and I leave it to you to figure it out.

The only other thing I can think of is a relation with the unsolved "Schloss Bellevue" issue (#72 (comment) - node with similar tag as way causing the way to disappear), but that may as well be unrelated...

[mboeringa]

@ThomasEmge , any progress on this issue?

[ThomasEmge]

Unfortunately not yet. This one is really puzzling as I am trying to find a more compact test case and I haven't found one. It will take a little longer to find a resolution but it is certainly on my agenda to fix.