Make use of node tags like 'highway' or 'crossing'

[easbar]

So far we keep node tags only for barrier nodes, but with a few small changes we can also provide the tags of all nodes belonging to the current edge to the tag parsers. This allows us to write encoded values that consider traffic lights, pedestrian crossings etc.

We read the node tags during the second OSM pass (because only then do we know which nodes belong to the ways we are interested in) and then we need to keep around until we read the ways in the second pass. Keeping all the node tags is not as bad as it sounds, though. I did a test for Germany and there were 540_000 nodes with barrier tags (which we already keep around anyway) and the total number of nodes with tags was around 2_600_000, so around five times as many.

Here are the most frequent node tags (only nodes belonging to highways were counted and note that nodes carrying multiple tags appear once for each tag in these numbers):

highway: 1.12%, 791784
barrier: 0.76%, 537916
name: 0.51%, 364319
crossing: 0.42%, 294333
created_by: 0.39%, 276287
public_transport: 0.35%, 246989
bus: 0.34%, 242477
bicycle: 0.28%, 200287
foot: 0.26%, 182614
tactile_paving: 0.26%, 180970
entrance: 0.25%, 178177
noexit: 0.21%, 150552
traffic_sign: 0.18%, 125280
access: 0.18%, 124076
crossing:island: 0.15%, 108724
direction: 0.14%, 97463
railway: 0.13%, 91601
network: 0.10%, 68049
kerb: 0.09%, 63401
button_operated: 0.09%, 61035
wheelchair: 0.08%, 55419
source: 0.08%, 54207
traffic_signals:sound: 0.08%, 54145
ref:IFOPT: 0.07%, 52075
traffic_signals:direction: 0.07%, 51788

We could reduce the number of tags we need to store by excluding some we definitely don't need, like created_by=JOSM for example, but not even sure if this is really needed. If we want to save memory we should probably rather focus on using a more efficient way to store the tags as currently this is simply a List<Map<String, Object>>. But this is probably something for later and first we should check how useful the node tags can be (hence this draft PR).

The node tags are available to all tag parsers via:

List<Map<String, Object>> nodeTags = (List<Map<String, Object>>) way.getTag("node_tags");

and this list contains one entry for each point in the point list of the current edge.

Related: #2430 (comment)

todo:

• try some real-world examples, maybe add some encoded values that make use of node tags
• fix existing tests
• review code, pay special attention to barrier node handling
• add some tests
• check performance, memory requirements and number of node tags also for global imports
• if needed: exclude unnecessary node tags
• try more memory-efficient way to store node tags -> EdgeKVStorage. For now not necessary as memory usage does not seem much higher.

[easbar]

This could also be interesting to improve urban density calculations (traffic lights, bus stops, street lamps, zebra etc. all indicate that a street belongs to some kind of populated area).

[karussell]

We split barrier nodes into separate (short) edges where both nodes have these barrier tags. The adjacent edges only have one node with these barrier tags and so we need allMatch and anyMatch is not sufficient.

[easbar]

I think we should do this already one level higher in OSMReader#handleEdge#setArtificialWayTags, something like:

        if (!nodeTags.isEmpty() && (boolean) nodeTags.get(0).getOrDefault("gh:split_node", false))
            way.setTag("gh:split_node_edge", true);
        else
            way.removeTag("gh:split_node_edge");

This way the downstream tag parsers can recognize edges that were created by splitting a node by looking at the gh:split_node_edge tag and can access the tags of the split node by looking at the node tags for either of the two nodes (they are the same anyway).

[easbar]

We can do this even earlier: The moment we create the barrier edge we now add the gh:barrier_edge tag for the reader way.

We could even use the node tags of the split node as value of the gh:barrier_edge tag (instead of simply true)? But we can also obtain these tags in the parsers by looking at the node tags of either of the two nodes of these edges, like we currently do, not sure if one would be more clear than the other. We could even do both :)

[easbar]

I'd rather create the compact map in OSMNodeData.setTags, so it is kind of up to this 'storage' class how to the tags are stored (and compacted and such).

[easbar]

Let's rename handleNodeTags? It sounds like something general, but cit is only called for edges tagged as gh:barrier_edge, and the same for the JavaDoc comment.

[easbar]

In my last commit I added this LongSet to replace the artificial (and possibly confusing?) gh:split_node tag, and so we do not need the remove operation for the tags (in case we like to store the node tags with EdgeKVStorage or similar).

Now we use the following data structures for the node data:

• idsByOSMNodeIds is a (very large) map that maps the full range of possible (64bit) OSM node IDs to a smaller (32bit) range of indices of nodes we actually use to build the graph (typically those contained in ways using the highway tag).

• nodeTagIndicesByOSMNodeIds is a similar, but smaller, map that does the same, but only for nodes for which we keep the tags for

• nodeTags is a list that maps the int indices we get via nodeTagIndicesByOSMNodeIds to the actual tags

• nodesToBeSplit is a separate hash set of OSM node IDS that shall be split. for this we need a way to remove/unset values.

Quite possibly we could combine some of these to either save memory or speed up the import, or both? For example we could combine idsByOsmNodeIds and nodeTagIndicesByOsmNodeIds, which would definitely be an improvement if we stored tags for all nodes. Or we could use the int index we get via nodeTagIndicesByOsmNodeIds to replace the nodesToBeSplit LongSet with a BitSet (using the int index).

[easbar]

Quite possibly we could combine some of these to either save memory or speed up the import, or both?

It looks like whenever we do a lookup into nodeTagIndicesByOSMNodeIds we also do a lookup into idsByOSMNodeIds anyway, which means we could get rid of the lookup into nodeTagIndicesByOSMNodeIds. But since there are less nodes for which we store tags than there are nodes we need to map the (larger) int index we get from idsByOSMNodeIds to a smaller, more compact index. At least as long as we want to keep the node tags list compact. So actually we would then need another int-int lookup, and I'm not sure if overall this will be better, but it might be worth a try.

[easbar]

Did you check for how many nodes we now store the tags (compared to the total number of nodes we map the coordinates for)?

[easbar]

A bit off-topic, but it also might speed up the import if we replaced:

 int curr = idsByOsmNodeIds.get(osmNodeId);
 if (curr == EMPTY_NODE)
       idsByOsmNodeIds.put(osmNodeId, newNodeType);

with something like

  int index = idsByOsmNodeIds.indexOf(osmNodeId);
  int curr = idsByOsmNodeIds.indexGet(index);
  if (curr == EMPTY_NODE)
        idsByOsmNodeIds.indexPut(index, newNodeType);

but not sure if using an index like this is possible with GHLongIntBTreeMap.

[karussell]

For the planet import (planet-230227.osm.pbf, openjdk 17) this branch uses slightly more memory than master (minimum Xmx was 88g instead of 87g). Without the KVStorage the memory usage would have been 6gb more.

Still, it takes the same time for the 2 passes (~6400s).

One needs to take a bit more care to implement the node handling in the TagParsers to avoid a slow down. E.g. before this change the slow down was ~7%.

Seems to work. Now we can exclude e.g. fords and ferries via a custom model too.