Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bus and metro stations not showing up? #337

Closed
wimpie3 opened this issue Oct 6, 2021 · 20 comments · Fixed by #338
Closed

Bus and metro stations not showing up? #337

wimpie3 opened this issue Oct 6, 2021 · 20 comments · Fixed by #338

Comments

@wimpie3
Copy link

wimpie3 commented Oct 6, 2021

I'm not able to show public transport stations on my map. POI's like hospitals, parkings and shops appear, but no stations or bus stops.

My layout json uses the "poi" source layer with a filter on "bus", "rail","airport".

Is Tilemaker using an alternative name for the transportation layer? Or does it currently skip those points while making the vector layers?

@systemed
Copy link
Owner

systemed commented Oct 6, 2021

Ah, I see. Try building #338 and see if that makes it better.

@wimpie3
Copy link
Author

wimpie3 commented Oct 6, 2021

I'm on Windows so that will be difficult...

@systemed
Copy link
Owner

systemed commented Oct 6, 2021

You don't need to build tilemaker itself, it's just a change to the .lua file. If you change the process-openmaptiles.lua file to have that fix, then rerun tilemaker, it should (fingers crossed) work for you.

@wimpie3
Copy link
Author

wimpie3 commented Oct 6, 2021

Seems to be working! Thank you!

@systemed
Copy link
Owner

systemed commented Oct 6, 2021

Excellent!

@systemed systemed closed this as completed Oct 6, 2021
@wimpie3
Copy link
Author

wimpie3 commented Oct 7, 2021

OK, on further investigation it seems that only the bus stations are visible. No metro or tram stations. Is this possible?

@systemed
Copy link
Owner

systemed commented Oct 7, 2021

Certainly possible. Could you give me an example of a metro or tram station that you'd like to see included, with its OSM object link (so that I don't have to go scratting through the impenetrable OSM wiki public transport documentation)?

@wimpie3
Copy link
Author

wimpie3 commented Oct 7, 2021

https://maputnik.github.io/editor/#19.77/50.8513828/4.3531603/20/8
There is a tram station in the center which is not visible when passing through tilemaker.

@systemed
Copy link
Owner

systemed commented Oct 7, 2021

Can you give me an OSM object link, please. For example, https://www.openstreetmap.org/node/123456

@wimpie3
Copy link
Author

wimpie3 commented Oct 7, 2021

That is bizarre, cause the tram icons I can see on the maputnik link aren't visible on the openstreet map! OSM only shows bus stops...

@systemed
Copy link
Owner

systemed commented Oct 7, 2021

Switch to the transport layer or OPNVKarte on osm.org; find the tram stop; use the "Query features" tool to click on it; select the one you want; send the OSM object link like https://www.openstreetmap.org/node/123456 .

@wimpie3
Copy link
Author

wimpie3 commented Oct 7, 2021

@systemed
Copy link
Owner

systemed commented Oct 7, 2021

bfe1bb9 should fix this - it now generates POIs like this:

  "properties": {
    "class": "railway",
    "name:latin": "Gare du Nord - Noordstation",
    "rank": 2,
    "subclass": "tram_stop",
    "vt_layer": "poi"
  }

(output from vt2geojson http://localhost:8080/14/8390/5495.pbf when converting a Brussels extract)

@wimpie3
Copy link
Author

wimpie3 commented Oct 7, 2021

So this should work, right? Hmmm... it's not...

      "id": "poi-tram_stop",
      "type": "symbol",
      "source": "openmaptiles",
      "source-layer": "poi",
      "minzoom": 15,
      "filter": [
        "all",
        [
          "==",
          "$type",
          "Point"
        ],
        [
          "==",
          "class",
          "railway"
        ],
        [
          "==",
          "subclass",
          "tram_stop"
        ]
      ],

@systemed
Copy link
Owner

systemed commented Oct 7, 2021

Don't know, I'm afraid - debugging styles is a bit outside tilemaker's scope ;)

I'd recommend using vt2geojson as above for debugging - you can capture a vector tile in GeoJSON format and then look through it with a text editor to see if the data you want is in there. (You could use ogr2ogr 2.3 or later to do the same.)

@systemed
Copy link
Owner

systemed commented Oct 8, 2021

Could you give the exact command line that you're using, and link to the .osm.pbf extract?

@wimpie3
Copy link
Author

wimpie3 commented Oct 8, 2021

tilemaker --input belgium-latest.osm.pbf --output=tiles/ --process process-openmaptiles.lua --config config-openmaptiles.json

Source of the PBF is from geofabrik.

Extract ends with "Sorting nodes" and returns to the command prompt. I've retried it with the original lua script and it's failing as well. I've recently installed Windows 11 on my laptop, might be due to that?

@systemed
Copy link
Owner

systemed commented Oct 8, 2021

I don't have any experience with Windows but it's possible that you might have run out of RAM, especially if Windows 11 itself takes up more RAM than your previous version.

If you have an SSD, running with --store /path/to/somewhere/on/your/ssd will reduce RAM consumption by using the SSD for temporary storage. Alternatively, if you can install the Osmium tool, one easy way to reduce Tilemaker's memory consumption is to run osmium renumber over the extract first, then use that in Tilemaker with the --compact switch.

I'll run a quick check here on the Belgium extract (on Linux) to see if I can reproduce the issue.

@wimpie3
Copy link
Author

wimpie3 commented Oct 8, 2021

--store didn't help, still exits right after "Sorting nodes".

UPDATE: found the issue, with the original LUA script everything works again, so there must be a mistake in there somewhere...
UPDATE 2: yep, made a typo while applying the language patch, sorry

Might be an interesting idea to show an error message in the future instead of exiting back to the command prompt :-)

@systemed
Copy link
Owner

systemed commented Oct 8, 2021

Ah, that's worth knowing. It does usually output a message on Linux/macOS so maybe that's a Windows executable thing. Glad it's working anyway :)

cldellow added a commit to cldellow/tilemaker that referenced this issue Dec 29, 2023
This PR generalizes the idea of `node_keys`, adds `way_keys`, and fixes systemed#402.

I'm not too sure if this is generally useful - it's useful for one of my
use cases, and I see someone asking about it in systemed#190
and, elsewhere, in onthegomap/planetiler#99

If you feel it complicates the maintainer story too much, please reject.

The goal is to reduce memory usage for users doing thematic extracts by
not indexing nodes that are only used by uninteresting ways.

For example, North America has ~1.8B nodes, needing 9.7GB of RAM for its node
store. By contrast, if your interest is only to build a railway map, you
require only ~8M nodes, needing 70MB of RAM. Or, to build a map of
national/provincial parks, 12M nodes and ~120MB of RAM.

Currently, a user can achieve this by pre-filtering their PBF using
osmium-tool. If you know exactly what you want, this is a good
long-term solution. But if you're me, flailing about in the OSM data
model, it's convenient to be able to tweak something in the Lua script
and observe the results without having to re-filter the PBF and update
your tilemaker command to use the new PBF.

Sample use cases:

```lua
-- Building a map without building polygons, ~ excludes ways whose
-- only tags are matched by the filter.
way_keys = {"~building"}
```

```lua
-- Building a railway map
way_keys = {"railway"}
```

```lua
-- Building a map of major roads
way_keys = {"highway=motorway", "highway=trunk", "highway=primary", "highway=secondary"}`
```

Nodes used in ways which are used in relations (as identified by
`relation_scan_function`) will always be indexed, regardless of
`node_keys` and `way_keys` settings that might exclude them.

A concrete example, given a Lua script like:

```lua
function way_function()
  if Find("railway") ~= "" then
    Layer("lines", false)
  end
end
```

it takes 13GB of RAM and 100 seconds to process North America.

If you add:

```lua
way_keys = {"railway"}
```

It takes 2GB of RAM and 47 seconds.

Notes:

1. This is based on `lua-interop-3`, as it interacts with files that are
   changed by that. I can rebase against master after lua-interop-3 is
   merged.

2. The names `node_keys` and `way_keys` are perhaps out of date, as they
   can now express conditions on the values of tags in addition to their
   keys. Leaving them as-is is nice, as it's not a breaking change.
   But if breaking changes are OK, maybe these should be
   `node_filters` and `way_filters` ?

3. Maybe the value for `node_keys` in the OMT profile should be
   expressed in terms of a negation, e.g. `node_keys = {"~created_by"}`?
   This would avoid issues like systemed#337
cldellow added a commit to cldellow/tilemaker that referenced this issue Dec 29, 2023
This PR generalizes the idea of `node_keys`, adds `way_keys`, and fixes systemed#402.

I'm not too sure if this is generally useful - it's useful for one of my
use cases, and I see someone asking about it in systemed#190
and, elsewhere, in onthegomap/planetiler#99

If you feel it complicates the maintainer story too much, please reject.

The goal is to reduce memory usage for users doing thematic extracts by
not indexing nodes that are only used by uninteresting ways.

For example, North America has ~1.8B nodes, needing 9.7GB of RAM for its node
store. By contrast, if your interest is only to build a railway map, you
require only ~8M nodes, needing 70MB of RAM. Or, to build a map of
national/provincial parks, 12M nodes and ~120MB of RAM.

Currently, a user can achieve this by pre-filtering their PBF using
osmium-tool. If you know exactly what you want, this is a good
long-term solution. But if you're me, flailing about in the OSM data
model, it's convenient to be able to tweak something in the Lua script
and observe the results without having to re-filter the PBF and update
your tilemaker command to use the new PBF.

Sample use cases:

```lua
-- Building a map without building polygons, ~ excludes ways whose
-- only tags are matched by the filter.
way_keys = {"~building"}
```

```lua
-- Building a railway map
way_keys = {"railway"}
```

```lua
-- Building a map of major roads
way_keys = {"highway=motorway", "highway=trunk", "highway=primary", "highway=secondary"}`
```

Nodes used in ways which are used in relations (as identified by
`relation_scan_function`) will always be indexed, regardless of
`node_keys` and `way_keys` settings that might exclude them.

A concrete example, given a Lua script like:

```lua
function way_function()
  if Find("railway") ~= "" then
    Layer("lines", false)
  end
end
```

it takes 13GB of RAM and 100 seconds to process North America.

If you add:

```lua
way_keys = {"railway"}
```

It takes 2GB of RAM and 47 seconds.

Notes:

1. This is based on `lua-interop-3`, as it interacts with files that are
   changed by that. I can rebase against master after lua-interop-3 is
   merged.

2. The names `node_keys` and `way_keys` are perhaps out of date, as they
   can now express conditions on the values of tags in addition to their
   keys. Leaving them as-is is nice, as it's not a breaking change.
   But if breaking changes are OK, maybe these should be
   `node_filters` and `way_filters` ?

3. Maybe the value for `node_keys` in the OMT profile should be
   expressed in terms of a negation, e.g. `node_keys = {"~created_by"}`?
   This would avoid issues like systemed#337
cldellow added a commit to cldellow/tilemaker that referenced this issue Dec 29, 2023
This PR generalizes the idea of `node_keys`, adds `way_keys`, and fixes systemed#402.

I'm not too sure if this is generally useful - it's useful for one of my
use cases, and I see someone asking about it in systemed#190
and, elsewhere, in onthegomap/planetiler#99

If you feel it complicates the maintainer story too much, please reject.

The goal is to reduce memory usage for users doing thematic extracts by
not indexing nodes that are only used by uninteresting ways.

For example, North America has ~1.8B nodes, needing 9.7GB of RAM for its node
store. By contrast, if your interest is only to build a railway map, you
require only ~8M nodes, needing 70MB of RAM. Or, to build a map of
national/provincial parks, 12M nodes and ~120MB of RAM.

Currently, a user can achieve this by pre-filtering their PBF using
osmium-tool. If you know exactly what you want, this is a good
long-term solution. But if you're me, flailing about in the OSM data
model, it's convenient to be able to tweak something in the Lua script
and observe the results without having to re-filter the PBF and update
your tilemaker command to use the new PBF.

Sample use cases:

```lua
-- Building a map without building polygons, ~ excludes ways whose
-- only tags are matched by the filter.
way_keys = {"~building"}
```

```lua
-- Building a railway map
way_keys = {"railway"}
```

```lua
-- Building a map of major roads
way_keys = {"highway=motorway", "highway=trunk", "highway=primary", "highway=secondary"}`
```

Nodes used in ways which are used in relations (as identified by
`relation_scan_function`) will always be indexed, regardless of
`node_keys` and `way_keys` settings that might exclude them.

A concrete example, given a Lua script like:

```lua
function way_function()
  if Find("railway") ~= "" then
    Layer("lines", false)
  end
end
```

it takes 13GB of RAM and 100 seconds to process North America.

If you add:

```lua
way_keys = {"railway"}
```

It takes 2GB of RAM and 47 seconds.

Notes:

1. This is based on `lua-interop-3`, as it interacts with files that are
   changed by that. I can rebase against master after lua-interop-3 is
   merged.

2. The names `node_keys` and `way_keys` are perhaps out of date, as they
   can now express conditions on the values of tags in addition to their
   keys. Leaving them as-is is nice, as it's not a breaking change.
   But if breaking changes are OK, maybe these should be
   `node_filters` and `way_filters` ?

3. Maybe the value for `node_keys` in the OMT profile should be
   expressed in terms of a negation, e.g. `node_keys = {"~created_by"}`?
   This would avoid issues like systemed#337

4. This also adds a SIGUSR1 handler during OSM processing, which prints
   the ID of the object currently being processed. This is helpful for
   tracking down slow geometries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants