Lua library for flex backend

[joto]

The flex backend uses Lua much more than before. Many Lua scripts will need similar code to handle common cases like parsing/converting tags, checking for existence of certain tags etc. Some code for that is already here. This code is compiled into osm2pgsql and available as functions in the osm2pgsql namespace.

Please use this issue to propose functions for the library or discuss needs.

[joto]

• Remove common unused tags (by key or prefix) like source, or note
• Check for specific tags (for instance to decide whether something should go into a specific table)
• Check for way tags that should be handled as polygons
• Convert special tag formats (with semicolons, numbers with units, etc.) What do we need here?

[mmd-osm]

A few ideas I could find in the Overpass docs:

• Number conversions

  • Check if string is a number
  • Convert string to a number

• Semicolons

  • Split semicolon delimited string into table
  • Check if string exists in this table
    (this could be one function: check if semicolon delimited string contains string)

• Units

  • Prefix / Suffix -> convert "34 mph" into "34" and "mph".

OSRM also has a number of Lua function they use for preprocessing. I haven't looked through them in detail, but it's not unlikely that they already implemented some functions that may be relevant for osm2pgsql as well (e.g. tokenizer, unit conversion routines, some debug output, ...):

• https://github.com/Project-OSRM/osrm-backend/tree/master/profiles
• https://github.com/Project-OSRM/osrm-backend/tree/master/profiles/lib
• https://github.com/Project-OSRM/osrm-backend/blob/master/profiles/examples/postgis.lua (calling external Postgis datasources)

[pnorman]

Why do these need to go directly into osm2pgsql?

[joto]

@pnorman They don't need to go, this would just be something helpful so that not everybody has to reinvent the wheel. Of course it could be maintained outside osm2pgsql, but I this way it is easier to use for everybody.

[gravitystorm]

I'd like to see the current integer handling available as a library function, namely the (currently default in pgsql backend) support for ranges in integer tags. For example, <tag k="population" v="10000-20000" /> with an int4 column type is outputted as 15000.

[pnorman]

I have a few functions from cleartables that would probably be useful, I'll dig them up

[pnorman]

Where should the test code for init.lua go?

[joto]

@pnorman I have started writing tests for init.lua in a local branch somewhere under tests as simple Lua scripts using the Lua-builtin assert, but I don't know yet how to best integrate this with CTest and the rest of the tests. So far I have been running them "manually" by calling the Lua interpreter directly.

I see several issues here:

• Some tests need the whole osm2pgsql framework around the Lua code. Those should be run as part of the other osm2pgsql tests.
• All unit tests that only test Lua functions and don't need osm2pgsql around it should probably tested without it, just to keep things easy and simple.
• We could use the Lua interpreter binary and just call that with a test.lua script or we write our own "framework" for calling the tests. Calling the Lua binary is easier, but there is no guarantee that it is available on the system, we have to find its path etc. At the moment I am thinking of going this route and see whether any problems come up. We can always switch to something more custom later.

[pnorman]

I've always gone the route of having a test lua file with asserts and using the lua interpreter. Adding it as a dev dependency isn't a problem to me.

[joto]

I have looked through the things mentioned in this thread and added a few functions. But I'd rather be a bit conservative here, because we have to support whatever we add for a long time.

@gravitystorm The population range you mentioned is only a few dozen times in the data, it doesn't look like it is worth it to support this (any more).

I am closing this issue here. Please open new issues if you have specific needs.

[StyXman]

• The most extensive lua code I know is https://github.com/SomeoneElseOSM/SomeoneElse-style/blob/master/style.lua maybe there are some ideas there.
• Maybe some handling for open hours?