Basic Export

[manuelroth]

• Basic Export with bounding box, Name, Point(Centroid)
• Without ranking, hierarchie

[lukasmartinelli]

We will create a Docker container that attaches to our database and at the
beginning just query the countries and places and dumps them in CSV format.

For the following features

• Countries (Admin level 0)
• Places

[sfkeller]

Regarding the export format spec. I propose to use GeoCSV:
http://giswiki.hsr.ch/GeoCSV#CSV_file_format_specification
The specific fields need and their properties (mandatory, data type) still to be defined.

[manuelroth]

We decided to prepare an export container which takes the table osm_places as example. And exports as much information from the following list as possible:

*osm_id - MUST BE UNIQUE "DOCUMENT ID" accross complete database

display_name - exactly as in Nominatim (may be improved later)

*name (=utf-8)
name_en
name_de
name_es
name_fr
name_ru
name_zh

*class
*type

*north (=boundingbox)
*south
*east
*west

*lat
*lon

scalerank - we have it
place_rank - nominatim has it

importance - exactly as in nominatim calculated

country (=country code, ISO-3166 2-letter country code)

street=<housenumber> <streetname>
city=<city>
county=<county>
state=<state>
country=<country>
postalcode=<postalcode>

(= a la nominatim http://wiki.openstreetmap.org/wiki/Nominatim)

? timestamp - osm modification?

[sfkeller]

Following additions:

[Update - comments moved to #3]

[klokan]

@lukasmartinelli @manuelroth:

Agree to generate only "osm_places" for now. Basic export should be .tsv with columns which you have already available.
@klokantech team will index it with: https://github.com/klokantech/osmnames-sphinxsearch (docker is now ready).
A basic sample data: https://github.com/klokantech/osmnames-sphinxsearch/blob/master/sample.tsv
Please include as many columns from early spec mentioned above as possible.

This is fine for the Basic export - and once you have docker for this we can close this ticket.

@sfkeller:
Please let the initial basic export be as discussed above - without additions at this point in time - to not block development of osmnames-sphinxesearch.

To discuss details of the data format I have created a ticket #3. I am moving your comments from this ticket to #3. Hopefully you agree.

[lukasmartinelli]

How can we join for example a place label of rapperswil with the boundaries of rapperswil?

Just looked at the data and in OSM the administrative boundaries are not polygons but linestrings which are not linked together - so can we extract administrative area polygons out of OSM data?

There is also no osm_id of the administrative area that matches the osm_id of the place label or other linking data.

Perhaps @sfkeller knows more about how to link administrative boundaries to place labels?

For example Rapperswil is just a point.

And this is a border of my hometown.

[lukasmartinelli]

For places I can get out a export but have no way yet to get bounding boxes.

https://gist.github.com/lukasmartinelli/e40a48d93cc05b2dbb15

https://github.com/osm2vectortiles/osm2vectortiles/tree/feature/export-geonames/src/export-geonames

[sfkeller]

How can we join for example a place label of rapperswil with the boundaries of rapperswil?
...
Perhaps @sfkeller knows more about how to link administrative boundaries to place labels?

In OSM this is done in the data with relations: Relations aggregate admin. boundaries and the same relation usually also contans a node with the center of the admin. unit (i.e. "the pace label"). See e.g. City (= county of) Zürich: http://www.openstreetmap.org/relation/1682248 . Unfortunately that's not yet in the OSM data for Uznach http://www.openstreetmap.org/relation/1683953 nor Rapperswil-Jona.

AFAIK that reference from a place or building (having point coordinates) to the next outer admin. boundary is calculated in Nominatim and e.g. also in the "?" function in osm.org which uses Overpass API.

In either case it's probably simply an ST_Witihin function ordered by area size.Now I don't know if you can calculate this?

[lukasmartinelli]

In OSM this is done in the data with relations: Relations aggregate admin. boundaries and the same relation usually also contans a node with the center of the admin. unit (i.e. "the pace label"). See e.g. City (= county of) Zürich: http://www.openstreetmap.org/relation/1682248 . Unfortunately that's not yet in the OSM data for Uznach http://www.openstreetmap.org/relation/1683953 nor Rapperswil-Jona.

Thanks a lot. Forgot about the relations concept.
We just looked at this in more detail. It is definitly quite some work to relate the labels with the polygons.

We need to make polygons of all those relations for the boundaries in imposm3.
This requires a new imposm3 mapping - which means kind of forking osm2vectortiles and changing the mapping. We also need a new mapping anyway because we don't have any data for some fields specified in the export specification (buildings have no adresses mapped for example).

In either case it's probably simply an ST_Witihin function ordered by area size.Now I don't know if you can calculate this?

Once we have the polygons we can start try to relate the labels with the polygons.
And yes the only method I currently see is to guess in which administrative boundary a label is with function like ST_Within and a admin level that corresponds to e.g. place=city.

[manuelroth]

We decided to make the basic export for the layer roads. Because this layer does not need any joining with an other table and we don't have to rewrite the mapping for now.

The following fields can be extracted:
display_name
*class
*type

*north (=boundingbox)
*south
*east
*west

*lat
*lon

[manuelroth]

What is exactly the reason why we need the lat/lon value of the geometry?- Do we need the lat/lon value to highlight the geometry like in the picture below:

I understand the cause for the bounding box => zooming in until the bounding box fits the viewport.

Would appreciate help @klokan @sfkeller

[lukasmartinelli]

What is exactly the reason why we need the lat/lon value of the geometry?- Do we need the lat/lon value to highlight the geometry like in the picture below:

This matters to decide whether to use st_centroid or st_pointonsurface to convert linestrings and polygons to points.

[sfkeller]

We really have to be aware that this is about geographic names (not building addresses).
P.S. IMHO streets and housenumbers (and coordinates of them) have nothing to do here.

Geographic names can be represented either as point (landmark plus some "surrounding", "Bundesplatz"), linear ("Albiskette") or areal geometry (city of Zurich).

Bounding Box can be a mandatory field(s) but it's a derived or approximated info from the "real" geometries.

Field lat/lon in this context can only mean that's it's the (center) location of a geographic name.

[klokan]

Ideally the point should represent where the label typically is located.

The real reason for point in a Search API is the cheap distance calculation. Typically fulltext engines supports scoring based on distance only for a point index (as calculations and indexes on bbox are significantly more expensive and distance is only one part of the final rank for given query).
Points are for this reason aslo typically used for filtering visible features.

[klokan]

We are looking forward to get the early first export at least for roads - to test the server side on some larger close to real database.

In case any other table can be exported directly of with easy to do joins, please add it too to the initial export. We hope to get the export soon from you.

Important for meeting on Monday is having analyse of data tables described in osm2vectortiles/osm2vectortiles#156

[klokan]

Thanks for the basic export with download link at osm2vectortiles/osm2vectortiles#157 @manuelroth @lukasmartinelli

We are working on a first deploy of osmnames search indexing the roads to test the performance of the search engine powered by sphinx search: klokantech/osmnames-sphinxsearch#3

[klokan]

Basic data deployed in very early version of the search at: http://osmnames.klokantech.com/.

Thanks @manuelroth @lukasmartinelli

[klokan]

Tables with/without geometry documented: osm2vectortiles/osm2vectortiles#156

[lukasmartinelli]

Mapzen also offers a similar service: http://whosonfirst.mapzen.com/