Issue importing data and serving tiles

[jerryplin]

I was able to eventually get through the steps outlined in https://www.baremaps.com/examples/openstreetmap/ to show a map of Liechtenstein with the Blueprint style.

I've been trying to get this to work with Switzerland and the Openstreetmap-vecto style. When I import the switzerland-latest.osm.pbf file, I encountered exceptions such as:

[WARN ] 2021-01-28 11:20:12.881 [ForkJoinPool.commonPool-worker-83] GeometryHandler - Unable to build the geometry for relation 1713904
java.lang.NullPointerException: null
at org.locationtech.jts.algorithm.locate.IndexedPointInAreaLocator.locate(IndexedPointInAreaLocator.java:88) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.isInRing(EdgeRing.java:261) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.EdgeRing.findEdgeRingContaining(EdgeRing.java:87) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.findShellContaining(HoleAssigner.java:106) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHoleToShell(HoleAssigner.java:78) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:72) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.HoleAssigner.assignHolesToShells(HoleAssigner.java:40) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.polygonize(Polygonizer.java:255) ~[jts-core-1.17.1.jar:?]
at org.locationtech.jts.operation.polygonize.Polygonizer.getGeometry(Polygonizer.java:188) ~[jts-core-1.17.1.jar:?]
at com.baremaps.importer.geometry.GeometryHandler.handle(GeometryHandler.java:104) ~[baremaps-importer-0.3.4.jar:?]
at com.baremaps.osm.domain.Relation.accept(Relation.java:47) ~[baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.EntityHandler.accept(EntityHandler.java:20) ~[baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.EntityHandler.accept(EntityHandler.java:15) ~[baremaps-osm-0.3.4.jar:?]
at java.util.function.Consumer.lambda$andThen$0(Consumer.java:65) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1541) [?:?]
at com.baremaps.osm.BlockEntityHandler.handle(BlockEntityHandler.java:27) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.pbf.DataBlock.handle(DataBlock.java:42) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.BlockHandler.accept(BlockHandler.java:15) [baremaps-osm-0.3.4.jar:?]
at com.baremaps.osm.BlockHandler.accept(BlockHandler.java:9) [baremaps-osm-0.3.4.jar:?]

and then error messages such as:

[WARN ] 2021-01-28 11:20:32.414 [ForkJoinPool.commonPool-worker-123] GeometryHandler - Unable to build the geometry for relation 7521607
java.lang.NullPointerException: null
[WARN ] 2021-01-28 11:20:37.100 [ForkJoinPool.commonPool-worker-15] GeometryHandler - Unable to build the geometry for relation 5503310
java.lang.NullPointerException: null

It's unclear to me whether the import job finished despite the errors, and perhaps gave up after encountering the errors. Either way, when I try to run the server, I get exceptions such as:

[ERROR] 2021-01-28 11:47:05.277 [armeria-common-worker-epoll-2-6] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_ways_z12" does not exist
Position: 320
[ERROR] 2021-01-28 11:47:05.276 [armeria-common-worker-epoll-2-5] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_relations_z12" does not exist
Position: 320
[ERROR] 2021-01-28 11:47:05.276 [armeria-common-worker-epoll-2-4] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_water_polygons" does not exist
Position: 335

I believe something similar happened when I tried to import us-west. My larger goal is to import the OSM data for the world.

On a tangential note, I have used imposm3 with some success before; could Baremaps' server be made to work with imposm imports? (e.g. specifying an appropriate mapping file)

Thank you.

[bchapuis]

[WARN ] 2021-01-28 11:20:12.881 [ForkJoinPool.commonPool-worker-83] GeometryHandler - Unable to build the geometry for relation 1713904

The warnings displayed during the import are due to a few complex relations in OpenStreetMap that are not easily translatable into valid geometries with the JTS Polygonizer. The Poligonizer throws a NullPointerException) which does not stop the import procedure, hence the choice of displaying a warning. The affected relations get saved in the database with a geometry set to null and I hope that the log (which contains the ids) will help at fixing these corner cases in the future. ;)

[ERROR] 2021-01-28 11:47:05.277 [armeria-common-worker-epoll-2-6] TileService - org.postgresql.util.PSQLException: ERROR: relation "osm_ways_z12" does not exist
Position: 320

Here, osm_ways_z12 corresponds to a materialized view used by openstreetmap-vecto to simplify geometries at a lower zoom levels (_z12) in order to improve performences. You can take a look at the import script and at the materialized view file to create the missing views and indexes.

Any data stored in postgis can be served with baremaps, so I guess that it should be possible to use imposm3 or osm2psql. Our goal with Baremaps is to create an cohesive pipeline from one end (import) to the other (export), so that it can easily be used on a desktop and then deployed on the cloud.

Our roadmap also includes the creation of a world map, so any feedback or bug fix is welcome. For now, our main limitation is infrastructure cost, i.e., a database for the whole world probably weight around 3-4TB.

[jerryplin]

I didn't realize there was a separate import script. After I ran various lines manually, I am able to reproduce the map on the demo page. Thanks!