Move external shapefiles to tables in the DB #4092

pnorman · 2020-03-26T05:03:31Z

This adds a script that loads files into the DB based on a
YAML file listing the data sources. The script can be run
while rendering is going on, as it swaps old tables with
new ones in a transaction.

Loading is done by using ogr2ogr to load into a temporary
schema, clustering, then the swap in transaction. The status
of the tables is tracked in the external_data table, which
lists the last modified date of each table. This allows the
loading script to use conditional GETs and only download and
update for sources which have changed.

Fixes #4089

Rendering tested with a planet DB in Antarctica and a local coastline.

If I were rewriting the python script I'd probably do it differently now that I'm better, but it's existing code that works. I'm happy with it's functionality, it's just internal refactoring I would do.

pnorman · 2020-03-27T05:10:51Z

I'm going to run some benchmarking on this

scripts/get-external-data.py

project.mml

pnorman · 2020-03-27T22:20:59Z

Averaged over zoom 0 to zoom 12, shapefiles in the DB takes 514 minutes, shapefiles on disk take 521 minutes. This is a 1% difference, but I'm not sure it's above my margins of error. In any case, performance isn't a concern with this PR.

jeisenbe · 2020-03-28T01:08:30Z

Averaged over zoom 0 to zoom 12, shapefiles in the DB takes 514 minutes, shapefiles on disk take 521 minutes.

Is that the time to render all metatiles at these 13 zoom levels, or just to render the oceans and icesheets?

pnorman · 2020-03-28T01:26:42Z

Is that the time to render all metatiles at these 13 zoom levels, or just to render the oceans and icesheets?

Everything.

jeisenbe · 2020-03-28T02:24:48Z

For Docker, will ogr2ogr already be included in the container? No changes are needed other than the switch to scripts/get-external-data.py?

pnorman · 2020-03-28T02:40:19Z

For Docker, will ogr2ogr already be included in the container? No changes are needed other than the switch to scripts/get-external-data.py?

I added gdal-bin to the dockerfile, that should be enough. If someone wants to test it, that would be good.

jeisenbe · 2020-03-28T02:41:22Z

I believe having the coastlines as ways in the database would make it easier to solve #712 (hide admin boundary lines on the coastline)

jeisenbe · 2020-03-28T02:43:36Z

So I will need to remove the docker image and then rebuild the image to test this? Unfortunately that takes up a large amount of bandwidth, so I won't be able to do it until our airport is reopened again (hopefully sometime in mid-April or at least by May?).

@Adamant36 or @sommerluk - would you be able to test this PR with Docker?

Adamant36 · 2020-03-28T07:39:23Z

@Adamant36 or @sommerluk - would you be able to test this PR with Docker?

Sure. I'll give it a try over the weekend.

sommerluk · 2020-03-28T09:52:50Z

No, I can't test it. Currently, I've here often a bandwidth of 0,0 Mbit/s, and in the best case sometimes 0,9 Mbit/s. Sorry.

pnorman · 2020-03-29T03:52:12Z

I'd like to target this for after 5.1

jeisenbe · 2020-04-03T02:07:22Z

@Adamant36 - would you have a chance to test this over the weekend?

Adamant36 · 2020-04-03T05:46:11Z

Unfortunately not. My VM setup broke and I'm probably going wait until 20.04 comes out to setup it up again so I dont have to deal with it twice. I can test it after that though.

jeisenbe · 2020-04-04T15:38:46Z

Note that this would give up more options for fixing problems like #621, so it is beneficial for more flexibility in rendering options, in addition to the improved performance.

jeisenbe · 2020-04-26T16:17:11Z

I'm back in the USA, so I can test this once the merge conflict is resolved.

This adds a script that loads files into the DB based on a YAML file listing the data sources. The script can be run while rendering is going on, as it swaps old tables with new ones in a transaction. Loading is done by using ogr2ogr to load into a temporary schema, clustering, then the swap in transaction. The status of the tables is tracked in the `external_data` table, which lists the last modified date of each table. This allows the loading script to use conditional GETs and only download and update for sources which have changed.

pnorman · 2020-04-27T19:24:04Z

Rebased.

pnorman · 2020-04-28T04:10:32Z

I've added the new dependencies to the import dockerfile and readme

jeisenbe · 2020-04-28T17:41:57Z

Great, this now works as expected. I was a little surprised that the shapefiles are no longer downloaded as separate files at all.

Will updating the shapefiles require manually running scripts/get-external-data.py or is this going to happen automatically?

jeisenbe · 2020-04-28T18:46:59Z

INSTALL.md


 ```
-scripts/get-shapefiles.py
+scripts/get-external-data.py
 ```

 This script downloads necessary files, generates and populates the *data* directory with all needed shapefiles, including indexing them through *shapeindex*.


This line does not seem to be correct now

pnorman · 2020-04-28T19:02:18Z

Will updating the shapefiles require manually running scripts/get-external-data.py or is this going to happen automatically?

Typically you'd automate this with a cron job

jeisenbe · 2020-04-29T04:41:14Z

Ideally we should document how to do this, though that doesn't need to be included in this PR.

I'm tested this PR in docker, and would be ready to approve it, but I see a review by @imagico is pending.

pnorman · 2020-04-30T02:49:57Z

I'm tested this PR in docker, and would be ready to approve it, but I see a review by @imagico is pending.

I had requested a review because I hadn't gotten any other response.

I'd go ahead and merge this now, but I want to do it at the start of a release cycle

imagico

Sorry for the delay, it seems to work fine.

What is a bit annoying is that with the coastlines in the database you have to duplicate the coastline data if you have several test databases (like for real data and abstract tests) and switch between them via localconfig.

LorenzoStucchi · 2020-06-11T09:37:54Z

Hi, I followed all the instructions for Docker but I still have issues reported here and I don't have a table called external_data on my gis database.

Adamant36 · 2020-06-11T09:45:39Z

The shape files never loaded for me when I did a clean install and ran Kosmtik. It just errored out after the map loaded and said that the shapefiles where missing.

LorenzoStucchi · 2020-06-11T09:52:19Z

So which could be the error?

Here the log file.

jeisenbe · 2020-06-11T16:35:26Z

Here's the relevant part of the log:

Trace
    at ProjectServer.raise (/usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:261:13)
    at /usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:75:30
    at /usr/lib/node_modules/kosmtik/node_modules/generic-pool/lib/generic-pool.js:283:11
    at loaded (/usr/lib/node_modules/kosmtik/node_modules/mapnik-pool/index.js:23:37)
Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer
[httpserver] /openstreetmap-carto/tile/4/7/7.png?t=1591820781712 500
Trace
    at ProjectServer.raise (/usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:261:13)
    at /usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:75:30
    at /usr/lib/node_modules/kosmtik/node_modules/generic-pool/lib/generic-pool.js:283:11
    at loaded (/usr/lib/node_modules/kosmtik/node_modules/mapnik-pool/index.js:23:37)
Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer
[httpserver] /openstreetmap-carto/tile/4/8/7.png?t=1591820781712 500
Postgis Plugin: ERROR:  relation "icesheet_polygons" does not exist
LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "...
                                           ^
in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;'
  encountered during parsing of layer 'icesheet-poly' in Layer
Trace
    at ProjectServer.raise (/usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:261:13)
    at /usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:75:30
    at /usr/lib/node_modules/kosmtik/node_modules/generic-pool/lib/generic-pool.js:283:11
    at loaded (/usr/lib/node_modules/kosmtik/node_modules/mapnik-pool/index.js:23:37)


2020-06-10 20:26:10.916 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2020-06-10 20:26:10.993 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2020-06-10 20:26:11.423 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-06-10 20:26:12.042 UTC [24] LOG:  database system was shut down at 2020-06-10 20:25:50 UTC
2020-06-10 20:26:12.311 UTC [1] LOG:  database system is ready to accept connections
2020-06-10 20:28:46.061 UTC [51] ERROR:  relation "icesheet_polygons" does not exist at character 36
2020-06-10 20:28:46.061 UTC [51] STATEMENT:  SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;

jeisenbe · 2020-06-11T16:38:48Z

@LorenzoStucchi was this a new installation, or an update to a previous version which was working before?

jeisenbe · 2020-06-11T16:39:48Z

@Adamant36 re: "The shape files never loaded for me when I did a clean install and ran Kosmtik":

When did this happen? Was it with the latest release?

jeisenbe · 2020-06-11T16:43:40Z

Sorry, was it the latest commit in this PR, 025d9ce or on the master branch, 5724017
?

LorenzoStucchi · 2020-06-11T16:57:15Z

@jeisenbe the last update is here.

@LorenzoStucchi was this a new installation, or an update to a previous version which was working before?

It is a new installation on Docker on Windows. But I realise that I have to run twice the command docker-compose up import because the first time I have this error:

Creating network "openstreetmap-carto_default" with the default driver
Creating openstreetmap-carto_db_1 ... done                                                                              Creating openstreetmap-carto_import_1 ... done                                                                          Attaching to openstreetmap-carto_import_1
import_1   | Waiting for PostgreSQL to be running
import_1   | Timeout while waiting for PostgreSQL to be running
import_1   | psql: could not translate host name "db" to address: Name or service not known
import_1   | createdb: could not connect to database template1: could not translate host name "db" to address: Name or service not known
import_1   | osm2pgsql version 1.2.1 (64 bit id space)
import_1   |
import_1   | Allocating memory for dense node cache
import_1   | Allocating dense node cache in one big chunk
import_1   | Allocating memory for sparse node cache
import_1   | Sharing dense sparse
import_1   | Node-cache: cache=512MB, maxblocks=8192*65536, allocation method=11
import_1   | Mid: pgsql, cache=512
import_1   | Connection to database failed: could not translate host name "db" to address: Name or service not known
import_1   |
import_1   | Error occurred, cleaning up
import_1   | DB writer thread failed due to ERROR: Connection to database failed: could not translate host name "db" to address: Name or service not known
import_1   |
import_1   |
import_1   | /usr/bin/env: 'python3\r': No such file or directory
openstreetmap-carto_import_1 exited with code 127

jeisenbe · 2020-06-11T17:04:24Z

@LorenzoStucchi did you check out the latest commit of the master branch on this repository, or did you check out the shapefiles_in_db branch in this PR?

LorenzoStucchi · 2020-06-11T17:20:39Z

@LorenzoStucchi did you check out the latest commit of the master branch on this repository, or did you check out the shapefiles_in_db branch in this PR?

@jeisenbe I used the master branch with git clone

jeisenbe · 2020-06-11T17:31:41Z

OK, I've updated to the latest version of Docker , and now I am seeing the same errors as reported by @LorenzoStucchi and @Adamant36:

kosmtik_1 | Trace kosmtik_1 | at ProjectServer.raise (/usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:261:13) kosmtik_1 | at /usr/lib/node_modules/kosmtik/src/back/ProjectServer.js:75:30 kosmtik_1 | at /usr/lib/node_modules/kosmtik/node_modules/generic-pool/lib/generic-pool.js:283:11 kosmtik_1 | at loaded (/usr/lib/node_modules/kosmtik/node_modules/mapnik-pool/index.js:23:37) kosmtik_1 | Postgis Plugin: ERROR: relation "icesheet_polygons" does not exist kosmtik_1 | LINE 1: SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "... kosmtik_1 | ^ kosmtik_1 | in executeQuery Full sql was: 'SELECT ST_SRID("way") AS srid FROM icesheet_polygons WHERE "way" IS NOT NULL LIMIT 1;' kosmtik_1 | encountered during parsing of layer 'icesheet-poly' in Layer kosmtik_1 | [httpserver] /openstreetmap-carto/tile/4/10/9.png?t=1591895615864 500 kosmtik_1 | [httpserver] /openstreetmap-carto/poll/ 200

The import process was otherwise fine: if I switch back to the latest release (v5.2.0) there is no problem, even without re-importing the data.

jeisenbe · 2020-06-11T17:42:38Z

However, when on the last commit in this PR or on the latest commit on the master branch, I now notice this error after the import process completes:

/usr/bin/env: 'python3': No such file or directory

This is noted after the data is imported:

Osm2pgsql took 2s overall
node cache: stored: 23660(100.00%), storage efficiency: 42.62% (dense blocks: 1, sparse nodes: 23659), hit rate: 100.00%
/usr/bin/env: 'python3': No such file or directory

jeisenbe · 2020-06-11T18:03:30Z

OK, I killed and removed all the docker processes and images, and tried starting again from the last commit in this PR. This worked fine.

To do this, run docker-compose kill, then run docker image ls and find the id numbers of all docker images.
Use docker image rm <id number of image> for to remove each image.

Then you can use docker-compose up import to rebuild the whole thing: this will involve downloading everything again and will take quite a lot of time, especially the step INFO:root:Checking table water_polygons.

You might need to enter "y" at some point to say that you want to recreate the container.

During the import process, the shapefiles should be downloaded and imported, if this has not been done already.

Then run docker-compose up kosmtik as usual and wait for this to be rebuilt. You again might be prompted to type "y" to rebuild to docker image.

jeisenbe · 2020-06-11T18:15:47Z

I have also tried this with the latest commit on the master branch, and it works on my setup.

@Adamant36 and @LorenzoStucchi could you try this and see if it works?

LorenzoStucchi · 2020-06-11T18:41:08Z

I tried but not succeded, using the 5.2version I have this issue at the end of docker-compose up import, and no request of confirmation.

Attaching to openstreetmap-carto_import_1
import_1   | Waiting for PostgreSQL to be running
import_1   | Timeout while waiting for PostgreSQL to be running
import_1   | psql: could not translate host name "db" to address: Name or service not known
import_1   | createdb: could not connect to database template1: could not translate host name "db" to address: Name or service not known
import_1   | osm2pgsql version 1.2.1 (64 bit id space)
import_1   |
import_1   | Osm2pgsql failed due to ERROR: Usage error. For further information see:
import_1   |    osm2pgsql -h|--help
import_1   |
openstreetmap-carto_import_1 exited with code 1

jeisenbe · 2020-06-11T19:30:54Z

You mean that you checked out the tag v5.2.0 in git prior to attempting this?

If you are switching between v5.2.0 and the current master branch, I'm afraid you will need to take the steps outlined above in #4092 (comment) - kill docker-compose, remove the docker relevant image(s), and rebuild.

LorenzoStucchi · 2020-06-11T21:15:41Z

I cancelled everything with the procedure explained and using v5.2.0 I have the error previously explained

Adamant36 · 2020-06-12T03:14:57Z

Reloading everything worked for me. Hopefully there will be a better solution at some point though.

LorenzoStucchi · 2020-06-12T07:41:44Z

I retry with v.5.2.0 and after the total reset of docker, it works. It is possible to do it on the Docker dashboard under the Troubleshoot panel, there is the command reset factory defaults.

pnorman force-pushed the shapefiles_in_db branch from a340b8c to c17e901 Compare March 26, 2020 05:06

pnorman marked this pull request as ready for review March 26, 2020 22:49

pnorman self-assigned this Mar 27, 2020

amandasaurus reviewed Mar 27, 2020

View reviewed changes

scripts/get-external-data.py Show resolved Hide resolved

amandasaurus reviewed Mar 27, 2020

View reviewed changes

project.mml Show resolved Hide resolved

pnorman removed their assignment Mar 28, 2020

jeisenbe added code performance labels Mar 28, 2020

jeisenbe mentioned this pull request Mar 28, 2020

Don't render high(local) administrative borders on coastlines #712

Open

imagico mentioned this pull request Apr 4, 2020

Maritime borders too prominent #621

Open

pnorman requested a review from imagico April 26, 2020 07:11

pnorman added 2 commits April 26, 2020 13:41

Add gdal-bin to the docker image

4111063

pnorman force-pushed the shapefiles_in_db branch from 0994b5f to 4111063 Compare April 26, 2020 20:41

jeisenbe reviewed Apr 28, 2020

View reviewed changes

imagico approved these changes Apr 30, 2020

View reviewed changes

pnorman merged commit 025d9ce into gravitystorm:master May 9, 2020

pnorman deleted the shapefiles_in_db branch January 30, 2021 18:15

tajchert mentioned this pull request Apr 5, 2021

Update carto style to 5.3.1 and shapefile script Overv/openstreetmap-tile-server#176

Closed

sommerluk mentioned this pull request Oct 2, 2022

Low-zoom ocean shapefile not rendered correctly on some setups #2101

Closed

Move external shapefiles to tables in the DB #4092

Move external shapefiles to tables in the DB #4092

Conversation

pnorman commented Mar 26, 2020

pnorman commented Mar 27, 2020

pnorman commented Mar 27, 2020

jeisenbe commented Mar 28, 2020

pnorman commented Mar 28, 2020

jeisenbe commented Mar 28, 2020

pnorman commented Mar 28, 2020

jeisenbe commented Mar 28, 2020

jeisenbe commented Mar 28, 2020

Adamant36 commented Mar 28, 2020

sommerluk commented Mar 28, 2020

pnorman commented Mar 29, 2020

jeisenbe commented Apr 3, 2020

Adamant36 commented Apr 3, 2020

jeisenbe commented Apr 4, 2020

jeisenbe commented Apr 26, 2020

pnorman commented Apr 27, 2020

pnorman commented Apr 28, 2020

jeisenbe commented Apr 28, 2020

jeisenbe Apr 28, 2020

Choose a reason for hiding this comment

pnorman commented Apr 28, 2020

jeisenbe commented Apr 29, 2020

pnorman commented Apr 30, 2020

imagico left a comment

Choose a reason for hiding this comment

LorenzoStucchi commented Jun 11, 2020

Adamant36 commented Jun 11, 2020 • edited Loading

LorenzoStucchi commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

jeisenbe commented Jun 11, 2020 • edited Loading

LorenzoStucchi commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

LorenzoStucchi commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

jeisenbe commented Jun 11, 2020 • edited Loading

jeisenbe commented Jun 11, 2020

LorenzoStucchi commented Jun 11, 2020

jeisenbe commented Jun 11, 2020

LorenzoStucchi commented Jun 11, 2020

Adamant36 commented Jun 12, 2020

LorenzoStucchi commented Jun 12, 2020

Adamant36 commented Jun 11, 2020 •

edited

Loading

jeisenbe commented Jun 11, 2020 •

edited

Loading

jeisenbe commented Jun 11, 2020 •

edited

Loading