Various scripts for importing and running a mirror of OSM with tegola
Shell PLpgSQL
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.env-example
.gitignore
README.md
imposm3.json
natural_earth.sh
osm_land.sh
postgis_helpers.sql
postgis_index.sql
tegola-natural-earth.toml
tegola.toml

README.md

Tegola OSM

This repo houses instructions and configuration files to aid with standing up an OpenStreetMap export and Natural Earth dataset into a PostGIS enabled database that uses tegola for creating and serving vector tiles.

Repo config files

  • imposm3.json - an imposm3 mapping file for the OSM PBF file.
  • tegola.toml - a tegola configuration file for the OSM import produced by imposm3.

Dependencies

Download the OSM planet database in PBF format

curl -O https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf

Import the OSM export into PostGIS using imposm3

./imposm3 import -connection postgis://username:password@host/database-name -mapping imposm3.json -read /path/to/osm/planet-latest.osm.pbf -write
./imposm3 import -connection postgis://username:password@host/database-name -mapping imposm3.json -deployproduction

Import the OSM Land and Natural Earth dataset (requires gdal, Natural Earth can be skipped if you're only interested in OSM)

Option 1: Embed Credentials

Update the database credentials inside of natural_earth.sh and osm_land.sh, then run each file: ./natural_earth.sh && ./osm_land.sh. This will download the natural earth and osm land datasets and insert it into PostGIS under a database named natural_earth and osm respectively.

Option 2: Create a dbcredentials.sh file

Create a dbcredentials.sh file which will be shared with the osm_land script. This option is ideal for when the natural_earth and osm databases will reside on the same database server, and will use the same credentials. Ensure that the following variables are defined in your file:

DB_HOST="mydbhost"
DB_PORT="myport"
DB_USER="myuser"
DB_PW="mypassword"

Once you have configured the dbcredentials.sh file, run the scripts as above: ./natural_earth.sh && ./osm_land.sh

Option 3:

Create separate configuration files in the same pattern as the above dbcredentials.sh file and pass the path to the config file using the -c option. This is ideal if you have two different servers for the databases. Ensure the file you create follows this format:

DB_NAME="mydb"
DB_HOST="mydbhost"
DB_PORT="myport"
DB_USER="myuser"
DB_PW="mypassword"

Once you have configured the files, run the scripts with the -c flag and provide the path to the credentials file, ie: ./natural_earth.sh -c natural_earth_creds.sh && ./osm_land.sh -c osm_creds.sh

Usage:

Both scripts support a -v flag for debugging. natural_earth.sh also supports a -d flag, which will drop the existing natural earth database prior to import if set. Since the osm_land.sh imports into a database shared with other data, it lacks this functionality. Instead, only the relevent tables are dropped.

Install SQL helper functions

Execute postgis_helpers.sql against your OSM database. Currently this contains a single utility function for converting building heights from strings to numbers which is important if you want to extrude buildings for the 3d effect.

psql -U tegola -d database-name -a -f postgis_helpers.sql

Setup SQL indexes

Execute postgis_index.sql against your OSM database.

psql -U tegola -d database-name -a -f postgis_index.sql

Launch tegola

./tegola -config=tegola.toml

Open your browser to localhost and the port you configured tegola to run on (i.e. localhost:8080) to see the built in viewer.

Data Layers

To view these data layers in a map and query the features for a better understanding of each data layer, use the Tegola-OSM Inspector. The data layers described here are in the "Tegola-OSM" database as laid out in the tegola.toml (i.e., not the Natural Earth database that is specified in tegola-natural-earth.toml).

source Description
ne Natural Earth data, version 4
osm OpenStreetMap data, current
osm land OpenStreetMap-derived land polygons from openstreetmapdata.com, currentness depends on last pull

Note: All layers also have the data fields: layer id and geometry. An empty where column means that all features are retained.

populated_places

points

zoom source table/layer data fields where
0-2 ne ne_110m_populated_places scalerank, labelrank, name, min_zoom, featurecla, rank_max
3-4 ne ne_50m_populated_places scalerank, labelrank, name, min_zoom, featurecla, rank_max
5-20 ne ne_10m_populated_places scalerank, labelrank, name, min_zoom, featurecla, rank_max

country_lines

zoom source table/layer data fields where
0-2 ne ne_110m_admin_0_boundary_lines_land featurecla, name, min_zoom
3-4 ne ne_50m_admin_0_boundary_lines_land featurecla, name, min_zoom
5-10 ne ne_10m_admin_0_boundary_lines_land featurecla, name, min_zoom

country_lines_disputed

lines

zoom source table/layer data fields where
3-4 ne ne_50m_ne_50m_admin_0_boundary_lines_disputed_areas featurecla, name, min_zoom
5-10 ne ne_10m_ne_50m_admin_0_boundary_lines_disputed_areas featurecla, name, min_zoom

country_label_points

zoom source table/layer data fields where
3-20 ne ne_10m_admin_0_label_points sr_subunit, scalerank

country_polygons

zoom source table/layer data fields where
0-2 ne ne_110m_admin_0_countries featurecla, name, name_long, abbrev, adm0_a3, min_zoom, min_label, max_label
3-4 ne ne_50m_admin_0_countries featurecla, name, name_long, abbrev, adm0_a3, min_zoom, min_label, max_label
5-10 ne ne_10m_admin_0_countries featurecla, name, name_long, abbrev, adm0_a3, min_zoom, min_label, max_label

state_lines

zoom source table/layer data fields where
0-2 ne ne_110m_admin_1_states_provinces_lines featurecla, name, adm0_name, min_zoom
3-4 ne ne_50m_admin_1_states_provinces_lines featurecla, name, adm0_name, min_zoom
5-10 ne ne_10m_admin_1_states_provinces_lines featurecla, name, adm0_name, min_zoom

land

polygons

zoom source table/layer data fields where
0-2 ne ne_110m_land featurecla, min_zoom
3-4 ne ne_50m_land featurecla, min_zoom
5-7 ne ne_10m_land featurecla, min_zoom
8-20 osm land land_polygons

admin_lines

zoom source table/layer data fields where
8-12 osm admin_boundaries_8-12 admin_level, name, type admin_level IN (1,2,3,4,5,6,7,8)
13-20 osm admin_boundaries_13-20 admin_level, name, type admin_level IN (1,2,3,4,5,6,7,8,9,10)

state_label_points

zoom source table/layer data fields where
3-20 ne ne_10m_admin_1_label_points name, scalerank

landuse_areas

Nature reserves, military land, forest, leisure, wood, etc. polygons

zoom source table/layer data fields where
3-5 osm landuse_areas_gen0 name, class, type, area type IN ('forest','wood','nature reserve', 'nature_reserve', 'military') AND area > 1000000000
6-9 osm landuse_areas_gen0_6 name, class, type, area type IN ('forest','wood','nature reserve', 'nature_reserve', 'military') AND area > 100000000
10-12 osm landuse_areas_gen1 name, class, type, area
13-20 osm landuse_areas name, class, type, area

water_areas

polygons

zoom source table/layer data fields where
3-5 osm water_areas_gen0 name, class, type, area type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank') AND area > 1000000000
6-9 osm water_areas_gen0_6 name, class, type, area type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank') AND area > 100000000
10-12 osm water_areas_gen1 name, class, type, area type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank') AND area > 1000
13-20 osm water_areas name, class, type, area type IN ('water', 'pond', 'basin', 'canal', 'mill_pond', 'riverbank', 'dock')

water_lines

zoom source table/layer data fields where
8-12 osm water_lines_gen0 name, type type IN ('river', 'canal')
13-14 osm water_lines_gen1 name, type type IN ('river', 'canal', 'stream', 'ditch', 'drain', 'dam')
15-20 osm water_lines name, type type IN ('river', 'canal', 'stream', 'ditch', 'drain', 'dam')

transport_lines

Roads, airport runways, ferry routes, paths, etc.

zoom source table/layer data fields where
3-4 ne ne_10m_roads_3 name, min_zoom, min_label, type, label min_zoom < 5 AND type <> 'Ferry Route'
5-6 ne ne_10m_roads_5 name, min_zoom, min_label, type, label min_zoom <= 7 AND type <> 'Ferry Route'
7-8 osm transport_lines_gen0 type, tunnel, bridge, ref type IN ('motorway','trunk','motorway_link','trunk_link','primary') AND tunnel = 0 AND bridge = 0
9-10 osm transport_lines_gen1 ref, class, type type IN ('motorway', 'trunk', 'primary', 'primary_link', 'secondary', 'motorway_link', 'trunk_link')
11-12 osm transport_lines_11-12 name, ref, class, type, tunnel, bridge, access, service type IN ('motorway', 'motorway_link', 'trunk', 'trunk_link', 'primary', 'primary_link', 'secondary', 'secondary_link', 'tertiary', 'tertiary_link', 'rail', 'taxiway', 'runway', 'apron')
13 osm transport_lines_13 name, ref, class, type, tunnel, bridge, access, service type IN ('motorway', 'motorway_link', 'trunk', 'trunk_link', 'primary', 'primary_link', 'secondary', 'secondary_link', 'tertiary', 'tertiary_link', 'rail', 'residential', 'taxiway', 'runway', 'apron')
14-20 osm transport_lines_14-20 name, ref, class, type, tunnel, bridge, access, service

transport_areas

Airports, etc. polygons

zoom source table/layer data fields where
12-20 osm transport_areas name, class, type

transport_points

Airports, helipads, etc.

zoom source table/layer data fields where
14-20 osm transport_points name, class, type

amenity_areas

Fire stations, banks, embassies, government, police stations, schools, universities, etc. polygons

zoom source table/layer data fields where
14-20 osm amenity_areas name, type

amenity_points

Fire stations, banks, embassies, government, police stations, schools, universities, etc.

zoom source table/layer data fields where
14-20 osm amenity_points name, type

other_points

Man made, historic, military, barriers, power towers, etc.

zoom source table/layer data fields where
14-20 osm other_points name, class, type

other_lines

Man made, historic, military, barriers, power lines, etc.

zoom source table/layer data fields where
14-20 osm other_lines name, class, type

other_areas

polygons Man made, historic, military, power, barriers, piers, etc.

zoom source table/layer data fields where
6-8 osm other_areas_filter name, class, type area > 1000000
9-20 osm other_areas name, class, type

buildings

polygons

zoom source table/layer data fields where
14-20 osm buildings name, height, type

How long does it take to import the entire planet?

If you run this import, please send in a PR to report your import machine specs and how long it takes.

@peldhose: 11.30 hours on a Google cloud server with 8 vCPU, 30GB RAM and 1TB storage (400GB used)