Skip to content
On Request Vector Tiles server based on OpenMapTiles and Kartotherian
JavaScript Shell Dockerfile HTML CSS Ruby
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
artillery Replace own redis cache by nginx reverse proxy cache Jan 3, 2020
kartotherian Replace own redis cache by nginx reverse proxy cache Jan 3, 2020
nginx Overpass the nginx limit request for tiles expiration Feb 13, 2020
openmaptiles @ 7b0dbd0 Update openmaptiles Jan 30, 2020
scripts
.gitignore Remove git submodules as gh-pages Jan 10, 2020
.gitmodules Remove git submodules as gh-pages Jan 10, 2020
LICENSE
README.md Review README Jan 13, 2020
docker-compose-benchmark.yml Split docker-compose.yml for graphite & artillery Jan 6, 2020
docker-compose.yml Split docker-compose.yml for graphite & artillery Jan 6, 2020
screen.jpeg Add screen capture Nov 14, 2019

README.md

Makina Maps

On Request Vector Tiles server based on OpenMapTiles and Kartotherian with the ability to:

  • Build Vector Tiles on Demand from the OpenMapTiles database
  • Served Mapbox GL Style, with sprites and fonts
  • Render Raster version of Mapbox GL style
  • Cache vector and Raster tiles

screen

Install

Install as system dependencies: git, make, docker and docker-compose.

Clone Git reposittory

Get the project:

git clone --recurse-submodules https://github.com/makinacorpus/makina-maps.git
cd makina-maps

Get GL Json Styles et fonts:

git clone -b gh-pages https://github.com/openmaptiles/osm-bright-gl-style.git styles/osm-bright-gl-style
git clone -b gh-pages https://github.com/openmaptiles/klokantech-basic-gl-style.git styles/klokantech-basic-gl-style
git clone -b gh-pages https://github.com/openmaptiles/fonts.git

Setup Docker images

Waiting for next OpenMapTiles Tools release, compile one module to allow usage of imposm config at import-osm step. From any directory:

git clone -b imposm-0.8.1 https://github.com/frodrigo/openmaptiles-tools.git
cd openmaptiles-tools/docker/import-osm
docker build -t openmaptiles/import-osm:3.1.0-imposm-0.8.1 .
cd ../../..

Fetch or build docker images:

docker-compose build
cd openmaptiles
# Ignore failures as we temporary provide an image manualy
docker-compose pull --ignore-pull-failures

Prepare OpenMapTiles

Prepare OpenMapTiles configuration:

cd openmaptiles
make

Import OpenStreetMap Data

OpenMapTiles initial Load

Use the management scripts from openmaptiles directory.

cd openmaptiles

Import generic data, not from OpenStreetMap:

../scripts/10-import-generic.sh

Prepare import by download OpenStreetMap data and setup configuration for an area. The area names are from Geofabrik:

../scripts/20-import-prepare.sh europe/andorra

Import the OpenStreetMap extract from data directory:

../scripts/30-import-extract.sh

The scripts 20-import-prepare.sh or 30-import-extract.sh can be replayed with the same or other area.

Update OpenMapTiles data

From the openmaptiles directory.

Run the updater. It loops over pending updates, then wait for new update.

../scripts/40-update.sh

You can stop with CTRL-C, it will quit at the end of the current update.

Imposm marks tiles to expire. Then a script in the nginx container watches and expires tiles in the nginx cache.

Run the tiles server

From root directory. Start the OpenMapTiles database and the web server.

(cd openmaptiles && docker-compose up -d postgres) && sleep 10
docker-compose up

Access to cached tiles and services at:

Configuration

Configuration files are Kartotherian configuration files config.yaml and sources.yaml.

config.yaml

Specific configuration part.

services:
  - name: kartotherian
      variables:
        OPENMAPTILES_V3_TILES_URLS:
        - http://localhost:8080/openmaptiles_v3/{z}/{x}/{y}.pbf
        - http://127.0.0.1:8080/openmaptiles_v3/{z}/{x}/{y}.pbf
        - http://[::1]:8080/openmaptiles_v3/{z}/{x}/{y}.pbf
        OPENMAPTILES_V3_TILES_URLS_INTERNAL:
        - http://nginx:80/openmaptiles_v3_raster/{z}/{x}/{y}.pbf
        BRIGHT_TILES_URLS:
        - http://localhost:8080/bright/{z}/{x}/{y}.png
        - http://127.0.0.1:8080/bright/{z}/{x}/{y}.png
        - http://[::1]:8080/bright/{z}/{x}/{y}.png

      modules:
      - "tilelive-tmstyle"
      - "@kartotherian/tilelive-tmsource"
      - "@mapbox/tilejson"
      - kartotherian_gl # Render raster tiles

      requestHandlers:
      - kartotherian_gl_style_server # Serve Mapbox GL Style
      - kartotherian_sources_list_server # Serve the list of sources

      sources_server:
        prefix_public: http://localhost:8080 # External hostname, should be changed to https://example.com

      styles:
        prefix_public: http://localhost:8080 # External hostname, should be changed to https://example.com
        prefix_internal: http://kartotherian:6533 # Internal, required for render raster
        paths:
          styles: /styles # Path to styles, in the Docker container
          fonts: /fonts # Path to fonts, in the Docker container
        font_fallback: Klokantech Noto Sans Regular # Serve this font when the font is not found
        styles:
          basic: # Name of the style
            style: klokantech-basic-gl-style/style-local.json # Relative path the style JSON
            sources_map: # Map of the style source (`mbtiles://{v3}`) to source name from `sources.yaml`
              v3: openmaptiles_v3
            sources_map_internal: # Internal, required for render raster
              v3: openmaptiles_v3_raster

styles.yaml

Mapbox GL native raster

source_name: # Unique source identifier
  public: true
  formats: [png, jpeg, webp]
  uri: kartotherian+gl:///
  params:
    style: bright # Style defined in to `config.yaml`
  overrideInfo:
    tiles: {var: BRIGHT_TILES_URLS} # From `config.yaml`

Benchmark

Import size and time

Specific only on 8 CPUs (import-osm, import-sql and psql-analyze, without docker pulling time).

Area PBF size Imposm cache Postgres size Time 8 CPUs / SSD Time 4 CPUs / HD 1 d Update
Andorra 243 Ko 3.3 Mo 3.5 Go 36 s 1 min 21 s
Alsace 100 Mo 156 Mo 4.5 Go 3 min 20 s 4 min 32 s
Aquitaine 214 Mo 374 Mo 6.4 Go 6 min 40 s 8 m 39 s 1 min
Austria 559 Mo 781 Mo 9.4 Go 23 min 26 min 35 s
France 3.5 Go 35 Go 105 min 210 min 58 s
Europe 20 Go 35 Go 206 Go 12 h 22 min 3 d 23 h 3 h

Database

Size of Imposm cache.

docker-compose run import-osm bash -c "du -h /cache/"

Size of the current database.

docker-compose exec postgres psql openmaptiles openmaptiles -c "
SELECT
  pg_size_pretty(sum(pg_relation_size(pg_catalog.pg_class.oid))::bigint) as table_size
FROM
  pg_catalog.pg_class
  JOIN pg_catalog.pg_namespace ON
    relnamespace = pg_catalog.pg_namespace.oid
WHERE
  pg_catalog.pg_namespace.nspname = 'public';
"

Show slow queries

ALTER DATABASE openmaptiles SET log_min_duration_statement = 100;

And grep slowest queries

docker logs openmaptiles_postgres_1 |& grep 'LOG:  duration:' | cut -d ':' -f 3- | sed 's/BOX3D([^)]*)//g' | sort -n

Metrics

Server metrics could be available on StatsD / Graphite on http://localhost:8899

The metrics logs are not used by default and could be enabled by adding docker-compose-benchmark.yml to docker-compose and uncommenting the metrics section from config.yaml.

Queries

Use Artillery.io to benchmark the tiles server. Use docker-compose-benchmark.yml.

Generate one set of tiles coordinates to be requested. Coordinates are tile ranges at zoom level 14 (https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/)

# Continantal Europe
docker-compose -f docker-compose.yml -f docker-compose-benchmark.yml run --rm artillery bash -c 'ruby artillery.rb 8445-9451 5356-5891 | egrep "^14," > artillery.csv'
# Aquitaine
docker-compose -f docker-compose.yml -f docker-compose-benchmark.yml run --rm artillery bash -c 'ruby artillery.rb 8000-8200 5800-6000 | egrep "^14," > artillery.csv'
# Bordeaux area
docker-compose -f docker-compose.yml -f docker-compose-benchmark.yml run --rm artillery bash -c 'ruby artillery.rb 8157-8171 5895-5909 | egrep "^14," > artillery.csv'
# Paris area
docker-compose -f docker-compose.yml -f docker-compose-benchmark.yml run --rm artillery bash -c 'ruby artillery.rb 8285-8311 5621-5645 | egrep "^14," > artillery.csv'

Clear the tiles cache first. Then request the tiles server.

docker-compose run --rm nginx rm -fr /cache/*
docker-compose -f docker-compose.yml -f docker-compose-benchmark.yml run --rm artillery artillery run artillery.yaml

Random order tiles request on mixed urban and rural area, without concurrency. Time from server, HTTP included.

Source Delay
Zoom 12 mixte Europe 50 ms
Zoom 13, mixte Europe 49 ms
Zoom 14, mixte Europe 60 ms
Zoom 14, urban Paris 250 ms
From cache 5 ms
You can’t perform that action at this time.