Install Rendering (Ubuntu 16.04)

Jonathan Beliën edited this page Apr 30, 2018 · 28 revisions

The purpose to this server is to render tiles. The tiles for https://tile.osm.be/ are streamed by another server.

Requirements

  • Server with at least 30Go RAM
  • Server with at least 8 (v)CPU
  • Ubuntu 16.04 (obviously)

Update operating system

Just to be sure everything is up to date :

sudo apt-get update
sudo apt-get dist-upgrade

And restart your server :

sudo reboot

Install

Install required softwares and libraries

sudo apt-get install tar unzip wget bzip2 build-essential autoconf protobuf-c-compiler make cmake g++ \
                     libboost-all-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev libprotobuf-c0-dev \
                     libxml2-dev libfreetype6-dev libpng12-dev libtiff5-dev libicu-dev libgdal-dev libcairo-dev \
                     libtool libcairomm-1.0-dev libagg-dev libgeotiff-epsg libboost-dev libboost-system-dev \
                     libboost-filesystem-dev libexpat1-dev zlib1g-dev\
                     lua5.3 liblua5.3-dev \
                     apache2 apache2-dev \
                     munin \
                     fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted fonts-hanazono ttf-unifont \
                     gdal-bin libgdal1-dev \
                     libmapnik-dev mapnik-utils python-mapnik \
                     python-yaml

Install PostgreSQL & PostGIS

sudo add-apt-repository 'deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install -y postgresql-10 postgresql-10-postgis-2.4

Install NodeJS & CartoCSS (Mapnik Stylesheet Compiler)

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g carto

Create database

sudo -u postgres createuser -s $USER
createdb gis
psql -d gis -c 'CREATE EXTENSION postgis; CREATE EXTENSION hstore;'

Install osm2pgsql

git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql
mkdir build && cd build
cmake ..
make
sudo make install

Install mod_tile

sudo mkdir /var/lib/mod_tile
sudo chown $USER /var/lib/mod_tile

git clone git://github.com/SomeoneElseOSM/mod_tile.git
cd mod_tile
./autogen.sh
./configure
make
sudo make install
sudo make install-mod_tile
sudo ldconfig

Create mod_tile module for Apache

Create /etc/apache2/mods-available/tile.load file :

LoadModule tile_module /usr/lib/apache2/modules/mod_tile.so

Create /etc/apache2/mods-available/tile.conf file :

<IfModule tile_module>
   LoadTileConfigFile /usr/local/etc/renderd.conf
   ModTileRenderdSocketName /var/run/renderd/renderd.sock
   # Timeout before giving up for a tile to be rendered
   ModTileRequestTimeout 0
   # Timeout before giving up for a tile to be rendered that is otherwise missing
   ModTileMissingRequestTimeout 30
</IfModule>

Update mod_tile configuration file :

Update needed parameters :

  • num_threads : should match the number of (v)CPU
  • plugins_dir : use mapnik-config --input-plugins to get the directory where Mapnik plugins are stored

Add following to file /usr/local/etc/renderd.conf :

[osmbe]
URI=/osmbe/
TILEDIR=/var/lib/mod_tile
XML=/home/ubuntu/src/openstreetmap-carto-be/mapnik.xml
HOST=localhost
TILESIZE=256
MAXZOOM=18
CORS=*

[osmbe-fr]
URI=/osmbe-fr/
TILEDIR=/var/lib/mod_tile
XML=/home/ubuntu/src/openstreetmap-carto-be/mapnik-fr.xml
HOST=localhost
TILESIZE=256
MAXZOOM=18
CORS=*

[osmbe-nl]
URI=/osmbe-nl/
TILEDIR=/var/lib/mod_tile
XML=/home/ubuntu/src/openstreetmap-carto-be/mapnik-nl.xml
HOST=localhost
TILESIZE=256
MAXZOOM=18
CORS=*

Enable mod_tile :

sudo a2enmod tile
sudo service apache2 restart

Render

Clone openstreetmap-carto-be repository

git clone git://github.com/jbelien/openstreetmap-carto-be.git

Clone render_list_geo.pl repository

git clone git://github.com/alx77/render_list_geo.pl.git

Run the render process

To render osmbe :

sh openstreetmap-carto-be/scripts/render/process.sh

To render osmbe-fr :

sh openstreetmap-carto-be/scripts/render/process.sh fr

To render osmbe-nl :

sh openstreetmap-carto-be/scripts/render/process.sh nl

If you want to know in details what will happen, check https://github.com/jbelien/openstreetmap-carto-be/blob/master/scripts/render/process.sh !

Timing (30GB RAM - 8 vCPU)

Belgium

Test done on 2018-04-20 for osmbe rendering.

  • Import to PostgreSQL : 728 seconds
  • Rendering :
    • Levels 0-7 : Rendered 22016 tiles in 24.41 seconds (901.83 tiles/s) - Whole World
    • Level 8 : Rendered 64 tiles in 10.40 seconds (6.15 tiles/s) - Belgium only
    • Level 9 : Rendered 128 tiles in 13.99 seconds (9.15 tiles/s)
    • Level 10 : Rendered 384 tiles in 56.52 seconds (6.79 tiles/s)
    • Level 11 : Rendered 768 tiles in 37.82 seconds (20.31 tiles/s)
    • Level 12 : Rendered 2240 tiles in 36.70 seconds (61.04 tiles/s)
    • Level 13 : Rendered 7680 tiles in 57.38 seconds (133.84 tiles/s)
    • Level 14 : Rendered 29440 tiles in 81.37 seconds (361.82 tiles/s)
    • Level 15 : Rendered 117760 tiles in 198.16 seconds (594.26 tiles/s)
    • Level 16 : Rendered 471040 tiles in 591.69 seconds (796.10 tiles/s)
    • Level 17 : Rendered 1850496 tiles in 2036.63 seconds (908.61 tiles/s)
    • Level 18 : Rendered 7381760 tiles in 7343.25 seconds (1005.24 tiles/s)

Belgium and surroundings

Test done on 2018-04-25 for osmbe rendering.

  • Import to PostgreSQL : 5995 seconds
  • Rendering :
    • Levels 0-7 : Rendered 22016 tiles in 47.04 seconds (468.04 tiles/s) - Whole World
    • Level 8 : Rendered 64 tiles in 50.14 seconds (1.28 tiles/s) - Belgium only
    • Level 9 : Rendered 128 tiles in 39.01 seconds (3.28 tiles/s)
    • Level 10 : Rendered 384 tiles in 203.99 seconds (1.88 tiles/s)
    • Level 11 : Rendered 768 tiles in 123.30 seconds (6.23 tiles/s)
    • Level 12 : Rendered 2240 tiles in 71.15 seconds (31.48 tiles/s)
    • Level 13 : Rendered 7680 tiles in 105.70 seconds (72.66 tiles/s)
    • Level 14 : Rendered 29440 tiles in 137.80 seconds (213.64 tiles/s)
    • Level 15 : Rendered 117760 tiles in 308.39 seconds (381.86 tiles/s)
    • Level 16 : Rendered 471040 tiles in 770.40 seconds (611.43 tiles/s)
    • Level 17 : Rendered 1850496 tiles in 2431.56 seconds (761.03 tiles/s)
    • Level 18 : Rendered 7381760 tiles in 8632.45 seconds (855.12 tiles/s)
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.