tuning & monitoring

[hugokoopmans]

hi there, I have set up a tileserver for our wildlife protection organisation Sensing Clues (see here ).
I have set up a docker host on a 4 core 16 GB RAM server.
The tileserver is setup and I imported 7 countries ( NL, Tanzanie, Kenia, South Africa, Namibia, Zambia, Zimbabwe) I expect to load more countries soon.

My question/issue is how do monitor what is going on on the tile server?

I have pre-renderd tiles like so:

render_list -n 8 -a  -f  -m ajt  -z 9  -Z 15

And tested with some zoom levels from 0 to x but if I look at the volumes in docker with

docker system df -v

I see 57 GB of postgresql data but only see 350 MB of renderd tiles?

VOLUME NAME LINKS SIZE
9bb722d5661210d1cc218bfa29f951453f01b6344f2999fdd92661bb890bb919 1 424B
openstreetmap-data 1 57.25GB
openstreetmap-rendered-tiles 1 334.9MB

No matter what I do with render_list seems it does not pre-render any more tiles?

Memory usage:

CONTAINER ID        NAME                                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
acb8612ddd66        tileserver                          395.06%             3.697GiB / 15.66GiB   23.60%              10.9MB / 128MB      30GB / 262MB        235
69f87448cb4a        registry                            0.00%               4.355MiB / 15.66GiB   0.03%               9.73MB / 99.2MB     2.15GB / 0B         13
872f5d1e6647        portainer                           0.00%               8.332MiB / 15.66GiB   0.05%               9.91MB / 59.3MB     7.73GB / 93.3MB     14
9a5bc06c2d41        hello                               0.00%               2.086MiB / 15.66GiB   0.01%               31.9kB / 0B         4.33MB / 0B         2
84ffec2790ff        letsencrypt-nginx-proxy-companion   0.12%               11.08MiB / 15.66GiB   0.07%               130kB / 70.5kB      3.68GB / 73.7kB     18
f37924d45aa6        nginx-proxy                         0.12%               14.43MiB / 15.66GiB   0.09%               335MB / 350MB       3.69GB / 377kB      33

If i look at htop postgresql seems busy, also with some jobs over more then 3 hours ????

 renderer gis [local] select

I in a previous post i played around with THREADS and shm_size the docker-compose yml

Assumptions / Questions:

1. As I have only 4 cores THREADS > 8 seems not usefull? I do not see any impact
2. As I have 16 GB of RAM and this is a more or less dedicated tileserver I hardly see the memory usages exceed 25% . How can I make use of more memory?
3. I am on postgres 10 is the latest image on 11 available? I have seen moderate speed increas but especially on the import part, does it also speed up rendering itself?
4. Are there any ways to monitor and understand better if and how the renderd process actually has renderd tiles? How many tiles are actually prerenderd etc etc. With 335 MB in the renderd_tiles folder something seems not right?
5. I have seen posts that pre_rendering takes a lot of time, but that is in hours not in days my tileserver is now up for quite a while (week) I have uses render_list from within docker via exec but again I do not know when it is finished what result to expect and how to optimize pre-rendering.

For completeness my current docker-compose file:

version: '3'
services:
  tileserver:
    container_name: tileserver
    image: overv/openstreetmap-tile-server
    restart: always
    ports:
        - "8099:80"
    volumes:
        - 'openstreetmap-data:/var/lib/postgresql/10/main'
        - 'openstreetmap-rendered-tiles:/var/lib/mod_tile'
        - '/data/docker/tileserver:/data'
    environment:
        THREADS: 8
    shm_size: 256M
    command: run

volumes:
  openstreetmap-data:
    external: true
  openstreetmap-rendered-tiles:
    external: true

[Istador]

4. Are there any ways to monitor and understand better if and how the renderd process actually has renderd tiles? How many tiles are actually prerenderd etc etc. With 335 MB in the renderd_tiles folder something seems not right?

I use (among other tools) munin to monitor a few servers. Luckily mod_tile does come with munin plugin scripts to monitor the rendering process. So they should already be present inside your container, you might just need to execute them properly.

apache2 and postgresql, that run inside the container, can also be monitored by munin, but for those two there are also other tool(s) around to monitor them.

---

To inspect what currently is being worked on, there is output of the renderd process that is collected by docker and can be watched with it:

$ docker-compose logs --follow tileserver

BTW: docker does save but not rotate these logs by default, so it (slowly) grows and grows and grows over time until the harddisk (of /var/lib/docker/) is full. It might make sense to configure a log rotation for it, e.g. like this:

services:
  tileserver:
    # ...
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "10"

As for the apache2 and postresql services running inside of the container: they are started with service so their runtime output should by accessible via journalctl, but both should also have logfiles inside /var/log/ by default.

[hugokoopmans]

ok seems my renderd remains very busy all the time:

renderd[121]: DEBUG: START TILE ajt 9 256-263 160-167, age 3.17 days
renderd[121]: Rendering projected coordinates 9 256 160 -> 0.000000|6887893.492837 626172.135713|7514065.628550 to a 8 x 8 tile
renderd[121]: DEBUG: DONE TILE ajt 13 4216-4223 2712-2719 in 661.364 seconds
debug: Creating and writing a metatile to /var/lib/mod_tile/ajt/13/0/16/10/121/136.meta

renderd[121]: DEBUG: START TILE ajt 9 256-263 168-175, age 3.17 days
renderd[121]: Rendering projected coordinates 9 256 168 -> 0.000000|6261721.357125 626172.135713|6887893.492837 to a 8 x 8 tile
renderd[121]: DEBUG: DONE TILE ajt 14 8432-8439 5416-5423 in 758.299 seconds
debug: Creating and writing a metatile to /var/lib/mod_tile/ajt/14/0/33/5/242/8.meta

renderd[121]: DEBUG: START TILE ajt 16 33760-33767 21704-21711, age 4.61 days
renderd[121]: Rendering projected coordinates 16 33760 21704 -> 606604.256471|6760702.277771 611496.226282|6765594.247581 to a 8 x 8 tile
renderd[121]: DEBUG: DONE TILE ajt 16 33760-33767 21696-21703 in 998.452 seconds
debug: Creating and writing a metatile to /var/lib/mod_tile/ajt/16/0/133/52/236/0.meta

In this post it seems that tiles are re-renderd every 3 days by default? That is not necessary for my application so I need to make sure the rerendering does not take place. My 4 core server seems now very bussy keeping up rerendering all tiles untill zoom level 15. It seems it cannot keep up ... does that make sense to you?

[hugokoopmans]

ok so we have " To allow re-rendering but not have the client wait for such tiles, set your ModTileRequestTimeout to 0 instead of 3".

In the default acpache.conf we have ModTileRequestTimeout = 0 right?

[crnm21]

3 days is default for declaring tiles old if not configured otherwise.
Easy solution: in /var/lib/mod_tile and/or /var/lib/mod_tile/ajt create a file called planet-import-complete with the creation date of your last postgres db import.
(e.g.: touch -d "2 days ago" planet-import-complete )
With this file present, mod_tile won't rerender tiles if they are younger than the planet-import-complete filetime. (Which means you'll have to use other techniques to update tiles).

(btw you could have read about this solution in the help thread you linked to yourself).

[hugokoopmans]

1. Are there any ways to monitor and understand better if and how the renderd process actually has renderd tiles? How many tiles are actually prerenderd etc etc. With 335 MB in the renderd_tiles folder something seems not right?

I use (among other tools) munin to monitor a few servers. Luckily mod_tile does come with munin plugin scripts to monitor the rendering process. So they should already be present inside your container, you might just need to execute them properly.

apache2 and postgresql, that run inside the container, can also be monitored by munin, but for those two there are also other tool(s) around to monitor them.

To inspect what currently is being worked on, there is output of the renderd process that is collected by docker and can be watched with it:

$ docker-compose logs --follow tileserver

BTW: docker does save but not rotate these logs by default, so it (slowly) grows and grows and grows over time until the harddisk (of /var/lib/docker/) is full. It might make sense to configure a log rotation for it, e.g. like this:

services:
  tileserver:
    # ...
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-files: "10"

As for the apache2 and postresql services running inside of the container: they are started with service so their runtime output should by accessible via journalctl, but both should also have logfiles inside /var/log/ by default.

seems I am doing something wrong while trying to create the log rotation

ERROR: for tileserver  Cannot create container for service tileserver: unknown log opt 'max-files' for json-file log driver

?

my complete docker-compose:

version: '3'
services:
  tileserver:
    container_name: tileserver
    image: overv/openstreetmap-tile-server
    restart: always
    ports:
        - "8099:80"
    volumes:
        - 'openstreetmap-data:/var/lib/postgresql/10/main'
        - 'openstreetmap-rendered-tiles:/var/lib/mod_tile'
        - '/data/docker/tileserver:/data'
        # get localtime from host
        - '/etc/timezone:/etc/timezone:ro'
        - '/etc/localtime:/etc/localtime:ro'

    environment:
        VIRTUAL_HOST: 'tiles.x.org'
        LETSENCRYPT_HOST: 'tiles.x.org'
        LETSENCRYPT_EMAIL: hugo.koopmans@x.org
        THREADS: 8
        AUTOVACUUM: 'off'
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-files: "10"
# test certs
#        LETSENCRYPT_TEST: 'true'
#        ACME_CA_URI: 'https://acme-staging.api.letsencrypt.org/directory'
    shm_size: 256M
    command: run

volumes:
  openstreetmap-data:
    external: true
  openstreetmap-rendered-tiles:
    external: true

any suggestions?

[Istador]

seems I am doing something wrong while trying to create the log rotation

ERROR: for tileserver  Cannot create container for service tileserver: unknown log opt 'max-files' for json-file log driver

?

[...]

any suggestions?

My bad, it's max-file and not max-files.
https://docs.docker.com/compose/compose-file/#logging

[hugokoopmans]

Hi @Istador thx for your help. I am still strugling with a proper setup, especially now we keep adding countries to the tileserver as new customers come in .

I now have Netherlands, 7 African countries en recently added India Portugal Spain and Italy.

Now I upgraded my machine to be 8 cores 32 GB 200GB disk so still rather small (but sponsored so not 100% in my control). We are serving Natureparks in these countries, sometimes also interested in corridors between Narute parks.

Last night I bumped into this:

Osm2pgsql failed due to ERROR: Connection to database failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

So apparently 16 THREADS is to much for the default max_connections of 250 if I understand correctly?

In your reply to issue 5 you specked out some details on your setup, did you already get consent from your employer to share your setup detail ;-) ?

I would be most interested in a postgresql conf file and docker file that work smoothly on a machine more or less my size...

I think I would like to create my own image using postgresql 11 but overwhelmed by all the options and settings in the conf files to make it work smoothly.

[Istador]

Last night I bumped into this:

Osm2pgsql failed due to ERROR: Connection to database failed: FATAL:  remaining connection slots are reserved for non-replication superuser connections

So apparently 16 THREADS is to much for the default max_connections of 250 if I understand correctly?

Well, the default value for max_connections is 100. If you're running the newest version of this docker image (from 2019-04-23), then it was increased to 250 by introducing a custom postgresql.conf.

I don't know out of my head which value I used in my setup, that was in July 2018 (when Google Maps drastically changed their pricing model).

According to osm2pgsql-dev/osm2pgsql#885 the amount of connections depends on the amount of threads running and tables in the database (7 or more). Because you faced this error with osm2pgsql, please note that if you're serving tiles with renderd (16 threads) and at the same time append to the same database with osm2pgsql (16 threads), then your total thread count is at least 32 (or more, depending on what else is accessing the database (background tasks, monitoring, etc.)). Theoretically with 7 x 32 = 224 you should be fine with a setting of 250 for max_connections, but apparently you're not.

(Please verify if you're running the latest version of this docker image or not. Working with an existing system that is/was running the old version, requires an docker-compose pull to download the latest image version and docker-compose up -d to recreate and restart the container.)

• Number Of Database Connections - PostgreSQL wiki
• Tuning Your PostgreSQL Server - PostgreSQL wiki

In your reply to issue 5 you specked out some details on your setup, did you already get consent from your employer to share your setup detail ;-) ?

Nope, nothing new on that. I asked my boss two or three times about it (last time in March), but only got the reply that the request will be forwarded to our CEO.

The setup details that I shared already aren't that much of a problem, because I don't disclose anywhere who my employer is and even then this information wouldn't be that sensitive, but I have to operate based on my memory in my free time and can not dedicate paid working time on it.

The only problem I have is, that I can't contribute any commits/changes that I did for the setup (e.g. new environment variables for memory and flatnodes, changes to run.sh to update/extend the database and expire/re-render tiles) to this project or an public fork without approval (because my work contract grants my employer all usage rights on the work products I develop for him).