Setting OVERPASS_META to "no" results in startup error.

[MichielDeMey]

This might be a regression from be98c86.

Setting the environment variable OVERPASS_META to no results in the following error message when trying to boot the dispatcher:

File /db/db/nodes_meta.bin present. Please use parameter --meta
File /db/db/ways_meta.bin present. Please use parameter --meta
File /db/db/relations_meta.bin present. Please use parameter --meta
File /db/db/user_data.bin present. Please use parameter --meta
File /db/db/user_indices.bin present. Please use parameter --meta

I believe this issue is here: https://github.com/wiktorn/Overpass-API/blob/master/docker-entrypoint.sh#L45

According to the help message, and this comment drolbr/Overpass-API#446 (comment) we should either omit or include the --meta flag.

Usage:  /app/bin/init_osm3s.sh  Planet_File  Database_Dir  Executable_Dir  [--meta]
        where
    Planet_File is the filename and path of the compressed planet file, including .bz2,
    Database_Dir is the directory the database should go into, and
    Executable_Dir is the directory that contains the executable update_database.
    Add --meta in the end if you want to use meta data.

[wiktorn]

This looks to me like you've created your overpass instance with OVERPASS_META=yes and then mounted the same directory to a container with OVERPASS_META=no. Is this the case?

This will not work, you can't change OVERPASS_META parameter on existing data, as either you have surplus data or you would be lacking the data. This is safeguard so you will not have some inconsistencies.

[MichielDeMey]

Unfortunately that’s not the case. The issue occurs on a clean boot of the container. No previous files yet, so a clean init.

[wiktorn]

Clean boot of container, but did you mount anything under /db ? Clean container doesn't contain any files there and you have messages:

File /db/db/nodes_meta.bin present

Suggesting, that something exists there.

What I'm saying, is you need to maintain the same value of OVERPASS_META for all containers using the same /db folder.

[MichielDeMey]

Nothing mounted under db, but container starts with OVERPASS_MODE=init and OVERPASS_META=no. Takes about an hour and a half to process and finally fails with the error message about the meta files.

[wiktorn]

I can't reproduce this, when starting the container using following commands:

docker run --restart=always \
  -e OVERPASS_META=no \
  -e OVERPASS_MODE=init \
  -e OVERPASS_PLANET_URL=http://download.geofabrik.de/europe/monaco-latest.osm.bz2 \
  -e OVERPASS_DIFF_URL=http://download.geofabrik.de/europe/monaco-updates/ \
  -e OVERPASS_RULES_LOAD=10 \
  -e OVERPASS_COMPRESSION=gz \
  -p 12346:80 \
  -i -t \
  --name overpass_monaco wiktorn/overpass-api

Monaco extract is really small, so it's good for testing.

Can you provide full docker command that you issue?

[MichielDeMey]

I had to apply the following PR to make the container boot in the first place: #15

After that, I've used the exact same Docker command as you suggested above and here is the output:

(...)

compute_geometry: Way 42959211 used in relation 8766644 not found.
Flushing to database ....... done.
Update complete.
There are still some updates remaining
INFO: Using replication server at http://download.geofabrik.de/europe/monaco-updates/
DEBUG: Using given sequence ID 2259
DEBUG: Starting download at ID 2260 (max 100 MB)
/app/bin/update_database --compression-method=gz --map-compression-method=gz --flush-size=1 --db-dir=/db/db
Reading XML file ...Update complete.
Update finished with status code: 3
Overpass ready, you can start your container with docker start
2019-05-27 11:49:27,851 CRIT Supervisor running as root (no user in config file)
2019-05-27 11:49:27,855 INFO supervisord started with pid 1
2019-05-27 11:49:28,862 INFO spawned: 'overpass_dispatch' with pid 8
2019-05-27 11:49:28,865 INFO spawned: 'nginx' with pid 9
2019-05-27 11:49:28,867 INFO spawned: 'fcgiwrap' with pid 10
2019-05-27 11:49:28,869 INFO spawned: 'update_overpass' with pid 11
2019-05-27 11:49:28,871 INFO spawned: 'dispatcher_areas' with pid 12
2019-05-27 11:49:28,873 INFO spawned: 'areas_rules' with pid 13
2019-05-27 11:49:28,903 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-27 11:49:30,120 INFO spawned: 'overpass_dispatch' with pid 45
2019-05-27 11:49:30,121 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:49:30,121 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:49:30,122 INFO success: update_overpass entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:49:30,122 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:49:30,122 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:49:30,136 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-27 11:49:32,943 INFO spawned: 'overpass_dispatch' with pid 56
2019-05-27 11:49:32,956 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-27 11:49:35,972 INFO spawned: 'overpass_dispatch' with pid 66
2019-05-27 11:49:35,987 INFO exited: overpass_dispatch (exit status 0; not expected)
2019-05-27 11:49:36,991 INFO gave up: overpass_dispatch entered FATAL state, too many start retries too quickly
2019-05-27 11:52:49,727 WARN received SIGTERM indicating exit request
2019-05-27 11:52:49,727 INFO waiting for fcgiwrap, nginx, update_overpass, areas_rules, dispatcher_areas to die
2019-05-27 11:52:49,727 INFO stopped: areas_rules (terminated by SIGTERM)
2019-05-27 11:52:49,728 INFO stopped: dispatcher_areas (terminated by SIGTERM)
2019-05-27 11:52:49,728 INFO stopped: update_overpass (terminated by SIGTERM)
2019-05-27 11:52:49,729 INFO stopped: fcgiwrap (terminated by SIGTERM)
2019-05-27 11:52:49,824 INFO stopped: nginx (exit status 0)
2019-05-27 11:52:51,716 CRIT Supervisor running as root (no user in config file)
2019-05-27 11:52:51,724 INFO supervisord started with pid 1
2019-05-27 11:52:52,728 INFO spawned: 'overpass_dispatch' with pid 8
2019-05-27 11:52:52,731 INFO spawned: 'nginx' with pid 9
2019-05-27 11:52:52,735 INFO spawned: 'fcgiwrap' with pid 10
2019-05-27 11:52:52,739 INFO spawned: 'update_overpass' with pid 13
2019-05-27 11:52:52,742 INFO spawned: 'dispatcher_areas' with pid 14
2019-05-27 11:52:52,744 INFO spawned: 'areas_rules' with pid 15
2019-05-27 11:52:52,764 INFO exited: overpass_dispatch (exit status 3; not expected)
2019-05-27 11:52:53,984 INFO spawned: 'overpass_dispatch' with pid 46
2019-05-27 11:52:53,986 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:52:53,987 INFO success: fcgiwrap entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:52:53,988 INFO success: update_overpass entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:52:53,988 INFO success: dispatcher_areas entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:52:53,989 INFO success: areas_rules entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-05-27 11:52:54,006 INFO exited: overpass_dispatch (exit status 3; not expected)
2019-05-27 11:52:56,836 INFO spawned: 'overpass_dispatch' with pid 57
2019-05-27 11:52:56,854 INFO exited: overpass_dispatch (exit status 3; not expected)
2019-05-27 11:52:59,857 INFO spawned: 'overpass_dispatch' with pid 67
2019-05-27 11:52:59,880 INFO exited: overpass_dispatch (exit status 3; not expected)
2019-05-27 11:52:59,881 INFO gave up: overpass_dispatch entered FATAL state, too many start retries too quickly

Trying to start the dispatcher manually using /app/bin/dispatcher_start.sh as the overpass user prints out:

overpass@4cf25b326ec1:/$ /app/bin/dispatcher_start.sh
File /db/db/nodes_meta.bin present. Please use parameter --meta
File /db/db/ways_meta.bin present. Please use parameter --meta
File /db/db/relations_meta.bin present. Please use parameter --meta
File /db/db/user_data.bin present. Please use parameter --meta
File /db/db/user_indices.bin present. Please use parameter --meta

[wiktorn]

Ok, I've found problem with my testing procedure. Thank you for your patience.

I've also fixed logging on stdout from other processes within container, so I think, that now it should be easier to spot such mistakes.

[MichielDeMey]

Thanks a lot, works like a charm now! 🎉