Initialization hangs at: "OSM file has no replication headers. Looking for newest OSM object."

[pcace]

Hi there, i am trying to initialize the overpass api with these environment vars:

      "Env": [
                "OVERPASS_RULES_LOAD=10",
                "OVERPASS_META=yes",
                "OVERPASS_MODE=init",
                "OVERPASS_PLANET_URL=https://download.geofabrik.de/europe-latest.osm.bz2",
                "OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/minute/",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.27.1",
                "NJS_VERSION=0.8.5",
                "NJS_RELEASE=1~bookworm",
                "PKG_RELEASE=1~bookworm",
                "DYNPKG_RELEASE=2~bookworm",
                "OTEL_VERSION=0.1.0",
                "OVERPASS_USE_AREAS=true",
                "OVERPASS_ALLOW_DUPLICATE_QUERIES=no"
            ],

the machine i am running it on is a 32GB Ram / 16Core machine hostet at hetzner.de.
i am currently wondering if it somehow crashed or not, since i have the same log message for more than a day:

2025-07-06 01:52:34 DEBUG: OSM file has no replication headers. Looking for newest OSM object.

which generally makes sense since i have pulled it from geofabrik. if i look at htop i can see that there are processes working on the update:

but there is currently nothing in the diffs folder:

(base) root@machine:/big/docker/overpass_europe# ls 
changes.log     cookie.jar      db/             diffs/          planet.osm.bz2  
(base) root@machine:/big/docker/overpass_europe# ls diffs/
(base) root@machine:/big/docker/overpass_europe# ls db/
node_frequent_tags.bin	    nodes.bin.idx	node_tags_global.bin	  relation_frequent_tags.bin	  relation_roles.bin.idx  relations_meta.bin		relation_tags_local.bin.idx  user_indices.bin.idx	ways.bin       ways_meta.bin.idx
node_frequent_tags.bin.idx  nodes.map		node_tags_global.bin.idx  relation_frequent_tags.bin.idx  relations.bin		  relations_meta.bin.idx	rules			     way_frequent_tags.bin	ways.bin.idx   way_tags_global.bin
node_keys.bin		    nodes.map.idx	node_tags_local.bin	  relation_keys.bin		  relations.bin.idx	  relation_tags_global.bin	user_data.bin		     way_frequent_tags.bin.idx	ways.map       way_tags_global.bin.idx
node_keys.bin.idx	    nodes_meta.bin	node_tags_local.bin.idx   relation_keys.bin.idx		  relations.map		  relation_tags_global.bin.idx	user_data.bin.idx	     way_keys.bin		ways.map.idx   way_tags_local.bin
nodes.bin		    nodes_meta.bin.idx	osm_base_version	  relation_roles.bin		  relations.map.idx	  relation_tags_local.bin	user_indices.bin	     way_keys.bin.idx		ways_meta.bin  way_tags_local.bin.idx
(base) root@machine:/big/docker/overpass_europe# 

these are the file sizes:

du -sch *
4.0K	changes.log
4.0K	cookie.jar
80G	db
4.0K	diffs
49G	planet.osm.bz2
129G	total

is there some missconfiguration going on, or do i just need to wait another day or two?
currently the docker container reports it is running since 4 days (this is what it took to process europe) and unhealthy:

4e6c2f89d791   wiktorn/overpass-api                 "/docker-entrypoint.…"   4 days ago     Up 4 days (unhealthy)   0.0.0.0:12345->80/tcp, [::]:12345->80/tcp                                                                                                                                                                                                     overpass_europe

to me it seems it is taking way too long, but i am not sure if it might be ok. should i restart the container? or just leave it?

Thanks a lot for help!

[zstadler]

I've reached a similar point after 44 hour of init. About 5.25 hours later, the latest timestamp was found, and within less than a second, the corresponding sequence ID was found.

I was under the impression that in init mode, the .osm.bz2 file is scanned for the latest timestamp by the
osmium fileinfo -e -g data.timestamp.last /db/planet.osm.bz2 command within

Overpass-API/docker-entrypoint.sh

Line 89 in 5396bf5

/app/bin/init_osm3s.sh /db/planet.osm.bz2 /db/db /app "${META}" "--version=$(osmium fileinfo -e -g data.timestamp.last /db/planet.osm.bz2) --compression-method=${OVERPASS_COMPRESSION} --map-compression-method=${OVERPASS_COMPRESSION} --flush-size=${OVERPASS_FLUSH_SIZE}" &&

As far as I can tell, that scan took about 3.5 hours.

@wiktorn,
Do you think it would be useful to capture that last timestamp and pass it to update_overpass.sh

Overpass-API/docker-entrypoint.sh

Line 94 in 5396bf5

/app/bin/update_overpass.sh -O /db/planet.osm.bz2 &&

which would add it as a --start-date flag to

Overpass-API/bin/update_overpass.sh

Line 51 in 5396bf5

/app/venv/bin/pyosmium-get-changes -vvv "$@" --cookie /db/cookie.jar --server "${OVERPASS_DIFF_URL}" -o "${DIFF_FILE}" -f /db/replicate_id

---

Some timestamps from docker compose logs --timestamps overpass, FYI.

Downloading the planet data:

overpass-1  | 2025-07-16T19:00:17.129481630Z No database directory. Initializing
overpass-1  | 2025-07-16T19:00:17.136533058Z   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
overpass-1  | 2025-07-16T19:00:17.136564387Z                                  Dload  Upload   Total   Spent    Left  Speed
100   341  100   341    0     0   8872      0 --:--:-- --:--:-- --:--:--  8973
100   384  100   384    0     0   7566      0 --:--:-- --:--:-- --:--:--  7566
100  152G  100  152G    0     0  34.6M      0  1:14:56  1:14:56 --:--:-- 39.1M1:14:50  1:12:35  0:02:15 30.5M
overpass-1  | 2025-07-16T20:15:13.379337171Z Running preprocessing command: 

End of the first scan for the latest timestamp :

overpass-1  | 2025-07-17T01:51:32.036555163Z Reading XML file ... elapsed node 1. Compute current ... ready. Flushing to database ....... done.

The database creation completion:

overpass-1  | 2025-07-18T15:05:43.002259241Z Update complete.
overpass-1  | 2025-07-18T15:05:43.181244239Z Database created. Now updating it.
overpass-1  | 2025-07-18T15:05:43.185340497Z Updating
overpass-1  | 2025-07-18T15:05:49.200599256Z 2025-07-18 15:05:49 DEBUG: OSM file has no replication headers. Looking for newest OSM object.

End of the second scan for the latest timestamp:

overpass-1  | 2025-07-18T20:21:36.106824278Z 2025-07-18 20:21:36 INFO: Using replication server at https://planet.openstreetmap.org/replication/minute/
overpass-1  | 2025-07-18T20:21:36.107181203Z 2025-07-18 20:21:36 DEBUG: Looking up sequence ID for timestamp 2025-07-06 23:59:59+00:00

Identification of the sequence ID

overpass-1  | 2025-07-18T20:21:36.375772665Z 2025-07-18 20:21:36 DEBUG: Starting download at ID 6673397 (max 100 MB)