apply_osc_to_db.sh / update_from_dir get stuck when e.g. "node in .osc not found in loaded data"

[SomeoneElseOSM]

See #774 for context. I've currently loaded an extract of "Britain and Ireland" from a couple of days ago and am trying to apply minutely updates.

apply_osc is started like this:

    apply_osc)
        ionice -c 2 -n 7 nice -n 17 nohup "${EXEC_DIR}/apply_osc_to_db.sh" \
            "${DIFF_DIR}" auto --meta=no \
            >> "${LOG_DIR}/apply_osc_to_db.out" 2>&1 &
        ;;

and from looking at that script it's clear that it is supposed to call update_from_dir and retry on failure. Unfortunately, when update_from_dir exits and is run again, the second time it appears to write no log entries and use no CPU.

op@h23:~$ date
Sun Feb 15 11:35:12 AM UTC 2026

op@h23:~$ ps -ef | grep update_from | grep -v grep
op        101948  101203  0 11:09 pts/4    00:00:00 ./update_from_dir --osc-dir=/tmp/osm-3s_update_zYGuCC/process_6985769 --version=2026-02-13T14\:27\:30Z --flush-size=0

op@h23:~$ ls -alt log/apply_osc_to_db.out
-rw-rw-r-- 1 op op 72695315 Feb 15 11:08 log/apply_osc_to_db.out

The last couple of lines of log/apply_osc_to_db.out are:

/opt/op/bin/apply_osc_to_db.sh: line 258: 101756 Aborted                 ./update_from_dir --osc-dir="$OSC_DIR" --version="$DATA_VERSION" $META --flush-size=0
2026-02-15 11:08:38: ERROR: update_from_dir failed (exit code: 134), retry 1/5

The process 101756 which aborted was replaced by process 101948 (see above), but that does not appear to be doing anything.

I am currently using minutely diffs from https://planet.osm.org/replication/minute/ . I'm aware of #565 ; I could use Geofabrik's daily diffs, but I'd prefer to have minutely updates. I'm aware what that does to database size and would eventually want to put in place something like trim_osc.py (previously mentioned at #774 (comment) ).

I'd expect, even if it encounters a node that it has not previously seen, that update_from_dir wouldn't just stall in this way?

[mmd-osm]

ERROR: update_from_dir failed (exit code: 134),

Well, that looks like the update_from_dir process crashed with an error, likely some corrupted index. Your log snippets above don't include more details about the exact issue. The previous line "101756 Aborted" might indicate that there's core file lying around somewhere unless you have turned that off. So that could be a segmentation fault, or an uncaught exception.

In this case, you likely find a "osm_base*.lock" file in your database directory and a few "*.shadow" files as well. As long as the lock file is in place, update_from_dir won't continue applying updates. You need to manually clean this up (remove lock file + shadow files), then continue the update.

My gut feeling is that your database is already corrupted by now, and futher attempts to apply the same diff will fail in a similar way. Like I've mentioned in the other issue, you're pretty much in "hic sunt dracones" territory.

[SomeoneElseOSM]

Yesterday I switched the update from the planet to Geofabrik's one that matches the extract I was using (and changed replicate_id to match that, of course), per #565 , and that has now handled the 3 daily updates since the extract date without a problem. If anything is corrupted it's not obvious when I query the database.

The errors before the final crash in the "apply" log were just:

compute_idx_and_geometry: Node 12455811636 used in way 28655803 not found.
terminate called after throwing an instance of 'std::logic_error'
what(): Bad geometry for way 28655803: nds 13544686867 (and then lots more nodes, and geometry info)

That way was in one of the planet .osc files, but neither it (nor its nodes) were in the the extract. I'd have expected the "apply" logic to ignore rather than fall over in that example.

[mmd-osm]

What is your —meta setting like?

--meta: Metadata handling mode (attic=full history, yes=metadata, no=current only)

—meta=attic definitely won’t work for your use case.

[SomeoneElseOSM]

In the "apply_osc" script I'm using (both now, with Geofabrik daily updates and before with planet.osm.org minutely ones) "--meta=no".

I have also removed the "--attic" from the dispatcher call since it makes no sense to try and update it if I didn't load it in the first place.

There's a copy of a gzipped failing logfile at https://map.atownsend.org.uk/tmp/apply_osc_to_db.out_20260215_1236.gz if you're interested.

[SomeoneElseOSM]

For info https://www.openstreetmap.org/user/SomeoneElse/diary/408252 is more detail on what I ended up doing.

This morning's update also went OK - despite lines such as

compute_idx_and_geometry: Way 1220157476 used in relation 18462575 not found.

(which got included by bbox in a Geofabrik update, obviously) it said

 done.
Update complete.
2026-02-17 02:58:59: Successfully applied batch up to OSC file 4698