fix(docker): bound the shutdown wait so a stuck process cannot strand PID 1

[XenuIsWatching]

Description

^{Explain the changes or enhancements you are proposing with this pull request.}

stop_process_pid signals each managed process and then waits on it forever:

kill "${PID}" || true
# wait for process exit
while [[ -e "/proc/${PID}" ]]; do sleep 0.1; done

A process that refuses to exit wedges PID 1. The container then sits Up serving nothing, and no restart policy can detect it, because policies only react to exits.

Valkey reaches exactly that state. It aborts its own shutdown when the snapshot it writes on exit cannot be written, so on a read-only /redis-data it catches SIGTERM, fails the save, and keeps running.

Getting there is routine, because trap 'exited=1 && shutdown' SIGINT SIGTERM EXIT means every fatal error path runs shutdown(), exactly when the environment is most likely to be broken. The path I hit is a failed startup migration: the daemon's restart policy starts every container at once at host boot and does not honour compose's depends_on (that is applied by docker compose up), so RomM regularly comes up before its database, alembic upgrade head fails, and error_log ends in exit 1.

Observed three times on a host whose storage layer briefly remounts read-only during boot:

t=34.6s   EXT4-fs (sdb): mounted ... r/w
t=83.6s   EXT4-fs (sdb): re-mounted ... ro
t=84s     romm container starts            <- 0.4s into the read-only window
t=175.6s  EXT4-fs (sdb): re-mounted ... r/w
t=199s    romm-db container starts         <- two minutes after romm

sqlalchemy.exc.OperationalError: (mariadb.OperationalError) Can't connect to server on 'romm-db' (115)
ERROR: Failed to run database migrations
INFO:  Stopping valkey-server
<hangs forever>

ps inside the wedged container showed only bash /init, valkey-server, and a live sleep 0.1. Every reboot needed a manual docker restart.

Changes

• stop_process_pid waits STOP_TIMEOUT (default 10s), then escalates to SIGKILL, with a second bounded wait and a warning if even that fails. A process that will not exit can no longer strand PID 1.
• A healthcheck on the romm service in the compose example, hitting the existing unauthenticated /api/heartbeat.
• STOP_TIMEOUT documented in env.template and docs/BACKEND_ARCHITECTURE.md.

Why this is the whole fix

Bounding the wait is enough on its own. Once PID 1 can exit, the failure becomes self-healing: migrations fail, exit 1, shutdown completes, the container exits, and restart: unless-stopped brings it back and retries until the database is up.

An earlier revision of this PR also retried the migrations in-process, but per review that only saves a few container restarts, which the restart policy already handles. It is a startup-time optimization riding along in a hang fix, and it cost two tunables the project would have to support, so I dropped it. Worth revisiting separately if the restart churn bothers anyone in practice.

One clarification, since it came up in review and is easy to get backwards: a healthcheck cannot recover this by itself. Docker healthchecks take no action, they only set status; nothing restarts an unhealthy container in plain Compose without an external agent (autoheal) or an orchestrator. The healthcheck here exists to make a hung PID 1 observable, not to fix it. Note also that the mechanism that failed in the trace above was romm-db's healthcheck combined with depends_on: condition: service_healthy, which the daemon skips at host boot.

Note on REDIS_SAVE_POLICY

48259b3 (after 5.0.0) added REDIS_SAVE_POLICY, and setting it empty sidesteps the valkey hang, since valkey skips the shutdown snapshot when no save point is configured. But the default is 3600 1, which configures one, so the hang is reachable on current master. That is a workaround at the cost of persistence, not a fix, and it only addresses valkey, whereas the unbounded wait is a hazard for every managed process.

Testing

stop_process_pid is exercised by a test script that extracts the function verbatim from the init script and runs it under the same errexit/nounset/pipefail/inherit_errexit options. An untested error path is the reason this bug exists, so I did not want to eyeball it.

• Normal process: stops in 0s, no escalation, no spurious SIGKILL.
• Process with trap "" TERM (valkey's exact behaviour): killed after STOP_TIMEOUT with the expected warning, instead of hanging.

Also verified:

• bash -n, shellcheck, shfmt 3.6.0, prettier 3.9.5, yamllint 1.38.0 all clean. The compose example produces no new yamllint findings versus master (23 pre-existing on both).
• The compose example parses under docker compose config, and the healthcheck command runs successfully inside a live container. $$ defers expansion to the container so a custom ROMM_PORT still works; without that, anyone setting ROMM_PORT would get a permanently unhealthy container.
• Separately, the valkey mechanism was reproduced directly in a throwaway container with a read-only bind mount: with --save "3600 1" it survives SIGTERM, with --save "" it exits cleanly.

Checklist

^{Please check all that apply.}

• I've tested the changes locally
• I've updated relevant comments
• I've assigned reviewers for this PR
• I've added unit tests that cover the changes

_{The init script has no test harness in-repo, so the coverage above lives in a standalone script that sources the function rather than in a committed suite. Happy to add a bats (or similar) target if the project wants shell tests committed.}

AI assistance disclosure

Per CONTRIBUTING.md: this change was written with AI assistance (Claude Code). The root-cause diagnosis came from a real reproduction on my own hardware; the AI wrote the patch and the test script, and I reviewed the result. No Fixes #: I searched open issues and did not find one covering this.

[gantoine]

why a DB migrate loop when the healthcheck would cover this case?

[XenuIsWatching]

Fair — given the shutdown fix, the migrate loop is largely redundant. Once PID 1 can actually exit, the path self-heals: migrations fail → exit 1 → shutdown now completes → container exits → restart: unless-stopped brings it back and it retries. The bounded wait is the load-bearing change here; the loop is an optimization on top of it. Happy to drop it and keep this to the SIGKILL escalation plus the healthcheck.

One correction on the premise though: a healthcheck can't recover this on its own, because Docker healthchecks take no action — they only set status. Nothing restarts an unhealthy container in plain Compose; that needs an external agent (autoheal) or an orchestrator (Swarm/k8s). I added the healthcheck because a hung PID 1 is otherwise completely invisible, not because it fixes anything.

And if you mean the existing romm-db healthcheck with depends_on: condition: service_healthy — that's the mechanism that already failed. It's applied by the docker compose CLI at up time, not by the daemon's restart policy at host boot. From the host this came from:

romm      StartedAt: 2026-07-29T13:05:24Z
romm-db   StartedAt: 2026-07-29T13:07:19Z

RomM started two minutes before its database with that depends_on in place. That's what makes the container's "database is reachable at startup" assumption unsafe regardless of how the restart is triggered.

So: want me to cut it down to just the bounded wait + healthcheck, or keep a slimmer version of the DB wait? The only argument for keeping one is cost — each failed alembic upgrade head takes ~45s to time out, so the restart-policy path churns the container a few times and dumps a traceback each round before it converges.

[gantoine]

@XenuIsWatching i want you to use your meat brain and decide 😉

[XenuIsWatching]

@XenuIsWatching i want you to use your meat brain and decide 😉

alright, i updated it along with the description and title

[gantoine]

Closing in favour of #4019, which goes after the same hang from the other end: replacing the hand-rolled supervisor with s6-overlay, where the bounded shutdown is a config file rather than code we maintain.

Your diagnosis is what made that possible, and the root cause (valkey aborting its own shutdown on a failed snapshot) is fixed directly there with --shutdown-on-sigterm force. Thanks for the writeup.