Skip to content

v2.1.0 — finishing what a kill interrupted

Latest

Choose a tag to compare

@amayer1983 amayer1983 released this 05 Aug 16:38

@NotRetarded's Docksentry exited 137 during an update, and he found out from a third-party monitor rather than from us (#2). Two separate holes behind that, and the first one could leave one of your services down.

An interrupted update left the container down, with nobody looking for it

A recreate goes stop → rename to <name>_old → build the run arguments → run.

The rollback that guards every other failure lives in an except handler — and a SIGKILL raises nothing. The process is simply gone. The container stayed stopped under a backup name, and a search of the whole codebase found nothing that ever went looking for it. Indefinitely down, no notification, no recovery.

The swap is journalled before the rename now, and finished on the next start: renamed back, started, and reported.

That it works from the journal rather than from the _old suffix is deliberate. Someone may legitimately run a container called foo_old, and renaming theirs would be a worse bug than the one being fixed. Four states, and three of them do nothing:

what we find what happens
<name> is live the swap landed — the backup belongs to the cleanup grace period
neither name present something removed both; say so, invent nothing
journal older than a day you have had time to intervene; report, do not move anything
the rename fails ask for a hand rather than claim success

Verified against real containers, stopped and renamed to reproduce the crash window.

A hard kill was never reported at all

The exit marker is written only on SIGTERM/SIGINT, so a SIGKILL left nothing behind and the next boot said nothing about it.

The old code read an absent marker as "first boot or unclean kill — we can't prove which" and stayed silent. That was true when it was written, and stopped being true in v2.0.0: every successful start records its version now, so a state file with no exit marker beside it is a hard kill, provably.

The message names exit 137's usual causes rather than only reporting the fact — host memory and CPU pressure is the next question anyone asks, and with the load line from v2.0.1 and the OOM flag from v2.0.2 you now have the readings to answer it.

Upgrading is the usual docker pull ghcr.io/amayer1983/docksentry:latest, or nothing at all with auto-selfupdate on.