Skip to content

v2.8.4 — a timed-out command is not a command that did not happen

Choose a tag to compare

@amayer1983 amayer1983 released this 16 Aug 16:13
· 2 commits to main since this release

@famewolf sent the log for the dependent that failed for ten days, and it turns the whole thing into a chain that reads end to end. Every step of it is now fixed.

What actually happened

The stop was given 60 seconds. The rename right after it had a hard-coded 10 — generous where it did not matter, mean where it did. It hit that limit.

And Docker completed the rename anyway. Our timeout stops us waiting; it does not stop the daemon working. His own words, which are the sentence this release is built on:

the rm times out after 15 seconds but the delete actually works

The exception then escaped recreate_dependent, which had no try around that call, so the rebuild and the rollback were both skipped. From then on the container existed only as <name>_old, and every later run found no such container, fell through to restart, and printed the same line. That is why it was consistent rather than intermittent: after the first failure the state was permanently wrong and nothing ever looked again.

What changed

The renames follow DOCKER_STOP_TIMEOUT like the rest of the shutdown path. 2.8.3 fixed kill and rm and missed these; this is the other half.

A rename that times out now checks what actually happened before deciding it failed.

A dependent left behind as <name>_old is put back under its own name and started on the next run. recovery.py has healed this exact shape for the main update path all along, but it runs off an in-flight note that only that path writes, so the dependent recreate was never covered by it. Deliberately narrow: only <name>_old, only when <name> itself is absent, only for a container already known to be a group dependent. A stray *_old belonging to someone else is not ours to move.

And an audit of that fix, which found a bug in the fix

The post-timeout check went through a helper that answers "probably yes" when its own inspect fails — a safe instinct for its original job and exactly wrong here, because a busy daemon is the very condition that produced the timeout. Both probes failing would have read as "the rename did not happen". It now distinguishes yes, no and could not tell; could-not-tell counts as failure and the next run heals it, rather than guessing.

The same audit found _rollback_to_old was the one place in the update path with no guards at all — and it is called from an exception handler, so a timeout inside it skipped the rename, the history write and the in-flight clear. Every step is guarded now.


Nothing to do on upgrade. If your containers are slow to stop, DOCKER_STOP_TIMEOUT is the one knob and it now governs the whole shutdown path.