Skip to content

v1.64.0 — Memory diagnostics on every death

Latest

Choose a tag to compare

@amayer1983 amayer1983 released this 01 Aug 18:51

When a container dies, the alert now tells you about memory — not only when Docker flagged an out-of-memory kill.

That distinction was the bug. The snapshot naming the biggest consumers was wired to fire on the OOM flag alone, and that flag comes from the container's own cgroup: it's reliable when a container hits its own memory ceiling, less so when the kernel picks a victim under host-wide pressure. So the commonest version of the problem — a greedy neighbour starves something, the neighbour is fine, the innocent one dies — produced an alert with nothing about memory in it at all. Reported by @NotRetarded in #2 after an app he'd just installed took down his Unifi container.

Crash-restarts and non-zero exits now carry it too, with the host's own state on the line above:

🔁 Unifi-OS-Server crashed (exit 137) and was restarted by its restart policy at 16:14:47 (restart #1).
Host memory (used/total): 14.8/15.6 GB · Swap 3.9/4.0 GB
Top memory at event time: some-new-app 9.1GiB · unifi-os 2.2GiB · postgres 890MiB

The host line comes first deliberately. A list of top consumers on its own invites you to blame whoever is at the top — and if there were 8 GB free, that's just your biggest container minding its own business. The line above it says whether memory was the story at all.

Nothing polls in the background for any of this. It costs one docker stats at the moment of the event, shared across every event in the same sweep so a mass death doesn't turn into twenty calls, plus a read of /proc/meminfo. Monitors watching a remote host print no host line — that file describes the machine Docksentry runs on, and wrong numbers under a remote container's name are worse than no numbers.

What this still won't do is convict anyone. The picture is taken when the monitor notices, and it polls every 60 seconds, so on a bad minute the culprit has already let go of what it took. Doing that properly means watching docker events rather than polling, which is a bigger piece of work and isn't in this release.

Upgrading is the usual docker pull ghcr.io/amayer1983/docksentry:latest, or nothing at all if you have auto-selfupdate on.