v1.2.0

Added

• Where this tool's watching ends, and what takes over. A reboot commanded on a host with processes wedged in uninterruptible sleep is accepted and never completes; the alarm this raises is correct and nothing in software can act on it. The README now names the kernel watchdog that can — RuntimeWatchdogSec and RebootWatchdogSec, with the check that the device exists first, since on a machine without one those settings are accepted and do nothing.

Upgrading

git pull. Nothing in the tool changed; the addition is a README section naming the kernel watchdog that recovers what this cannot.

Full history in CHANGELOG.md.

v1.1.0

Every check kind here asks one source one question. That is enough while the source is honest, and it stops being enough for anything that reports on other things — a status page, an inventory, a table of scheduled jobs. When one of those goes wrong it goes wrong confidently, and its wrongness is indistinguishable from good news, because nothing else is looking at the same fact.

agree asks the question twice, from two places that cannot fail the same way, and fails when the answers differ:

agree	the timer table lists every timer	systemctl list-timers --all --no-legend | grep -c '\.timer' ::: grep -c . /etc/timers.tsv

On the host these rules come from, those two answers differed by five. Five timers were firing on a schedule nobody had written down, and every report on that machine had been green throughout, because every report read the table.

Both sides answering nothing is a failure rather than agreement. Two commands that produce no output compare equal, and that is exactly the shape of a check that has quietly stopped checking: a typo on both sides, a tool that is not installed.

orphan_timers is the same idea pointed the other way. A timer whose service was deleted never fails — it fires, systemd finds nothing to start, and the job silently stops happening. no_failed_units cannot see that, and neither can anything that waits for a failure.

orphan_timers	every enabled timer still has its service

Both systemd kinds now call an overridable $SYSTEMCTL. That is not indirection for its own sake: without it neither could ever be shown a violation, and no_failed_units never had been. The stand-in in the suite holds two timers, one an orphan, and the rule has to name the orphan and leave the healthy one alone.

Six scenarios, each in both directions. Eighteen assertions.

Upgrading: replace deadman.sh. Existing check files keep working unchanged; the two new kinds are opt-in, and checks.d/10-example.checks shows both.

v1.0.0

Added

• Eight check kinds, configured one per line, covering containers, freshness
  markers, timestamp markers, a writable mount, a critical-finding file, failed
  systemd units and arbitrary commands.
• Checks for the watchers, not only the workload. Docker being up says
  nothing about whether the thing watching your images still checks anything.
• A failure endpoint that carries the reason. The first failing check's name
  and message travel with the alarm, so the notification says what broke rather
  than that something did.
• An empty configuration is a failure. A checks directory with nothing in it
  pings the failure endpoint rather than reporting a green run that verified
  nothing.
• --dry-run and --list, so the configuration can be seen and exercised
  before it is able to page anyone.
• Twelve end-to-end scenarios against real containers and a local HTTP
  server that records what was requested, so every claim about reporting is a
  measurement rather than a reading of the source.

Notes on what the checks refuse to do

• Every request uses curl -f. Without it curl exits zero on an HTTP error,
  and a revoked switch would swallow every alarm the host raises while the
  caller saw success.
• A run whose checks pass but whose ping does not land exits non-zero. Checks
  passing is not the same as the switch having been told.
• Nothing pipes into grep -q or head under pipefail: the producer dies of
  SIGPIPE, the pipeline returns 141, and the check fails at random. A false
  alarm here trains people to ignore the one channel that is meant to be
  trustworthy.
• The systemd check distinguishes an unreadable bus from a clean one. Treating
  empty output as "nothing failed" reports a host that cannot answer as healthy,
  forever.