fix(ops): one incident is one page — per-unit cooldown on the OnFailure notifier - #432
Merged
Merged
Conversation
…re notifier vitareba-app could not read its .env on 2026-08-28 (root-owned after a `chown --reference`). Restart=on-failure + RestartSec=3 restarted it 18 times in 60 seconds, every restart fired OnFailure, and every OnFailure sent its own Telegram: six identical "UNIT DOWN" messages for ONE incident, none of which made the fix arrive any sooner. The existing guard could not help. It silences a unit that comes BACK within 8s (deploy restarts, transient blips); a unit that cannot start at all never comes back, so the guard fell through and paged on every loop iteration. Page once per unit, hold for NOTIFY_COOLDOWN_SEC (30m), then re-page as a reminder while it is still down. Recovery clears the stamp, so the next genuine outage pages immediately instead of inheriting the last one's silence. Suppressed restarts still go to the journal — quiet in Telegram must not mean invisible in the logs. Also: the summary block added in #431 sat in the MIDDLE of the test file, so `[ "$fail" -eq 0 ]` under `set -e` aborted the run on the first failure and the cases below it never executed — a suite that reports on itself partially. One summary, at the end, after every case has run. Covered by scripts/hetzner/test-host-alerts.sh, which drives the shipped heredoc payload (no second copy to drift). Proven by mutation on a throwaway copy: dropping the cooldown, dropping the recovery clear, and freezing the cooldown each turn the suite red.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bombardment
vitareba-appcould not read its.envon 2026-08-28 (root-owned after achown --reference).Restart=on-failure+RestartSec=3restarted it 18 times in 60 seconds; every restart firedOnFailure=notify-failure@, and every one of those sent its own Telegram — six identical "UNIT DOWN" messages for one incident, none of which made the fix arrive sooner.The existing guard could not help. It silences a unit that comes back within 8s (deploy restarts, transient blips). A unit that cannot start at all never comes back, so the guard fell through and paged on every loop iteration.
The fix
Per-unit cooldown stamp under
$MON/state/paged_<unit>:NOTIFY_COOLDOWN_SEC(default 30m)Also fixed
The summary block added in #431 sat in the middle of the test file, so
[ "$fail" -eq 0 ]underset -eaborted the run on the first failure and every case below it never executed — the suite reported on a partial run of itself. One summary, at the end.Verification
scripts/hetzner/test-host-alerts.shextracts and drives the shippedNFheredoc — no second copy to drift. 33 passed, 0 failed.Proven by mutation, on a throwaway copy (never the file being pushed), hash-checked before and after:
🤖 Generated with Claude Code
https://claude.ai/code/session_012FQCaKPFv3JpN1RyLEXLZ4