feat(ntfy): alert on any systemd service failure

[etrobert-bot]

Closes #407

What

When any systemd service fails on a NixOS host (tower, leod, pi), push a notification to the self-hosted ntfy topic: title <unit> failed on <host>, body = the unit's last 15 journal lines (capped at 4000 bytes, under ntfy's 4096-byte message limit). New flake.nixosModules.ntfyFailureAlerts lives in modules/ntfy.nix next to the server/subscriber modules and is imported once by modules/nixos-base.nix, so every NixOS host gets it without a per-host import. aaron (darwin) doesn't import nixos-base, so it stays untouched.

The alert publishes with ntfy publish --quiet --title … from ntfy-wrapped (which carries NTFY_TOPIC=http://tower:2586/home), so the endpoint isn't named here; serviceConfig.TimeoutStartSec = "15s" bounds a hung delivery. --quiet is silent on success but still prints server errors, and the CLI exits non-zero on an HTTP 4xx/5xx — so a failed delivery lands the instance in systemctl --failed, the loop-safe local trace.

Approach

Two candidate mechanisms were researched against the systemd source (~/.cache/explore/systemd) and systemd.unit(5):

• DefaultOnFailure= manager setting: does not exist. No hits in src/core/ or man/ — the global drop-in is the only mechanism.
• Top-level drop-in (chosen): service.d/10-ntfy-failure.conf with OnFailure=ntfy-failure@%n.service applies to every system service (specifiers expand in dependency directives — config_parse_unit_deps calls unit_name_printf, src/core/load-fragment.c). Shipped via systemd.packages because /etc/systemd/system is a generated symlink tree and NixOS has no option for type-level drop-ins (generateUnits symlinks package drop-in dirs, nixos/lib/systemd-lib.nix).

Loop prevention

The task-book approach — a later-sorting drop-in resetting OnFailure= — does not work: systemd.unit(5) states "Dependencies (After=, etc.) cannot be reset to an empty list". Instead, an identically named drop-in in the more specific directory (ntfy-failure@.service.d/10-ntfy-failure.conf, containing only [Unit]) masks the service.d/ one, so alert instances never get OnFailure at all — a failed alert (e.g. tower unreachable) cannot trigger another alert, regardless of how it fails (exit code, timeout, OOM).

Verification

• nix fmt, nix flake check --accept-flake-config (statix/deadnix/yamllint/stylua): pass.
• nix build .#nixosConfigurations.tower...toplevel: pass; pi and leod toplevels evaluate.
• Closure inspection: the built system-units tree contains service.d/10-ntfy-failure.conf, the mask at ntfy-failure@.service.d/10-ntfy-failure.conf, and the ntfy-failure@.service template with ntfy-wrapped (topic URL baked in) + coreutils + systemd on PATH, TimeoutStartSec=15s, and the rendered script calling ntfy publish --quiet --title ….
• ntfy publish mechanics checked against a mock HTTP server: stdin becomes the message body, --title sets the X-Title header, NTFY_TOPIC supplies the topic, and the CLI exits 0 on a 200 / non-zero on a 429 (so a failed delivery lands in systemctl --failed). One live publish to tower's ntfy returned exit 0.
• Functional check against the running systemd 260 (user manager, same drop-in merge code, inert template, cleaned up afterwards):
  • OnFailure=ntfy-failure@dummy-ntfy-test.service.service shows on the dummy unit (%n expanded);
  • the alert instance shows OnFailure= empty and DropInPaths lists only the mask;
  • starting the failing dummy actually activated the templated alert instance (Result=success).

🤖 Generated with Claude Code

[etrobert-bot]

Reviewed with independent verification of the load-bearing systemd claims (against systemd 260 source in ~/.cache/explore/systemd, systemd.unit(5), and live tests on a running systemd 260 manager). Verdict: the mechanism is sound and the PR's claims all check out — no blocking findings. 1 suggestion and 2 questions below, plus two praise comments documenting the verification so it doesn't have to be redone.

What I verified independently:

• Loop-prevention masking — confirmed in docs, source, and live (see inline praise on the mask).
• Templated-service edge case — a failing dummy@tty1.service-style unit produces the double-@ instance ntfy-failure@dummy@tty1.service.service; it is a valid unit name, loads, is masked (empty OnFailure), activates on failure, and %i hands the script the exact failing unit name (see inline praise on the %n line).
• Built tower toplevel from the branch (nix build .#nixosConfigurations.tower...toplevel): /etc/systemd/system/service.d/10-ntfy-failure.conf and the mask are both in the generated tree; the rendered ntfy-failure@.service has curl + coreutils (tail) + systemd (journalctl) on its unit PATH and no User=, so it runs as root and can read the system journal.
• nix flake check --accept-flake-config: pass; pi and leod toplevels evaluate.
• On-tower delivery is DNS-independent: the built toplevel's /etc/hosts maps tower → 127.0.0.2 and ntfy listens on :2586, so tower's own alerts don't depend on pi's dnsmasq (no chicken-and-egg).
• The re-nesting of ntfy/ntfyDesktop under nixosModules = { ... } is indentation-only — bodies are semantically identical, darwinModules.ntfyDesktop untouched.
• Conventions: long-form flags throughout, /* bash */ language hint present, comments justified.

praise: Clean single-source design — the alert template reuses the same url/topic let-bindings as the server module, which is exactly why colocating this in modules/ntfy.nix was right.

(Reviewing as the PR author's account, so submitting as COMMENT rather than approve.)

[etrobert-bot]

praise: Verified the %n → instance round-trip independently, including the templated-failing-service edge case. Every valid service name is also a valid instance string: the instance charset is a superset of the unit-name charset and explicitly permits @ (src/basic/unit-name.c, unit_instance_is_valid: "We allow additional @ in the instance string"), and unit_name_is_valid splits at the first @ — so no systemd-escaping is needed here, even for \x2d-escaped names. Live test on systemd 260 (user manager, same drop-in/instantiation code): a failing dummy@tty1.service-style instance produced OnFailure=ntfy-failure@dummy@tty1.service.service; that double-@ name loaded and activated on failure, and %i handed the script back dummy@tty1.service verbatim — exactly what journalctl --unit needs.

[etrobert-bot]

Thanks for verifying the double-@ instance edge case independently — nothing to add.

[etrobert-bot]

praise: The same-name masking trick checks out against both docs and source, including the NixOS case where both files live in the same /etc/systemd/system tree. systemd.unit(5): "Files in type.d/ have lower precedence compared to files in name-specific override directories. … a file in type.d/ applies to a unit only if there are no drop-ins or masks with that name in directories with higher precedence." Implementation: unit_file_find_dropin_paths appends the service.d dirs last (src/shared/dropin.c: "Add this last as it's the most generic and should be able to be overridden by more specific drop-ins") and the file list is deduped by basename with earlier dirs winning (src/basic/conf-files.c, files_add: "Has this file already been found in an earlier directory?") — so template-dir beats type-dir regardless of which root they sit in. Confirmed in the built tower toplevel (both drop-ins present) and live on systemd 260: the alert instance shows OnFailure= empty with only the mask in DropInPaths.

[etrobert-bot]

Thanks for the independent docs+source+live confirmation of the same-tree masking case — nothing to add.

[etrobert-bot]

suggestion (non-blocking): Consider adding --fail. Without it curl exits 0 on HTTP 4xx/5xx (e.g. ntfy replying 429/503), the oneshot is marked successful, and the undelivered alert leaves no trace anywhere. With --fail, an HTTP-level delivery failure behaves like a connection-level one already does — the instance lands in systemctl --failed as a local trace, and the mask keeps it loop-safe. (A modest --retry N --retry-connrefused to ride out brief outages could be a follow-up, but plain --fail is the cheap win.)

[etrobert-bot]

Applied in 7d5aec0: curl --fail added, so an HTTP 4xx/5xx now lands the instance in systemctl --failed like a connection-level error already did (loop-safe via the mask). Kept --silent --show-error as-is — with --fail curl still prints curl: (22) The requested URL returned error: <code> to stderr, so the alert instance's journal records why. Left --retry out per minimal-v1; happy to follow up if lost alerts turn out to matter in practice.

[etrobert-bot]

question: Is unbounded alert volume acceptable for v1? OnFailure= fires on every transition into failed, so a timer-driven oneshot that fails on each tick alerts on every run (a 5-minute timer → ~288/day) — and per the APNs comment earlier in this file, the ntfy.sh relay free tier is ~250 msg/day, so a single flapping unit could exhaust iPhone delivery for the day in addition to spamming the topic. Probably fine across three quiet hosts; if intended, a sentence here acknowledging "no rate limiting, by design" would make it a documented decision rather than a surprise.

[etrobert-bot]

Yes — unbounded by design for v1, now stated in the module comment (7d5aec0: "Delivery is best-effort — no rate limiting and no retry").

I did evaluate the obvious minimal mechanism, StartLimitIntervalSec=/StartLimitBurst= on the template ([Unit] rate limiting, per-instance counters — which would be exactly the right granularity for a flapping unit), and rejected it because it doesn't actually work for the flapping-timer case: the start-limit counter lives on the unit object, and a templated instance that completes successfully goes inactive and is garbage-collected (unit_may_gc, src/core/unit.c — default CollectMode=inactive collects any inactive unit). Each new OnFailure trigger then loads a fresh instance with a fresh counter, so the limit would only ever throttle alert instances that are themselves failing (those stay loaded in failed state) — i.e. it bounds nothing while delivery succeeds, which is precisely the quota-exhaustion scenario. Anything that does work (persistent state, a dedup daemon) is out of proportion for three quiet personal hosts.

On the quota itself: only the iPhone APNs relay (~250 msg/day free tier) can be exhausted; the self-hosted topic and desktop subscribers are unaffected, so a flapper would still be loudly visible — arguably the alert doing its job.

[etrobert-bot]

question: Alerts fired while the delivery path is down are lost for good — accepted as a known limitation? From pi/leod, delivery needs tailscaled up on both ends: tower's 2586 is opened on tailscale0 only, and tower must resolve via MagicDNS to the Tailscale IP (pi's dnsmasq has no tower host-record, and the LAN IP would be firewalled anyway). So early-boot failures on pi/leod, or anything failing while tower is down, produce a failed ntfy-failure@… instance (visible in systemctl --failed, which is a genuinely useful local trace) but never deliver, and nothing retries later. Tower itself is safe even with DNS down (/etc/hosts → 127.0.0.2, verified in the built toplevel). If this is fine for v1 — I think it is — a short note in the module comment would close the loop.

[etrobert-bot]

Accepted as a known limitation for v1, yes — and thanks for verifying the tower-side /etc/hosts independence. An alert that can't reach tower (early boot on pi/leod, tailscaled down on either end, tower itself down) fails once and nothing retries; the failed ntfy-failure@… instance in systemctl --failed is the local trace. Rather than a note about delivery-path topology (which would go stale with any tailscale/firewall change), I folded this into the same one-line contract in the module comment as the rate-limiting answer (7d5aec0): "Delivery is best-effort — no rate limiting and no retry; an undelivered alert only leaves a failed ntfy-failure@ instance (systemctl --failed)." If lost boot-time alerts turn out to matter, a --retry/queueing follow-up can revisit it.

[etrobert-bot]

Review addressed in 7d5aec0 (branch rebased onto current main — wlsunset merge):

• Applied: curl --fail so HTTP-level delivery failures mark the alert instance failed instead of silently succeeding.
• Answered (no code beyond a comment line): no rate limiting and no retry in v1, now documented as a one-line best-effort contract in the module comment. StartLimitIntervalSec/Burst was investigated and rejected — successfully-completed instances are GC'd (unit_may_gc, CollectMode=inactive) taking their counters with them, so it would only throttle already-failing alerts; details in the inline threads.

Re-verified on the rebased tree: nix fmt clean, nix flake check passes, tower toplevel builds, and the rebuilt closure's ntfy-failure@.service script carries --fail. Unit semantics otherwise unchanged, so the earlier live drop-in/mask test still stands.

[etrobert-bot]

Round 2, reviewing 7d5aec0 (rebased onto 727b0db). Scope: the delta since round 1 plus adjudication of the declined rate-limit suggestion — round-1 verification not redone since unit semantics are unchanged by the rebase.

• Rebase: clean — git diff 727b0db3 7d5aec08 touches exactly the 4 PR files; range-diff shows the original commit unchanged plus one follow-up commit.
• --fail applied: confirmed in the rebuilt tower closure — the rendered ntfy-failure@ start script now reads curl --fail --silent --show-error --max-time 10. Keeping --silent --show-error is right: --fail still emits curl: (22) … on stderr, so the failed instance's journal records the HTTP code. Fine to leave --retry out for v1.
• Rate-limit decline: accepted — the GC reasoning is correct (verified against the source, see inline note on the new comment line).
• New best-effort comment line: meets the repo convention — states only current behavior, one non-obvious operational contract, two lines, no topology details that would go stale.
• All five round-1 threads have substantive replies; nothing left open from my side.

Verdict: Approved

[etrobert-bot]

note: I checked the StartLimit-can't-throttle-delivered-alerts claim against the systemd source before accepting the decline — it holds. The start-rate counter is per-Unit-object state (u->start_ratelimit, initialized in unit_new, src/core/unit.c:122; checked via ratelimit_below in unit_test_start_limit, unit.c:1872). After a successful oneshot run the instance is inactive, so under the default CollectMode=inactive unit_may_gc allows collection (unit.c:461-465), and nothing pins it in unit_gc_sweep: only UNIT_ATOM_REFERENCED_BY deps or explicit refs keep a unit GOOD (src/core/manager.c:1391-1409), and UNIT_ON_FAILURE / UNIT_ON_FAILURE_OF map to exactly their own single atoms — no REFERENCES/REFERENCED_BY (src/core/unit-dependency-atom.c:79,81). So a delivered alert instance is freed, and the next failure re-instantiates it with a fresh counter — StartLimitBurst= could only ever throttle instances that stay loaded, i.e. already-failing deliveries, as you said. The comment line you added is the right resolution: true-now, minimal, and documents the one non-obvious contract.

[etrobert-bot]

praise: Verified in the rebuilt tower closure — the rendered start script contains curl --fail --silent --show-error --max-time 10, and --show-error still surfaces curl: (22) The requested URL returned error: … in the failed instance's journal, so the local trace names the HTTP code. Exactly the cheap win intended; agreed on deferring --retry.

[etrobert]

If something needs to be added to leod/pi/tower probably the right place is to add in nixos-base.nix

[etrobert-bot]

Applied in 6462b86b: ntfyFailureAlerts is now imported once in modules/nixos-base.nix (imported by all three NixOS hosts) and dropped from the per-host leod/pi/tower module lists. aaron (darwin) does not import nixos-base, so it stays untouched.

[etrobert]

suggestion: Should we use the ntfy-wrapped instead to not have to specify url not topic?

[etrobert]

And simplify interface

[etrobert-bot]

Applied in b5f0939f: the alert now pipes the journal into ntfy publish --quiet --title … from ntfy-wrapped instead of curl, so the endpoint (NTFY_TOPIC=http://tower:2586/home) is no longer named here — the interface is just a title plus the stdin body.

Verified against a mock server that stdin becomes the message body, --title sets the X-Title header, NTFY_TOPIC supplies the topic, and the CLI exits non-zero on an HTTP 4xx/5xx (so a failed delivery still lands the instance in systemctl --failed, exactly as curl --fail did) while --quiet stays silent on success but still prints the server error on failure. One live publish to tower's ntfy returned exit 0. Also added serviceConfig.TimeoutStartSec = "15s" to replace curl's --max-time 10, bounding a hung delivery (loop-safe — the mask empties OnFailure regardless of how the alert fails).

[etrobert]

question: What would happen without this line?

[etrobert-bot]

Less than you might expect — it is not the line between bounded and unbounded. Type=oneshot is still subject to systemd's start timeout, which falls back to the system manager's DefaultTimeoutStartSec. We don't override that (nixos-base only sets DefaultTimeoutStopSec=15s, and only on the user manager), so the built tower's system.conf carries the upstream default of 90s. Without this line a hung delivery would just sit for up to 90s before systemd SIGTERMs it and marks the instance failed, instead of 15s.

Whether a hang is even possible: ntfy publish uses Go's http.DefaultClient (client/client.go:115), which has no client-level timeout; only http.DefaultTransport bounds the dial at 30s. So:

• connection refused / DNS failure → errors immediately (well under 15s either way);
• blackholed host (no SYN-ACK) → dial errors at ~30s without the line, killed at 15s with it;
• host that accepts the TCP connection but then stalls (e.g. tower up but ntfy wedged) → no app-level timeout, so it hangs until systemd's 90s without the line, 15s with it.

So the line only tightens the worst-case cleanup from 90s to 15s (and is what replaced curl's --max-time 10). It's defensive, not correctness-critical — I'm happy to drop it and rely on the 90s default if you'd rather keep the service config minimal.

[etrobert-bot]

Dropped — serviceConfig is back to just Type = "oneshot", so a hung delivery now falls back to systemd's 90s DefaultTimeoutStartSec. Folded into the ntfy-wrapped commit (29d7e52); tower still builds.