#77's timers were never installed: no scheduled backup has run, and validate cannot see it

[Gerrrt]

ScheduledJobMetricsAbsent is pending on the monitoring host and fires within
the hour. It is correct, and what it has found is that #77 was never finished
deploying.

What is actually true on the host

$ systemctl list-unit-files 'homelab*'
0 unit files listed.

$ ls /var/lib/node_exporter/textfile_collector/
No such file or directory

$ node_textfile_scrape_error{job="prometheus-metrics"}   1
$ node_textfile_scrape_error{job="oracle-metrics"}       0

make install-timers — step 77 of
runbooks/schedule-maintenance.md —
has not been run here. Nothing is installed, so nothing writes the textfile
directory, so the directory does not exist, so the collector errors on every
scrape. (oracle reads 0 because its agent still runs the pre-#77
config.alloy and has no textfile directory configured at all.)

What that costs

No scheduled maintenance has ever run. The most recent volume backup is
backups/volumes/20260829T064124Z, taken by hand on 2026-08-29. The weekly
volume backup, the nightly firewall backup, the nightly backup verification
and the weekly snmp-verify have between them run zero times.

Four of the five rules meant to catch that cannot fire. Every other rule in
backup.rules.yaml joins against homelab_job_max_age_seconds, which
install-timers.sh --install is what writes:

Rule	Reads
ScheduledJobStale	... > on(homelab_job) group_left() homelab_job_max_age_seconds
ScheduledJobFailed	homelab_job_last_exit_code
ScheduledJobNeverRan	homelab_job_max_age_seconds unless on(homelab_job) ...
SecretsKeyBackupUnproven	same join, scoped to verify-key-backup

With the declaration series absent they are all silent, which is exactly what
ScheduledJobMetricsAbsent's own comment predicts — "Nothing else in this file
can fire while this is true." That rule is the one unconditional statement in
the file and it is the only reason any of this is visible. It worked.

The gap worth fixing

make validate passes. install-timers.sh --check derives each timer's
period from systemd-analyze calendar over the .timer files in the
repository and asserts the declared max_age is at least twice it. That is a
good check and it is entirely repo-internal: it compares two copies of the
schedule that both live in git, and cannot see whether either has ever reached
the host.

So the repository agrees with itself, CI is green, make validate prints
scheduled job cadences, thresholds and units agree — and nothing runs. That is
the same shape as the fault #81
was filed about: everything checked except reality.

The runbook already anticipates the neighbouring symptom — its troubleshooting
table has "Timers exist but never fire" — but not this one, where the timers do
not exist at all.

Also wrong

docs/roadmap.md states in the present tense, under Done,
that "Four systemd timers on the monitoring host run make backup weekly ...".
They do not. check_docs.py cannot catch this — it guards counted claims, and
this is a claim about the state of a host.

Suggested fixes

1. Run make install-timers on the monitoring host, and take a backup — it has
   been two days and the last one was manual.
2. Give validate.sh a host-aware check that is skipped in CI and runs locally:
   systemctl list-unit-files 'homelab*' returning nothing on a host that has
   the stack deployed is a failure, not a pass. The existing have/skip
   pattern in validate.sh already models exactly this distinction.
3. Correct the roadmap entry to say what is deployed rather than what was built.

[Gerrrt]

Closed by the timers being installed on the monitoring host, plus three PRs.

On the host. systemctl list-unit-files 'homelab*' now reports 8, all four
timers enabled and scheduled (firewall 04:34, verify-backups 05:32, snmp-verify
Wed 06:44, volumes Sun 03:30). The textfile directory exists,
node_textfile_scrape_error went 1 → 0, and homelab_job_max_age_seconds
declares 5 jobs. ScheduledJobMetricsAbsent cleared, and the four rules that
were structurally unfireable are all health=ok and evaluating.

#216 — validate.sh now asks the host, not just the repository: on a host
running the stack, at least one homelab-* unit must be installed. Skips on a
checkout that is not the deployment host, so CI and laptops are unaffected. Plus
the roadmap correction — it claimed in the present tense that four timers "run"
here — and two runbook rows.

#217 — the schedule check itself ran in validate.sh and in no CI job, which
is the same asymmetry #68 was about. It now runs in CI with --require-all, and
--check asserts the CI step exists so it cannot drift out again. One skip
deliberately survives --require-all: systemd-analyze verify resolves
ExecStart against the real filesystem and is meaningless off the deployment
host.

#219 — and this is the part worth reading. Priming the jobs surfaced that
backup-firewall had never once succeeded. Three defects stacked behind each
other: sops matched its creation rules against /dev/stdin and refused to
encrypt; verify() decrypted as binary what it had encrypted as yaml, which
was unreachable until the first was fixed; and the first backup that landed
turned make validate red because yamllint linted the ciphertext. It now pulls
from 10.0.99.1 and verifies — config version 24.6, 62 firewall rules.

That is the argument for this whole issue, made concretely: a job nothing ran was
a job nothing tested, and it was broken in three places.

Still true and deliberately not fixed here: ScheduledJobNeverRan is pending
for backup-volumes and verify-key-backup. Both are correct — the installer
does not prime the volume backup because it quiesces the stack, and
verify-key-backup has no timer by design and wants a human with removable
media. A volume backup was taken by hand in the meantime
(backups/volumes/20260831T233204Z), but by hand, so it recorded no metric.

make validate passes clean on the monitoring host.