Add fleet-wide host health monitoring and missed-heartbeat alerts

[Nickfost]

Goal

Add set-and-forget health monitoring for every enrolled ci-fleet controller host, including machines at different homes, buildings, and VPS providers.

The existing local checks from #4 are the baseline. This issue extends them into a consistent host-health contract, scheduled execution, fleet-visible status, and actionable alerts.

A controller being offline must be detectable externally; a failed host cannot be expected to report its own failure.

Health contract

Each enrolled host must check and report, without exposing secrets:

• root and Docker filesystem utilization;
• inode exhaustion;
• available memory, sustained swap pressure, CPU/load pressure, and recent OOM evidence;
• Docker daemon availability and unhealthy or repeatedly restarting fleet containers;
• controller health and desired lifecycle state;
• configured and effective runner capacity;
• desired-state drift;
• stale fleet-owned containers, images, networks, volumes, and build cache;
• health, last success, and failure state of cleanup, drift, update, and health timers;
• unattended-upgrade status, failed package state, and pending reboot state;
• system clock synchronization;
• optional backup freshness through a generic host-local check or provider hook.

Thresholds must be conservative defaults with reviewed private overrides where appropriate. Machine addresses, monitoring credentials, provider identifiers, and alert destinations remain outside the public repository.

Reporting and alerting

Provide both:

1. concise human-readable output for terminal use; and
2. stable machine-readable output suitable for aggregation.

Install a systemd service/timer that runs automatically. Preserve the most recent redacted result and timestamp locally.

Support an outbound heartbeat or another provider-neutral reporting interface configured host-side. The fleet monitor must distinguish:

• healthy;
• warning;
• unhealthy;
• drained/maintenance;
• stale or missing heartbeat;
• intentionally disabled/retired.

A missing heartbeat must become unhealthy after a documented grace period. The design must work without inbound SSH and without exposing controller or GitHub App credentials.

Installer integration

The worker-controller installer must:

• install or upgrade the health service and timer;
• validate thresholds and host-local monitoring configuration;
• run a health check before declaring convergence successful;
• display the last health result in --check output;
• preserve monitoring configuration and credentials during rollback;
• remove only fleet-owned monitoring state during uninstall.

Controller installation must fail closed for immediate critical conditions such as an unavailable Docker daemon, invalid credential permissions, unsafe disk exhaustion, or an unhealthy candidate controller.

Safety

• Never print or transmit secret values.
• Never send project source, job logs, environment variables, GitHub tokens, or private keys.
• Never use global Docker prune as a health remedy.
• Never delete resources merely because a health check fails.
• Never hardcode private hosts, IP addresses, backup systems, or notification services in the public repository.
• A monitoring outage must not silently report every host as healthy.
• Health checks must not interrupt active jobs.

Tests

Add deterministic tests for at least:

• healthy host;
• disk warning and critical thresholds;
• inode exhaustion;
• Docker unavailable;
• unhealthy/restarting controller;
• desired-state drift;
• failed or stale maintenance timer;
• pending reboot and failed unattended upgrades;
• recent OOM evidence;
• warning versus failure exit codes;
• secret redaction;
• maintenance/drained state;
• stale/missing external heartbeat;
• installer integration and rollback preservation.

Documentation

Document:

• what is checked locally;
• what requires external liveness monitoring;
• default thresholds and private overrides;
• interpreting warning/failure output;
• configuring a provider-neutral heartbeat receiver;
• maintenance/drain suppression;
• adding, replacing, and retiring a monitored host;
• recovery steps for disk pressure, Docker failure, drift, failed timers, and missed heartbeat.

Link the guide from the README, documentation index, host-maintenance guide, desired-state guide, and host installer.

Acceptance

• One command produces redacted human and machine-readable health results.
• Every installed controller host receives the scheduled health timer.
• Disk space, inodes, memory/swap, OOM, Docker, controller, drift, timers, updates, and reboot state are covered.
• Missing-host detection works externally without inbound SSH.
• Drained and intentionally disabled hosts do not create false alarms.
• Critical installer and post-upgrade health failures trigger safe rollback.
• No secret values or private infrastructure inventory enter the public repository.
• Tests and operator documentation cover normal, warning, failure, maintenance, recovery, and retirement paths.

Related

• Add scoped cleanup scanner and host health checks #4 — existing scoped cleanup and local health-check baseline
• Document set-and-forget host and project onboarding #18 — set-and-forget host operations
• feat: Add idempotent install-host.sh to enroll and manage CI fleet Docker hosts #21 — host installer
• Define Git-authored desired state for fleet controllers and capacity #32 — Git-authored controller desired state