v2.8.2

CI: retry release publishing through transient GitHub API errors

The v2.8.2 tag run hit a GitHub API 503 during gh release create; the
fallback then assumed "create failed" meant "release exists" and died on
gh release upload against a nonexistent release. Now: check existence
explicitly with gh release view, retry the whole publish up to 5 times
30s apart, and only fail after that.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

v2.8.1

Ship SMART and IPMI alert rules with the Prometheus package

The smartctl/ipmi exporters and dashboards existed, but nothing alerted
when a drive or BMC sensor actually went bad. Nine disk rules (SMART
overall-health, NVMe critical-warning/spare/wear/media-errors, ATA
reallocated+pending sectors, SAS grown defects, uncorrected errors,
temperature) and three BMC rules (sensor critical/warning, collector
down). Metric names are taken from the queries the shipped dashboards
already use.

Like the Alertmanager-health rules, these live in the base rules.d and
are inert until matching series exist, so no per-exporter wiring is
needed. The *Increasing rules use delta() over 6h so drives with stable
historic error counts don't page forever.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

v2.8.0

Add Alertmanager HA clustering support (monitoring-configure-cluster)

Gossip runs over TCP+UDP 9094. The packaged Quadlet unit becomes
cluster-ready but inert: a small entrypoint wrapper (riding the existing
/etc/alertmanager bind mount) appends flags from an unowned
/etc/alertmanager/cluster.args, so the conffile unit is never edited and
Image-bump upgrades stay clean. Quadlet drop-ins would have been the
natural mechanism but need Podman >= 5.0, and Ubuntu 24.04 ships 4.9.

monitoring-configure-cluster writes the cluster flags, opens 9094 tcp+udp
on the firewall, restarts the service, and prints cluster status;
--disable reverses all of it.

Prometheus's alerting config moves from a static alertmanager:9093 target
to file_sd over /etc/prometheus/alertmanagers.d/ (the targets.d pattern):
the alertmanager package registers the local instance, and
monitoring-configure-cluster registers peers so alerts fan out to every
cluster member. 9094 is always published on the host but only
firewall-opened when clustering is enabled.

CI: new smoke step enables clustering against an unreachable TEST-NET
peer, asserts the running process carries the cluster flags, that the
cluster API answers, and that Prometheus discovers both alertmanagers,
then disables and asserts cleanup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

v2.7.0

Split metrics-stack into per-component packages

metrics-stack is now a metapackage over four new packages so partial
deployments work (Prometheus without Grafana, Grafana on another host,
alerting-free stacks):

- metrics-stack-common: the shared 'metrics' Podman network
- metrics-stack-prometheus: Prometheus, rules, targets.d, monitoring-add-exporter
- metrics-stack-alertmanager: Alertmanager, monitoring-configure-email
- metrics-stack-grafana: Grafana, provisioning, monitoring-add-dashboard

Alertmanager now registers its scrape target via monitoring-add-exporter
(the exporter-package pattern) instead of being hardcoded in prometheus.yml,
so Prometheus-only hosts don't carry a permanently-down target.

Clean break: no Replaces/Conflicts migration plumbing -- pre-split installs
must uninstall the old metrics-stack before installing these (data under
/var/lib survives either way).

CI: stack debs install as one transaction, alertmanager auto-registration
is asserted, and a new step removes Alertmanager and confirms Prometheus
and Grafana keep running without it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

v2.4.1

Split node_exporter dashboard into overview + CPU/memory/network/disk…

v2.4.0

Fixed

• RPM upgrades no longer leave the stack stopped. RPM runs the new package's %post before the old package's %preun, so upgrading any of these packages used to stop its services and deregister its Prometheus targets right after the new version started them — dnf upgrade left monitoring down. The pre-removal scripts now only act on a real uninstall. If you're on an earlier 2.x rpm, this upgrade is the last one affected (the fix ships in the new package, but the old package's unguarded script still runs once): after upgrading, run sudo systemctl restart prometheus alertmanager grafana on the base host, restart any upgraded exporter services, and re-run the exporter registration if a target went missing — or simply reinstall the exporter package. Upgrades from v2.4.0 onward are clean. Debian installs were never affected.

Changed

• Removing a package now cleans up fully: the firewall port its install opened is closed, and systemd is reloaded so no stale Quadlet-generated unit lingers.
• Debian note: plain apt remove keeps the Quadlet .container file (standard dpkg conffile semantics), so the service stays defined and would start again on reboot. Use apt purge to remove a package's services for good. dnf remove has no such split.
• monitoring-add-exporter accepts bracketed IPv6 targets, e.g. [2001:db8::1]:9100.
• monitoring-configure-email handles SMTP passwords containing single quotes.

Internal

• CI now runs the full build + smoke tests on every push and PR (not just tags), with a shellcheck lint job and an upgrade-path regression test.
• Packaging manifests simplified via convention-based staging; built packages verified identical.

v2.3.0

v2.3.0: add parallel Rocky Linux smoke-test lane for .rpm packages

v2.2.3

v2.2.3: fix CI exporter checks (systemd is-active instead of HTTP; ip…

v2.1.0

v2.1.0: add metrics-stack-dashboards-smartctl package

v2.0.3

v2.0.3: fix CI dashboard-provisioned check (retry instead of flat 5s …