Allow modules to provision alert rules

[Amygos]

Description

NS8 modules can already publish Prometheus scrape targets, Grafana datasources, and Grafana dashboards through Redis-backed keys consumed by the metrics module. They cannot publish their own Prometheus alert rules through the same supported integration.

Add a supported mechanism that lets a module instance define alerting rules for the metrics stack. For example, a PostgreSQL module should be able to publish a PostgresqlDown rule with its exporter target and notify the metrics module so Prometheus loads the rule.

This benefits module authors and administrators because application-specific alerts can be shipped and maintained by the module that owns the service, without manually editing generated files under rules.d.

Expected behavior:

• modules publish alert rules under module/<module_id>/metrics_alert_rules;
• a new metrics-alert-rules-changed event causes the metrics module to provision and reload alert rules;
• both a complete Prometheus rule file with groups: and a single alert rule payload are accepted;
• only alerting rules are accepted; recording rules are rejected;
• malformed YAML, unsupported schemas, missing alert or expr, and rules rejected by promtool are rejected;
• invalid updates keep the previous valid generated file and do not break Prometheus;
• missing or unsupported severity, incomplete bilingual annotations, missing metrics, and duplicate alert names generate warnings without rejecting the rule;
• missing-metric validation is best effort and fails open when Prometheus data or its query parser is unavailable;
• duplicate alert names are accepted, with documentation recommending module/application-prefixed names such as PostgresqlDown;
• module-provided alerts use the existing Alertmanager and alert-proxy flow, including Mimir forwarding and email for critical alerts when mail is configured;
• the existing experimental custom_alerts feature uses the same parser and validator, without strict backward compatibility for malformed or unsupported old entries.

Generated module rules are stored as:

rules.d/provision_<module_id>_<name>.yml

Add the event handler:

imageroot/events/metrics-alert-rules-changed/15handler

Prometheus target updates currently rely on file_sd_configs and do not need a reload. Alert rule files require an explicit reload. Add a systemd ExecReload operation that sends SIGHUP through the Prometheus container CID file. The event handler should use the systemd reload operation, verify the Prometheus reload result, and fall back to systemctl --user try-restart prometheus.service if the reload command fails.

Alternative solutions

• Keep only the existing experimental custom_alerts mechanism. This does not provide a clear cross-module contract.
• Require administrators to manage rules.d manually. This conflicts with the generated-state model because provisioning can overwrite those files.
• Accept only complete Prometheus rule files. Supporting single-rule payloads also makes the integration easier for module authors and extends the existing experimental format.
• Reject duplicate alert names. Warning is less disruptive for the first implementation; documentation should recommend unique module/application-prefixed names.
• Inject publisher identity labels automatically. Keep existing label behavior unchanged in v1 and evaluate provenance labels as a future improvement.

References

• Prometheus alerting rules: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
• Prometheus configuration reload: https://prometheus.io/docs/prometheus/latest/management_api/