Provision Grafana alert rules via Pulumi

[rdimitrov]

Context

Today only the Grafana datasources are provisioned in code (deploy/pkg/k8s/monitoring.go:546). Alert rules and dashboards live in Grafana's PostgreSQL DB and are edited via UI — not versioned, not reviewable, and lost if Grafana's DB ever resets.

Surfaced while triaging recurring Publish Endpoint Latency and Availability dropped below 95% alerts that turned out to be metric-pipeline artifacts post-deploy. Wanted to add for: 10m and noDataState: OK to both rules; can only do that via UI today.

Proposal

Add a grafana-alerts ConfigMap mounted at /etc/grafana/provisioning/alerting/ mirroring the existing datasources pattern, and move alert rules into it. Likely worth doing dashboards and notification policies at the same time.

Caveats

• Provisioned alerts are read-only in the UI — every change goes through a PR thereafter. Worth confirming team is OK with that.
• Existing rules need to be exported (Grafana provisioning API) and committed.

Acceptance

• Alert rules and notification policies provisioned from code
• Dashboards same (or explicitly out of scope)
• deploy/README.md updated with the new edit workflow

[rdimitrov]

fyi: @pree-dew @tadasant @BobDickinson - hey, I've been meaning to raise this because currently they are not stored anywhere, my reasoning is this should make all of our deployment be as much IaC as possible but happy to chat if it's unnecessary for now 👍

[tadasant]

Very supportive of this! I think it's a must-have for our mission to automate Registry work

[pree-dew]

@rdimitrov I also fully support this.

It should be in code as it helps with drift detection, auditability & rollback, parity across environment, consistency and also secret management (later for alert contact points).
But when it comes to Grafana, its bit painful in maintaining it via Pulumi.

• Grafana maintains dashboard config as JSON which are very difficult to maintain as even single UI change can expand upto like more than 100 lines in JSON. Since there is no visual feedback while making this change, makes it even harder.
• Alert config is sometimes tied with runtime behaviour, for eg: like specific labels that comes as part of query execution or varies as per environment, hard coding it make configuration brittle.
• Reconciliation is a pain if there is any drift between Pulumi state and Grafana internal state.

Pulumi is great at modeling infrastructure resources with stable IDs and clear lifecycles (VMs, namespaces, secrets). Grafana dashboards and alerts are more like configuration artifacts — they change frequently, are built iteratively, and have a complex internal object model. That's why we have seperate tool for Grafana as code like Grafonnet, grizzly etc.

Suggestion would be achieve this via:

• Datasources, folders, RBAC ===> Pulumi
• Alert contact points, notification policies ===> Pulumi
• Dashboard definitions ===> Grafonnet, if not possible then via configmap(although very hard to maintain big json describing UI elements)
• Alert rules ===> Grafonnet, if not possible then via configmap(no semantic diff and more grafana nuansces but still okay)

@rdimitrov Let me know if I can help with the issue, happy to pick it up.

[rdimitrov]

Yeah, that sounds reasonable and is why I just started the thread to chat about this 👍

I think another feasible option is to use https://github.com/grafana/gcx (prev. Grizzly), it allows you to fetch/apply all of these as resource files, spin up a local Grafana, etc.

It also mentions a few agentic things there too so it seems it should be a good foundation for automating parts of the registry work too