Grafana Alerts Sync (ngalert)

[noroutine]

Hi grr team

I'm investigating the approach to automate alerts sync in similar approach to dashboards (via jsonnet and https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/post.json)

Atm looks like grizzly supports only Cortex API for alerts, or did i miss anything?

Is there any plan/vision how managing alerts as a code falls into grizzly/grafana future? Can you help/advise on this what can be a potential here?

Thank you in advance

[JustinPealing]

I am also interested in this feature, and could have a go at submitting a PR if this would be welcome? I've started putting something together here

https://github.com/JustinPealing/grizzly/tree/GrafanaRule

Looking at the API the POST /api/ruler/grafana/api/v1/rules/{Namespace} endpoint looks to me like the right one. It accepts a rule group, so presumably some sort of GrafanaRuleGroup resource (with the Grafana prefix disambiguate it from the existing PrometheusRuleGroup resource) e.g.

apiVersion: grizzly.grafana.com/v1alpha1
kind: GrafanaRuleGroup
metadata:
  folder: A
  name: API
spec:
  interval: 1m
  rules:
    - annotations:
        __dashboardUid__: hzyirKunk
        __panelId__: "4"
      expr: ""
      for: 5m
      grafana_alert:
        condition: C
        data:
          - datasourceUid: BfH6yNu7k
            model:
              azureMonitor:
                aggregation: Average
                dimensionFilters: []
                metricDefinition: Microsoft.Web/serverFarms
                metricName: MemoryPercentage
                metricNamespace: Microsoft.Web/serverFarms
                resourceGroup: my-resource-group
                resourceName: my-resource
                timeGrain: auto
                top: "10"
              intervalMs: 1000
              maxDataPoints: 43200
              queryType: Azure Monitor
              refId: A
              subscription: 4559db58-874d-4360-ae76-c179609e3549
            queryType: Azure Monitor
            refId: A
            relativeTimeRange:
              from: 300
              to: 0
          - datasourceUid: "-100"
            model:
              conditions:
                - evaluator:
                    params:
                      - 60.3
                    type: gt
                  operator:
                    type: and
                  query:
                    params:
                      - A
                  reducer:
                    params: []
                    type: last
                  type: query
              datasource:
                type: __expr__
                uid: "-100"
              expression: A
              hide: false
              intervalMs: 1000
              maxDataPoints: 43200
              reducer: mean
              refId: B
              settings:
                mode: dropNN
              type: reduce
            queryType: ""
            refId: B
            relativeTimeRange:
              from: 0
              to: 0
          - datasourceUid: "-100"
            model:
              conditions:
                - evaluator:
                    params:
                      - 0
                      - 0
                    type: gt
                  operator:
                    type: and
                  query:
                    params: []
                  reducer:
                    params: []
                    type: avg
                  type: query
              datasource:
                type: __expr__
                uid: __expr__
              expression: ${B} > 60
              hide: false
              intervalMs: 1000
              maxDataPoints: 43200
              refId: C
              type: math
            queryType: ""
            refId: C
            relativeTimeRange:
              from: 0
              to: 0
        exec_err_state: Alerting
        intervalSeconds: 60
        no_data_state: NoData
        title: API Memory > 60%

[johnpena]

What would it take to get something merged here? Would love to see this and would be happy to assist if needed.

[mms-gianni]

Is there any progress on this topic? I would love to see it merged.

[JustinPealing]

After doing some more testing of my fork I discovered that it does not handle updates correctly, unfortunately team priorities have shifted and I haven't had a chance to return to this.

I can't remember the exact details but I think it was that the endpoint expects each rule to have the ID of the corresponding rule that is being updated. Probably it can make a call to get the existing rules and lookup the ID by matching on the name.

[chrismgrayftsinc]

@JustinPealing for what it's worth, I forked your fork and have something that works on updates. I don't expect it to be extremely robust, but it's working for me for now. https://github.com/chrismgrayftsinc/grizzly/tree/GrafanaRule

[julienduchesne]

v0.3.0 introduced support for ngalert resources