Handle Alerts Provisioning in K8s in same way it's done for Dashboards and Datasources

[dragoangel]

Idea

It will be much better if Grafana will have dedicated grafana-sc-alerts container which will have the possibility to find alerts configmaps in any K8s Namespace by grafana_alert=1 label which will allow people to create own Helm charts with both Project, it's Grafana dashboards and alerts in the same place, in own Namespace.

How its done now in Grafana 9.3

Mount ConfigMap to /etc/grafana/provisioning/alerting folder of Grafana container.

Problematic of current implementation (Grafana 9.3)

Having all alerts in one Helm chart (for example kube-prometheus-stack) will create big secret quickly which will reach K8s API limits and developers will have to develop many Helms to separate alerts in a smaller charts. These charts need to be deployed in the same namespace with Grafana and Grafana needs to mount all of them in the provisioning folder via values.yaml of kube-prometheus-stack. This is not a nice approach that in end someday will lead to missed mount and not user friendly in general.

I'd better add a comment here: #39510 but the discussion is locked.

[dragoangel]

Hi @JohnnyQQQQ, what you think about such idea?

[mwilfried]

Thx you @dragoangel but I was talking about the configmap containing the rule(s).

I've modified my values.yml according to your, then I create a configmap with data the content of a rule , but nothing visible in the UI

I see the yaml or json created in /etc/grafana/provisioning/alerting

[dragoangel]

@mwilfried ok, clear, alert configmap should look something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: example-rules
  labels:
    grafana_alert: "1"
    app: grafana
data:
  example.yaml: |-
    apiVersion: 1
    groups:
      - orgId: 1
        name: Example
        folder: Example
        interval: 60s
        rules:
          - uid: ExampleRule
            title: ExampleRule
            condition: B
            data:
              - refId: A
                queryType: ""
                relativeTimeRange:
                  from: 300
                  to: 0
                datasourceUid: elk
                model:
                  alias: some_metric
                  bucketAggs:
                    - field: '@timestamp'
                      id: "2"
                      settings:
                        interval: auto
                        min_doc_count: 0
                        trimEdges: 0
                      type: date_histogram
                  datasource:
                    type: elasticsearch
                    uid: elk
                  intervalMs: 1000
                  maxDataPoints: 43200
                  metrics:
                    - field: select field
                      id: "1"
                      type: count
                  query: '"some_query"'
                  refId: A
                  timeField: '@timestamp'
              - refId: B
                queryType: ""
                relativeTimeRange:
                  from: 300
                  to: 0
                datasourceUid: "-100"
                model:
                  conditions:
                    - evaluator:
                        params:
                          - 3
                          - 0
                        type: gt
                      operator:
                        type: and
                      query:
                        params:
                          - A
                      reducer:
                        params: []
                        type: sum
                      type: query
                  datasource:
                    name: Expression
                    type: __expr__
                    uid: __expr__
                  expression: ""
                  intervalMs: 1000
                  maxDataPoints: 43200
                  refId: B
                  type: classic_conditions
            noDataState: NoData
            execErrState: Error
            for: 1m
            annotations:
              __dashboardUid__: Example
              __panelId__: "1"
              summary: ExampleRuleSummary
              description: ExampleRuleDescription
            labels:
              namespace: example
              severity: critical
---

I usually create rules manually and use curl+yq to convert response from Grafana API JSON to YAML and use them in my configmaps.

[mwilfried]

@dragoangel Thx, for your help ! I've found why it's not working.

I get the alert through API, then i convert the json to yaml but while doing this, I don't get all the part of the file needed by grafana, it is missing this part :

    apiVersion: 1
    groups:
      - orgId: 1
        name: Example
        folder: Example
        interval: 60s
        rules:
          - uid: ExampleRule
            title: ExampleRule
            condition: B

Grafana 9.4.3 allows the export of Alert directly in yaml format with all the good informations (but not yet available in kube-prometheus-stack)

[dragoangel]

@mwilfried you always can override image tag to use in helm, and use Grafana version you like in kube-prometheus-stack

[mwilfried]

@dragoangel Thx, with the image tag override it is working !

[mwilfried]

@dragoangel It's ok for me for the dashboards but I didn't manage to make it work for the Alerts. Can you post the yaml of a config map pushing a rule ? It would be very helpful

Regards

[natellium]

Hey there! We want to let you all know that we are moving to Github Issues for feature requests. You can read more about the move here.

I will be closing this discussion, but if you’re still interested in this request being implemented please don’t hesitate to open a feature request. Sorry for the inconvenience, and thanks for your contributions!

cc @grafana/alerting-squad