Alert provisioning does not support using Environment Variables

[JamesGibo]

What happened:

When provisioning a slack alerting contact point, using an environment variable to set the URL, the provisioning template does not expand the environment variable. The same process works fine for other config files like when setting the URL for a Datasource.

https://grafana.com/docs/grafana/latest/administration/provisioning/#contact-points
https://grafana.com/docs/grafana/latest/administration/provisioning/#using-environment-variables

What you expected to happen:

The provisioning template should expand the environment variable in alerting provisioning.

How to reproduce it (as minimally and precisely as possible):
Use the following configuration file and reload grafana

# config file version
apiVersion: 1

contactPoints:
  - orgId: 1
    name: slack_1
    receivers:
      - uid: slack_1
        type: slack
        settings:
          # <string, required>
          recipient: alerting-dev
          url: ${SLACK_WEBHOOK}
          title: |
            {{ template "slack.default.title" . }}
          text: |
            {{ template "slack.default.text" . }}

Anything else we need to know?:

Environment:

• Grafana version: v9.1.2
• Data source type & version:
• OS Grafana is installed on: Docker
• User OS & Browser: Chrome, Ubuntu 20.04
• Grafana plugins:
• Others:

[zuchka]

Thanks for creating this issue, @JamesGibo

this sounds similar to the issue fixed by this PR. Perhaps helpful?

#54293

[bygui86]

Same issue here with Grafana v9.1.4
Here my screenshot:

@zuchka sorry but I don't think this issue is related to the one you mentioned.

@JamesGibo I have issues loading notification policies through provisioning, do you?
I'm not sure if alerting provisioning is 100% ready, I see tons of issues related to that in the last days.

[romankabakov]

I have the same issue with PagerDuty integration:
integrationKey: ${integrationKey}
Throw me an error

I did check the environment variable exists and is set correctly.

Grafana version: v9.1.4 (2186d0bbeb)

[JamesGibo]

I have issues loading notification policies through provisioning, do you?

I have managed to load notification polices OK through provisioning, but the alert provisioning documentation is lacking some key information and clear examples, I have had to look at the code a number of times.

[JamesGibo]

Thanks for creating this issue, @JamesGibo

this sounds similar to the issue fixed by this PR. Perhaps helpful?

#54293

@zuchka I don't think this issue is related, this PR bypasses interpolation for query data for alerting rules, where I have seen the issue is configuring the contact point not the rules, these are 2 separate files.

Interestingly the alert rules file does interpolate other fields meaning I had to escape the $

        # <duration, required> for how long should the alert fire before alerting
        for: 60s
        # <map<string, string>> a map of strings to pass around any data
        annotations:
          summary: "High CPU usage {{ $$labels.host }}"
          description: "Average CPU usage for host {{ $$labels.host }} is above 60%"
        # <map<string, string> a map of strings that can be used to filter and
        #                      route alerts
        labels:
          severity: Warning

[bygui86]

@JamesGibo can you please paste here an example of notification policies yaml file?
I'm struggling a lot and the documentation lacks quite important information.
I'm able to load only 1 notification policy as default, the last listed in yaml file, all others are kind of ignored or overwriting each other...

[JamesGibo]

@JamesGibo can you please paste here an example of notification policies yaml file? I'm struggling a lot and the documentation lacks quite important information. I'm able to load only 1 notification policy as default, the last listed in yaml file, all others are kind of ignored or overwriting each other...

I only have a single notification policy so have not tested multiple I am afraid.

[bygui86]

@JamesGibo ok no worries, but I have also another issue. Once the single notification policy is provisioned, then I'm not able to create other policies in Grafana UI. This sounds really weird... or am I missing something?
Here a screenshot

Under section "Specific routing" I should be able to create further policies right?
What about your experience?

[JamesGibo]

@JamesGibo ok no worries, but I have also another issue. Once the single notification policy is provisioned, then I'm not able to create other policies in Grafana UI. This sounds really weird... or am I missing something? Here a screenshot Under section "Specific routing" I should be able to create further policies right? What about your experience?

I would create a new issue for this as it is not related to the current issue, but i see the same.

[bygui86]

@JamesGibo just wanted to check with you.
I created a new issue right now: #55073

[JamesGibo]

Issue is still present in Grafana v9.1.5

[bygui86]

As alerting system is made of different components (contact points, templates, notification policies and alert rules) it would be really nice and important to check that each of them supports environment variables during provisioning.

[JamesGibo]

Issue is still present in Grafana v9.1.6

[armandgrillet]

Thanks for creating this issue, an engineer at Grafana Labs will look at this bug during the upcoming week.

[JohnnyQQQQ]

This will be fixed in 9.1.7 and 9.2.X

[jaygridley]

Suppose this work when calling provisioning API directly, i.e. Terraform? Seems not. We are running 9.2.0