Add template function to get the alert name in the templates

[iksaif]

What did you do?

ALERT GraphiteLowAvailability                                                                                                                                                                                     
      IF dc:http_availability:value{endpoint="graphite_render",duration="10m"} < 0.95                                                                                                                             
      FOR 20m                                                                                                                                                                                                     
      LABELS {                                                                                                                                                                                                    
        perimeter="observability",                                                                                                                                                                                
        severity="page",                                                                                                                                                                                          
      }                                                                                                                                                                                                           
      ANNOTATIONS {                                                                                                                                                                                               
        summary = "Low availability for {{ $labels.dc }}",                                                                                                                                                        
        description = "Metricroulette from {{ $labels.dc }} found an availability under 95% over the last {{ $labels.duration }} (current value: {{ $value }}%)",                                                 
        documentation = "https://playbooks/Graphite+Playbook#GaphitePlaybook-Alerts#{{ $alertname }}"                                                                            
      }  

What did you expect to see?

documentation = "https://playbooks/Graphite+Playbook#GaphitePlaybook-Alerts#GraphiteLowAvailability"

What did you see instead? Under which circumstances?

documentation = "https://playbooks/Graphite+Playbook#GaphitePlaybook-Alerts#"

Notes

Currently it looks like the template name contains the alertname, but I didn't find a way to access it.
An easy way to do that would be to add an alertname label and simply do $labels.alertname

If we can agree on a way to implement it, I'll be happy to do it.

cc: @Thib17

[fabxc]

{{ $labels.alertname }} should do the trick.

[brian-brazil]

This is something that you'd be better off doing from the Alertmanager side, rather than duplicating that line in every single alert.

[iksaif]

@fabxc {{ $labels.alertname }} doesn't seem to work, at least with 1.6.3 (I think it reads the labels from the timeseries, not from the alert)

@brian-brazil : While true, that would still be nice to have. Prometheus rules are easier to manage by users than alertmanager config.

[brian-brazil]

The alertmanager config only needs to be setup once.

[iksaif]

Different teams have different playbooks stores and different templates depending on the destination. But sure, in another context doing it could be cleaner.

Still, I think there is value into exposing $alertname or $labels.alertname here.

[brian-brazil]

It's not a label produced by the expression or added manually. The recommended way to do this is either via configuration management or notification templates.

[iksaif]

As I said, notifications templates in the routing tree are already not super easy to manage for us. Even less because while the prometheus isntances are handled by the users directly, the alertmanager instance is centralized.

That being said, event if the recommended way is to do it in the alertmanager, I don't think it would hurt to expose more data to the alert templates.

This could be as simple as:

diff --git a/rules/alerting.go b/rules/alerting.go
index 6a489cfa..5ef30f35 100644
--- a/rules/alerting.go
+++ b/rules/alerting.go
@@ -172,6 +172,7 @@ func (r *AlertingRule) Eval(ctx context.Context, ts model.Time, engine *promql.E
                for k, v := range smpl.Metric {
                        l[string(k)] = string(v)
                }
+               l[model.AlertNameLabel] = model.LabelValue(r.name)
 
                tmplData := struct {
                        Labels map[string]string
@@ -182,7 +183,7 @@ func (r *AlertingRule) Eval(ctx context.Context, ts model.Time, engine *promql.E
                }
                // Inject some convenience variables that are easier to remember for users
                // who are not used to Go's templating system.
-               defs := "{{$labels := .Labels}}{{$value := .Value}}"                                                                                                                                              
+               defs := "{{$labels := .Labels}}{{$value := .Value}}{{$alertname := .Labels.alertname }}"                                                                                                          
                                                                                                                                                                                                                  
                expand := func(text model.LabelValue) model.LabelValue {                                                                                                                                          
                        tmpl := template.NewTemplateExpander( 

We could also just set it to $alertname directly.

[brian-brazil]

I'm against this, as it causes needless repetition in alerting rules.

The alertmanager is where making your notifications look nice and useful belongs.

[iksaif]

As I said, doing that in the alertmanager doesn't seem to be as easy for us.

Let's say that I have:

• Team A taking care of Product Foo, with playbooks on playbooks/Foo/#Alertname
• Team A taking care of Product Bar, with playbooks on playbooks/BarTeam/#Alertname
• Team B taking care of Toto, with playbooks on stackoverflow/Toto/search?=#Alertname

Regarding templates:

• Team A and Team B have different notification templates, because they want different stuff.
• Team A and Team B also have different notification templates for different receivers (for example opsgenie has a specific documentation field, but for slack I just to add the link to text).

Now I need to effectively duplicate the logic to build the playbook url in all the templates that want to use it, instead of just being able to use $ .CommonAnnotations.documentation.

Also, let's stay that I want to use something else that alertmanager, I'll also need to duplicate that.

But maybe there is a smarter way to do that in the alertmanager that doesn't involve too much duplication ? (yeah, sure, it would be easier if there was a single http://playbook/$alertname destination, but that's currently not the case)

[brian-brazil]

Now I need to effectively duplicate the logic to build the playbook url in all the templates that want to use it

You're proposing here to duplicate that logic in every single alert. Doing it in 4 places per team (it'd usually be 2-ish) seems simpler to me.

(yeah, sure, it would be easier if there was a single http://playbook/$alertname destination, but that's currently not the case)

It'd be pretty rare that that would be the case across an organisation. Usually you'll only have consistency within a team.

What I have seen is that the annotation would be an override, for the odd exception that crops up.

[iksaif]

You're proposing here to duplicate that logic in every single alert. Doing it in 4 places per team (it'd usually be 2-ish) seems simpler to me.

Yep, because in our case we have more like ~4 severities per team (debug, ticket, urgent-ticket, page) times two (because, prod, preprod .. don't ask).

[brian-brazil]

I don't think we should be adding features that would be misused by the average user, merely because some users have more esoteric use cases.

Especially as if you need different message for prod vs preprod, then the feature you're requesting doesn't help you - as you'd need a completely separate set of rules files for preprod for this to work in the way you suggest.

[satterly]

Thought I should note here that the suggested fix by @brian-brazil to use notification templates in alertmanager is not supported for webhooks. Hopefully this will save others a lot of confusion and wasted time.