Add hidden labels for outgoing alerts

[roidelapluie]

As part of the alerts metrics relabelling,

I would like to have access to a hidden field

__for

which would be the "for" clause of the alert.

Then I can do:

alert_relabel_configs:
- source_labels: [__for]
  target_labels: backlog

[roidelapluie]

The challenge will be that that __for duration: should it part of the fingerprint of the labelset of the alert?

[brian-brazil]

Why do you want this?

If you want to specify a label, you can add it explicitly in your alerting rule.

[roidelapluie]

some of our notifiers are graphing the alerts and other promql that we pass as annotations; but getting the "for" information once for all the alerts would be a nice shortcut versus needed to ensure that he "for" label is aligned with the "for" duration of the alert. I propose a hidden label by default so that people who have no interest in that are not affected.

[roidelapluie]

  - alert: Degraded availability
    expr: avg_over_time(probe_success[2m]) < 1
    for: 5m
    labels:
      recipient: circuit
      for: 5m
      env: monit
      priority: P1
      alertid: "DCX-002"
      title: Limited Service Availability
    annotations:
      availability_graph_promql: 100*avg_over_time(probe_success{ {{"{{range $k, $v := $labels}}{{$k}}=\"{{$v}}\",{{end}}"}} }[$interval])
      availability_graph_title: Service Availability
      availability_graph_min: 0
      availability_graph_y_label: "%"
      availability_graph_fill: yes
      resp_graph_promql: count_values("code",max_over_time(probe_http_status_code{ {{"{{range $k, $v := $labels}}{{$k}}=\"{{$v}}\",{{end}}"}} }[$interval]))
      resp_graph_title: Return codes
      http_graph_promql: avg_over_time(probe_http_duration_seconds{ {{"{{range $k, $v := $labels}}{{$k}}=\"{{$v}}\",{{end}}"}} }[$interval])
      http_graph_title: HTTP Probe duration/phase
      http_graph_y_label: "seconds"
      http_graph_min: 0
      description: The URL {{"{{ $labels.target}}"}} is not reachable from DCX ({{"{{ $value | humanize }}%"}} available).
      resolved_description: The URL {{"{{ $labels.target}}"}} is now reachable from DCX.

[brian-brazil]

In alerting terms, we already have annotations for the role of non-identifying labels. I don't think this is something that we should be sending by default as it'd pollute the default notifications and others that list all annotations - and thus is also a breaking change. I think this is better served by being handled in configuration management.

[roidelapluie]

ack

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.