Template expansion on ALERT labels only happens upon sending to Alertmanager

[beorn7]

I might have missed historical context here, let's see...

If I use templating in ALERT labels, e.g.

ALERT FooBar
IF ...
LABELS {
  severity = "{{$labels.the_other_severity}}"
}

then the alert timeseries is saved with a literal severity="{{$labels.the_other_severity}}" label, while the label value has been expanded as expected when it is sent out to Alertmanager.

I believe that breaks the semantics as the expansion might change value depending on context without creating a new time series.

Or is that all expected and intended? (In which case we might want to document it.)
@fabxc @brian-brazil

[brian-brazil]

That isn't expected behaviour, the time series should have the expanded form.

[beorn7]

OK, marking this as bug.

[beorn7]

Lowering to P2. Label templating is relatively niche as a use case, and then certain things have to happen to make this bug harmful.

[StianOvrevage]

For the this is a bit of a nuisance. I'm writing our own alert GUI frontend and I use the ALERTS meta timeseries to get the list of active alerts. But, I'm not able to actually get the $value that caused an alert since the $value in ALERTS are 1 or 0. So I'm trying to add metrics.value = $value in LABELS { } but that does not work since the string is not actually expanded and just shows up as

test_instance="{{ $labels.instance }}" test_value="{{ $value }}"

in both the ALERTS timeseries and Prometheus own Alert GUI.

Another possibility would be to also include the actual $value (in addition to 1 and 0) in the ALERTS timeseries.

[fabxc]

As $value is often changing on each evaluation writing into a label is
generally not sane. It essentially generates a different alert.

The correct way to query this is evaluating the generating alerting
expression(s) dynamically.

On Sun, Jun 5, 2016, 4:20 PM Stian Øvrevåge notifications@github.com
wrote:

For the this is a bit of a nuisance. I'm writing our own alert GUI
frontend and I use the ALERTS meta timeseries to get the list of active
alerts. But, I'm not able to actually get the $value that caused an alert
since the $value in ALERTS are 1 or 0. So I'm trying to add metrics.value =
$value in LABELS { } but that does not work since the string is not
actually expanded and just shows up as

test_instance="{{ $labels.instance }}" test_value="{{ $value }}"

in both the ALERTS timeseries and Prometheus own Alert GUI.

Another possibility would be to also include the actual $value (in
addition to 1 and 0) in the ALERTS timeseries.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1677 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AEuA8u6rIP-jfTtwefOxLthriCx92kzpks5qItsTgaJpZM4IomEA
.

[StianOvrevage]

I understand. Would it be better to instead insert the originating query automatically into a separate label so that the alert GUI can query that instead? I'm guessing that is where generatorURL comes in in alertmanager?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.