Confirm annotations of resolved alert have appropriate semantics.

[tangr]

The alert resolved rule notification send the $value with last firing state value, it should be better send the current $value instead, or the last resolved state value, then the resolved notification will be much useful.
https://github.com/prometheus/prometheus/blob/master/notifier/notifier.go#L87-100

Environment

• System information:

Linux 3.10.0-514.26.2.el7.centos.plus.x86_64 x86_64

• Prometheus version:

1.8.0

• Alertmanager version:

0.8.0

[brian-brazil]

We never send the value with alerts. Can you give more detail on what you're asking for?

[tangr]

@brian-brazil
For example, in this alerting rule, I meaning the $value, when this notification resolved, $value of the resolved message are still more than 90, so the $value should be the last firing state, not the state after resolved. Is it more clearly for what I meaning?

ALERT memory_use_warning
IF 100 - 100 * (node_memory_MemAvailable or (node_memory_MemFree + node_memory_Buffers + node_memory_Cached)) / (node_memory_MemTotal + 0.1) > 90
FOR 3m
LABELS { severity = "warning" }
ANNOTATIONS {
short = "{{ $labels.instance }} {{ humanize $value }}%",
detail = "Host {{ $labels.instance }} {{ $labels.job }} memory usage {{ humanize $value }}%",
}

[brian-brazil]

The alertmanager doesn't remember previous annotations, it only has the most recent alert to go on - which is the one from the resolved alert. This is working as expected.

[tangr]

@brian-brazil , I know alertmanager doesn't remember previous annotations, what I understand is the prometheus updated the "endsAt" parameter, after send the notification to the alertmanager, then the alertmanager consider the notification as "Resolved".

What I want is when the prometheus update the "endsAt", it update the "value" at the same time.

[brian-brazil]

That'd need to be looked into further

[brian-brazil]

I've checked our logic, it's working as expected.

[eskp]

Any way to get the updated value in the resolve notification then?

[ojle]

Any way to get the updated value in the resolve notification then?

@brian-brazil
Is there any possibility for this one to be considered, ie. get the latest value in resolved notification?
As mentioned in the blog post https://www.robustperception.io/why-do-resolved-notifications-contain-old-values
"It often confuses users as to why resolved notifications don't contain updated annotations values..."
So if something is constantly confusing users, why shouldn't be considered, and applied a bit of different logic?
It all comes on human psychology that when you receive RESOLVED alert, you want to see current values, like "disk usage on /root is WARNING - 90% | disk usage on /root is OK - 15%"

[brian-brazil]

If you read that blog post, you'll see that this simply isn't possible.