ALERT labels do not overwrite.

[beorn7]

https://prometheus.io/docs/alerting/rules/ reads (about LABELS in ALERT rules): "Any existing conflicting labels will be overwritten."

That doesn't seem to be true. Existing labels are left intact, and the newly added label is ignored in case of a conflict.

Either we need to fix the docs, or the code.

What's the intended behavior? @brian-brazil @fabxc

[fabxc]

Overwritten should probably be used as the labels from the rule are more likely to be important in routing.
We can go the exported_ route to resolve conflicts with overwriting again.

[beorn7]

OK, important is to bring the behavior in line with the documentation then. I'm not sure if we really need the prefixing. It's only for an alerting rule, so we are not really throwing data away. It's all under control of the user.

[juliusv]

If the rule labels have precedence, what happens when overriding a label results in duplicate alerts? Should they just be deduped then?

[matthiasr]

yes, or if it's easier, sent double and deduped in Alertmanager

[brian-brazil]

Prefixing isn't required (can be done by an additional label if needed), and with the default receiver templates would just cause noise for those using it.

[beorn7]

If the label override creates duplicate alerts (== time series), it should be handled in the same way as we handle duplicate time series created by rules in general (via label replacement or plainly by duplicate rules). At the moment, we just silently feed the results into the same time series. That's not ideal, but if we want to change the behavior, we should do it consistently and not create a one-off solution just for LABELS in an ALERT rule.

[brian-brazil]

The is a bug in currentAlerts, the overwrite is going in the wrong direction so it'd ending up as a default instead.

[brian-brazil]

I have a fix, unittests will take some time though.

[brian-brazil]

Fix is at https://github.com/prometheus/prometheus/compare/am-label but github won't let me create a PR.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.