Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An option to disable the default sorting of Alerts for a group of Alerts #3901

Open
ricardogpsf opened this issue Jun 25, 2024 · 5 comments
Open

Comments

@ricardogpsf
Copy link

ricardogpsf commented Jun 25, 2024

Disabling the default sorting of Alerts in a group of Alerts would be helpful.

I saw old discussions in this repo about features like setting what label to use to sort the alerts and the chosen solution to sort alerts by their label values using job and instance as main labels (I can see the sorting code here).
But I think having the option to disable the default sorting would be good.

For example, my case is I have a Loki query configured as an alert and I already sort the result by values as I want. Still, alertmanager sorts the result by job, instance, and other labels, if we could disable the sorting the result would be the expected one.
That option could be enabled by default and disabling would be a configuration.

What do you think?

Thanks for your time.

@grobinson-grafana
Copy link
Contributor

Hi! 👋 How would you want Alertmanager to sort alerts in such case? Lexicographically by labels?

@ricardogpsf
Copy link
Author

By number values, it’s a query of erros counting and the those values are numbers (integers, I think).
I already sort the query result by those count values on “Loki side”, so having no sorting in alertmanager side would make sense for that type of cases.

@grobinson-grafana
Copy link
Contributor

Hi! 👋 I'm not sure what you mean by number values? Alerts in Alertmanager are basically a set of labels, annotations, a start time and an end time. That's the full set of data we can sort with, so we could sort newest alerts first as an alternative, for example.

@ricardogpsf
Copy link
Author

Sorry, I used the wrong terms.
There is a value associated with the alert, no? Not at the alertmanager side, but related to the query we use to fire an alert.
Like, in a query that triggers an alert when DB CPU usage is above 80%, we would have a value associated to the alert that could be 95% and we can show it in the alert description. I know that value itself does not go to alertmanager side, but the Alerts in an alert group go in a certain order depending on the query executed, right?!
So would be possible to keep the original order that alerts go to alert manage would be helpful without sorting them when they are on alertmanager.
Let me know if I'm wrong in the way I understand the alertmanager behavior (probably I'm).
Again, Thanks for you time.

@grobinson-grafana
Copy link
Contributor

There is a value associated with the alert, no?

It's not a strict requirement from Alertmanager. But as mentioned, alerts that come from Prometheus need a query that returns some kind of value.

I know that value itself does not go to alertmanager side, but the Alerts in an alert group go in a certain order depending on the query executed, right?!

I'm afraid that's not the case. Alerts in Alertmanger are not ordered, and Prometheus does not send them in a deterministic order. In addition, alerts can come from multiple Prometheus servers at the same time (even in the same groups).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants