-
Notifications
You must be signed in to change notification settings - Fork 12k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Alerting support for queries using template variables #6557
Comments
+1 |
|
+1 |
@bergquist it's unpractical using all for example when you have more than a dozen hosts. If for example only few of them are failing, (let's say 5), it is very useful to receive an email for each failing alert. This way is also much easier to integrate with other tools which in general expect one alert per metric. The current approach (using all) is pretty neat though when there are fewer instances or when you are alerting at service level (eg. # of jobs in queue). |
what @calind said, i've got multiple $host variables wich are working fine with the influxDB but not with the alerts |
+1 as well. Just a thought, since you are able to query with a template variable, wouldn't you just be able to do the same query with the alerting metrics and maybe iterate through the results to see which meet the alert criteria? |
@NotSoCleverLogin It would be possible. But would you want to change the behavior of alert rule based on what template varlue are selected? Using the all option for the template is the only way that makes sense for me. |
+1 I have a setup of X environments with the same components in each environment. We are currently using prometheus to alert on e.g cpu usage/disk usage etc. There we specify an alert for a query, and when the alert is triggered it will just state which environment the alert was triggered from. If we would do this with the All variable, that would work to some extent. But, using @calind's example, the screenshot would be filled with the trend of all cpus from all of my environments, and not just the environment where I would want to be informed about said problem. The graph will (or can) be obscured with information from other environments. In some scenarios it could be interesting to compare cpu in other environments, but there are no guarantees that what is happening in a test environment is happening in our production environment, etc. We are also looking into creating dashboards that can be used by operations, showing annotations for alerts in the "standard" overview dashboard. Given that we use 'env' template variables for these kind of dashboards it's not really possible for us to do that with how it is implemented right now. I would have to manually (at least to some extent) generate a "shadow" dashboard where the alerts are triggered (which makes me loose the annotations in the overview dashboard). Another thing I think template variables can help you do is to route the alerts (should you choose to implement such a feature) to different sources (some to operations if in production, to qa/developers if in test environments etc). |
+1 for supporting alerts on templated queries. |
@bergquist, some dashboards don't have an All option. For example system metrics by collectd (https://grafana.net/dashboards/24). Having an All option would certainly not be practical for let's say 10 or more servers. That's why the need to iterate trough template variables. |
Allowing use of All is a good and welcomed start. In Prometheus, queries need to be written in a different way to allow All:
Notice the extra tilde there, allowing for regular expression searching (and the wildcard in All). I have not benchmarked the possible performance impact of going from a straight query to a regex search query but at least for now it would apparently solve our problems. |
+1 |
1 similar comment
+1 |
not sure how it should be implemented, just know it's needed.. |
+1 |
+1 |
+1 |
perhaps if there was an option or simple way to save/export a dashboard with the template variables backed/pre-rendered into all the fields... this would perhaps be a good half way point until another solution is found. |
+1 for supporting alerts on templated queries. We currently use templating on all our dashboards so can't take advantage of this really cool feature. |
+1, we have a lot of templated dashboards, and we can't use alerting for now, we have to deduplicate dashboards for having alerts, and we so lose templating power |
+1, Almost all of our dashboards use template variables (and nested template variables). We would like to be able to set alerts on repeat panels to get individual alerts per template-variable group if needed. Plus this means that the alerting is dynamic and not super manual as it is now. DANGER: Variables in theory will be good to have, but we need to keep in mind that if some guy goes into your dashboard and changes the value and saves, the resulting alerting will be affected. Don't know if that's ok behaviour or not, will be complicated. |
+1 |
When working with grafana it feels like templating is encouraged everywhere and it feels wrong to create an extra set of graphs not using variables just to use the alerting feature... |
+1 for supporting alerts on templated queries. |
Hello Grafana community, the Grafana team has picked up the work on Alerting and we're in the process of redesigning it to make the best possible alerting experience happen 🔥 🚀 We would love to find out more about your needs as our beloved users. So if any of you are willing to have a 30-minute interview with me, Update: I got so many e-mails in such a short time, you all rock! I'll be reaching out to everyone who sent e-mails, we have enough interviewees now, thank you <3 |
The new beta version of alerting in Grafana 8 (opt-in with "ngalert" feature toggle) has moved alerts out of dashboards, so alert rule queries based on template variables no longer fits into the model. The new version does support "multi-dimensional" alerting based on labels which seems to be close to the core functionality underlying this issue. So one can have multiple alert instances from a single rule. More at #7832 (comment) . |
Is there a way to cover this use case: |
@fkaleo It doesn't seem to currently do this. The resolution of those variables when clicking the "create alert" from the dashboard in the system would be on the frontend side which I know less about it. But it sounds doable and like a good idea to me - can you create a new enhancement request for that? |
Can't find ngalert feature toggle |
If using [feature_toggles]
enable = ngalert |
@torkelo I've been keeping up to date with the latest Grafana releases hoping that eventually I'd be able to use variables in alert queries. I tried ngalerts and while I didn't spend a lot of time with them I had the following thoughts:
So for net-new dashboards and alerting ngalerts probably is a great thing. But what happens to many of us who are not using ngalerts and simply want to use a contant template variable in the alert query or in the alarm message itself? I still don't understand why template variables can't be used with alert queries. I use template variables heavily, whenever I can so that I have precise control over my dashboard queries. In that respect there is no problem. It is only when I have an alert attached that things go sideways. That is an "impedance-mismatch" I don't understand. |
Hi all, had a question about adding alerts. it’s suggested that the query triggering the alert can have labels from the query templated into the alert message. As per the documentation, this is how: Refer to the alert query labels in the alert rule name and/or alert notification message field by using the ${Label} syntax. Who knows where the ${Label} was created? Thanks so much. |
Yes, I don't understand why the Grafana dev team does not want to support this while so many many people are eager for this. |
someone may think because their business model is to make the software just good enough and make people pay for support to code the other little things missing.
did someone try to PR this? or pay for it?
22.03.2022 11:38:42 lujinke ***@***.***>:
… Yes, I don't understand why the Grafana dev team does not want to support this while so many many people are eager for this.
—
Reply to this email directly, view it on GitHub[#6557 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AAZQPLV2QTELD5ZDOLNAVV3VBGPLDANCNFSM4CWB7ZFA].
You are receiving this because you are subscribed to this thread.[Verfolgungsbild][https://github.com/notifications/beacon/AAZQPLWPHSTAZMHDVPL2AN3VBGPLDA5CNFSM4CWB7ZFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIAJU4SY.gif]
|
hi, not sure if such idea was mentioned before, but maybe it's possible to have two sets of variable values:
this way alerts would not be affected by someone changing variable values and saving the dashboard. |
Any progress on this Grafana? Sorry to be a nuisance but this pretty much invalidates the template variables for me. |
Hey, Any update on this? |
It's not quite clear what sort of further updates people are expecting on this. This feature has been implemented (per comment above) and the issue is correspondingly closed. Additional comments (and I'm conscious I'm right now a culprit here...) simply generate spam for the 800+ people still subscribed to the issue. Follow-up bug reports or enhancement requests ought to be captured in separate issues, while questions about the functionality can be addressed in the community forum. |
@amohamedhey @svet-b @Tomasz-Kluczkowski My understanding is that this ability was added to Grafana for Unified Alerting (introduced in Grafana 8). I don't think a lot of new feature work will be going into the legacy alerts as Unified Alerting is the "new normal". That is my understanding. @svet-b please correct me if I'm wrong. |
Hey, Any update on this? |
It would be pretty useful if grafana would support alerting for queries using template variables. The way I see it work it would be as follows:
The current workaround is to use an invisible wildcard metric, but the problem I see with this approach is that it loses context.
The text was updated successfully, but these errors were encountered: