-
Notifications
You must be signed in to change notification settings - Fork 12k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
alert not working with templating #6230
Comments
Template variables are not supported in alerting yet. Seems like the warning about it does not support influxdb queries. Ill fix it |
when will it be supported ? |
Not sure when templating will be supported. How would you expect templating to work in this case? |
Any estimate on when this will support Graphite? |
Would be very nice to have this |
Having same issue with Graphite as well. |
Same issue with Graphite too. |
Is there an open feature request? Looking for this as well. Thanks. |
Same issue with prometheus |
Since I'm not sure what the alternatives would be without using variables, I'd like to see this feature implemented as well. |
the alternative is to write alert queries that are wider in scope, that include wildcards/regex so they target multiple series. No need for variables. variables are for exploration & dynamic dashboards & dynamic filter. |
@torkelo First time using Grafana (or any other metrics related program) so I'm not very familiar with the concepts... So, if I have a query like this one |
yes, it will check all series. But currently it does not keep state per series. So if one server has high CPU the alert rule will trigger (With the info on which server) and send out a notification, but a minute later another server also has high cpu it will not trigger new notifications as the alert rule is still in firing state. |
so if I have a dashboard that creates dynamic graphs using Template Variables, then I should create a separate dashboard that doesn't use Template Variables that I would use for alerting on? |
@MrMMorris yes |
I want to alert if replicaset lag is more than 3600 seconds. It should mail. Is it possible to do it Graphana ?? |
@MrMMorris you don't need an entirely new dashboard, you can just duplicate the query, fill in the variable with what the alert should handle and hide the query from the graph. |
Templates define a view and thus it doesn't make sense to use them with alerting. But alerting can be set up using the template variable's underlying query, and I think in most cases this is what people are looking for. @bergquist 's comment over on #6557 sums it up well I think:
Something similar to the solution he describes is easy to set up once you know what to do (keep your "view" query using templates, and set up a hidden query using wildcards for the alert):
Here is an example set up with query "A" as my "display query" and query "B" as the "summary query" I use for alerting (aggregated with max over the series & tested w/ max in the alert tab as well). I'm posting this and suggesting a change to the docs/message because 30min ago when I came across this thread I thought I was going to have to create a graph for each of my servers in order to make alerts work. I suspect a lot of the angst above comes from similar misunderstandings. |
Hi, Sorry to open this topic. –=Solution=– Kind Regards, I understood that is not possible to have Alert in Grafana per each server present in the same Dashboard. If yes kindly give an example or the best recommendation. |
@fchiorascu have you tried my suggestion above? (Set up two queries: one for the "view" using templates, and a second hidden query using wildcards for alerting). |
@7yl4r so I have tried that technique and I can get it to alert, but it doesn't tell me what server it is alerting on. Is your solution supposed to provide that? Cause I couldn't find a way to figure out how to pass the specific server that is alerting. |
@MrMMorris : My solution is not able to tell you which series is alerting. For mine I can tell which series is alerting by looking at the graph included with the alert. |
I'll come back with a feedback till EoW regarding your proposal. |
@7yl4r Hi, two things are there from your provided solution.
Due to this behavior of alerting, currently alerting withing grafana is difficult to use. |
You are absolutely correct on point (2). As for (1), you are technically correct, but here is a screenshot of one of my alert messages: It doesn't explicitly tell me which series is causing the alert, but I can clearly see it is the |
I don't understand, are you arguing against making this feature available because you can't imagine why someone would need it? I can understand if this is what's available for the time being, but your wording makes it sound like you are pushing back against this being possible. The fact that I have to go to my monitoring frontend, login, find the dashboard, find the graph, and decipher which is alerting is many more steps than seeing an email alert and connecting to a server. |
Sorry if that came off as dismissive. I was trying to say that (2) should probably be prioritized over (1) unless a user story can be defined to demonstrate the issue.
Does this happen to you? That is a screenshot of the email I received; only one step. I guess there are probably other alerting configurations for which (2) would be more important. |
No worries, I didn't think that was your intention 😄 I will provide a rebuttal to being able to easily identify the server from the plot:
Hint: There are 3 servers that go above 15GB, but two of them overlap almost perfectly and one is barely visible on the black background and just barely pops above it ¯\(ツ)/¯ |
@MrMMorris our solution to the "mucks up the dashboard" is that we just have two different versions of each dashboard. One for people to look at in normal operations, one for alerts that only gets looked at when there's an alert. |
@sparr yea that has been suggested before, but solves IMO the least pertinent problem |
Hi, Sorry to be a thread excavator, but I was wondering if anything has changed in this regard?
Kind regards, Josh/Waffles |
@WafflesMcDuff to be clear, you just need one panel per server, not a whole dashboard. My solution to this class of problem has been to make my own templating system with a script that uses the API to download a dashboard then reproduce panels/rows with the correct names in them. I use it for environments instead of hosts, but the principle is the same. |
really don't understand why it's not implemented... |
Guys, everyone is waiting for this to be implemented, and for this reason we are all subscribed to this thread to get an email once it is done. But instead we receive silly "+1" messages and questions about milestones. Have some patience please, or otherwise take the situation in your own hands and send a pull request. Thank you |
Lol, patience, of course: 2 years this feature was suggested.. and has been closed :D |
+1 |
Is this feature added in newer version? |
ping |
Duplicate of #6557 |
Please continue any discussions regarding support for variables in alerting in #6557. |
i'm new in grafana and having difficulties to make the second query any help ? |
when using templating with host, the alert function is working abnormal. if I clear the templating setting in graph, the alert workings fine.
Please include this information:
the newest version
influxdb
centos 7
using templating with alert
should work together
alert works abnormal
IMPORTANT If it relates to metric data viz:
The text was updated successfully, but these errors were encountered: