-
Notifications
You must be signed in to change notification settings - Fork 12k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ngalert] Missing alert notifications in Grafana 8.1.3 and 8.1.4 #39295
Comments
Confirmed, no email alerts since v8.1.3
|
so 8.1.4 did not fix this? #38983 |
Considering that I have created this configuration for ngalert after v7.2 and that encryption does not cover email receivers, I believe that this would have absolutely nothing to do with reported issue. Others seem to confirm that the issue occurs on 8.1.4 as well (check #39009 (comment)) I also don't see how linked PR could affect only 8.1.3+ but not 8.1.2. On unrelated note, the linked PR fixed migration however for affected users this will change nothing, as migration already ran. |
Not notified on 8.1.3 and 8.1.4 |
Having the same issue with 8.2.0-beta1. |
no notification also for grafana 8.1.4 as docker container |
I can confirm the issue too |
I do see in my logs... Does this helps |
Same here (8.1.3 on FreeBSD). E-Mail + Pushover does not work (nothing else configures) |
I'm afraid I have not been able to reproduce this issue on either 8.1.3 or 8.1.4. It doesn't appear to be related to email notifications as others have replied this affects other notifications too, such as Pushover. This is the configuration I have been using:
and I created an alert rule with the following configuration: I can see that Grafana is attempting to send an email with both smtp to localhost (which I do not have a listening smtp server) and I also confirmed that a webhook notification can be sent to an HTTP server that I have also listening on localhost.
|
@gerobinson I assume the issue affects alerts directly managed by grafana only. |
@staeglis I'm not sure, @ImmoWetzel appears to be using Prometheus? I also created an alert on a CSV but I'm afraid I am receiving notifications for that alert also without issue. |
I use ngalert with prometheus as datasource
![image](https://user-images.githubusercontent.com/2955080/134207977-3adb4754-c25d-4331-9436-7dcfe0916efc.png)
But I also used a contact point which contains email and MS Teams…
Maybe just tell me how to create required logs
With kind regards
Immo Wetzel
|
On version 8.1.4, I have recreated more or less what George did. It still did not issue any notifications, and in their first appearance in the logs, everything displays as "resolved". I created this contact point: Then this notification policy: And the alert with these settings: This is what comes up in the logs as the first mention of these test alerts. No notification is ever attempted to be sent.
|
I created a simple Testalert: Alert name = Test 2 Contact points (Pushover and E-Mail) - worked before 8.1.3 Debug output:
I just downgraded to 8.1.2 and changed nothing else. After restarting I immediately received the Pushover notification:
Maybe a side note, but: the Tab "Notifications" in 8.1.3 was empty. In 8.1.2 there is a list of groups. "No grouping" has the firing alert. |
I've observed this too |
I have found the issue and identified the change in 8.1.3 that caused it. The bug occurs when an alert rule is configured to evaluate at an interval that is greater than the interval at which Grafana resends firing alerts to the internal Alertmanager to stop them resolving, which in the 8.1.3 and 8.1.4 releases is 30 seconds. To fix notifications in 8.1.3 and 8.1.4 it is acceptable to set the alert evaluation interval for an alert rule to an interval that is 30 seconds or lower. |
Can confirm. Setting "Evaluate every" to "20s" (instead of 1m) and it works again (8.1.3). Thanks! |
What happened:
Some users are reporting missing alert notifications on Grafana 8.1.3 and 8.1.4 which is fixed when downgrading to 8.1.2.
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
The text was updated successfully, but these errors were encountered: