Alerting: No Data should take "For" setting into account

[azhiltsov]

What happened:
Alert notification send immediately after "No Data" condition
What you expected to happen:
Alert notification should be delayed the same way as for "Alerting" condition obeying "For" setting
How to reproduce it (as minimally and precisely as possible):
Create alert rule
set 'If no data or all values are null' to "No Data"
set For "5m"
simulate "No Data" condition from the data-source side
Receive notification immediately

Anything else we need to know?:

Environment:

• Grafana version:
  5.4.3
• Data source type & version:
  carbonapi (graphite 0.12 compatible)
• OS Grafana is installed on:
  Linux Centos 7
• User OS & Browser:
• Grafana plugins:
• Others:

[kylebrandt]

I think the logic in pkg/services/alerting/eval_context.go > GetNewState() / getNewStateInternal() needs to change. GetNewState() will return on a no_data since it is no AlertStateAlerting , and this happens before the For logic could determine AlertStatePending.

[marefr]

Re-labeled this as a feature request since the For feature was not built with No Data in mind.

[patricktokeeffe]

In current releases (e.g. 6.4.3), the default alert rule contains both

• For: 5m
• and If no data or all values or all → No Data

At the same time, documentation notes:

Do not use For with the If no data or all values are null setting set to No Data. The triggering of No Data will trigger instantly and not take For into consideration. This may also result in that an OK notification not being sent if alert transitions from No Data -> Pending -> OK.

Because 'No data' is about handling errors for a rule and "For" applies to the rule evaluation (not the conditions/query), I think the type/bug label was appropriate.

[marefr]

@patricktokeeffe thanks. I agree and your comment make sense. Changed to bug.

[saiful-wartek]

As this issue is marked with prio/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough interest in it. , we want to report that we are also impacted with this issue on Grafana 8.5.21.

This is how our alert is configured:

The State History panel showed us this history:

As you can see, on Mar 14, 2023 15:53:42, the alert transitioned into OK state, but on Mar 14, 2023 15:55:40 the alert transitioned into No Data. As the "For" field is set to 5 minutes, the notification should not be sent immediately, but in reality it was sent immediately.

Happy to provide more details if that helps this issue moving through!

[justfortherec]

There has not been any progress on this issue for a few years and releases now. Is there any way how this could be put on the agenda again? Or do you recommend looking for workarounds and alternatives?

[JacobsonMT]

Hi @justfortherec, there's been a bit of progress here as of late. Grafana Alerting rules with NoDataState configuration set to Alerting will now respect "For" duration (See PR). The other parts of this issue are being tracked here: #55320

[Martchus]

We see the same issue on our instance - except that it happens with DatasourceError. Strangely, it worked at some point and only ceased to work recently. When it worked and when it ceased to work we were using version 9.4.7. Switching to 9.5.1 didn't help. We've been using the new alerting system for a while now. Just for additional context, that's the issue we're tracking this on our side: https://progress.opensuse.org/issues/128870

[manojm321]

I see a similar issue with 10.1.1

Alert settings:

For is set to 5min and eval to 1m

Documentation states:

Alert rules that are configured to fire when an evaluation returns no 
data or error only fire when the entire duration of the evaluation 
period has finished. This means that rather than immediately firing when
 the alert rule condition is breached, the alert rule waits until the 
time set as the For field has finished and then fires, reducing alert noise and allowing for temporary data availability issues.

But from he logs it's clear it fire immediately on error without waiting for 5min.

04:25:06 state=Normal
04:26:08 DataSourceError
04:26:08 Sending alert..

[jdverman]

We experience the same issue with AWS managed Grafana 9.4: DatasourceNoData immediately sending an alert instead of waiting for the 5 minute evaluation period. Please look into this!

[KaplanSarah]

The new 'Keep Last State' option as of 10.4 will respect the 'for' period.