[Feature Request] Notify if an alert is triggered regardless of previous state

[ghost]

What Grafana version are you using?

Grafana v4.0.0-beta2

What datasource are you using?

InfluxDB

What OS are you running grafana on?

Ubuntu 14.04

It appears that (after performing tests and looking at code) notifications are only fired when an alert's state changes.

I'm not sure if this is on the roadmap/radar but it would be good to be able to send the notification as long as it evaluates to alerting, regardless of the previous state. The concern here is that the alert notification goes unnoticed, and in the worst case, the time to discovery becomes very long. With repeat alerts, the hope is that the chance of missing it drops for each repeated alarm.

grafana/pkg/services/alerting/result_handler.go

Lines 53 to 89 in 05d4399

	if ctx.Rule.State != oldState {
	handler.log.Info("New state change", "alertId", ctx.Rule.Id, "newState", ctx.Rule.State, "oldState", oldState)
	cmd := &m.SetAlertStateCommand{
	AlertId: ctx.Rule.Id,
	OrgId: ctx.Rule.OrgId,
	State: ctx.Rule.State,
	Error: exeuctionError,
	EvalData: annotationData,
	}
	if err := bus.Dispatch(cmd); err != nil {
	handler.log.Error("Failed to save state", "error", err)
	}
	// save annotation
	item := annotations.Item{
	OrgId: ctx.Rule.OrgId,
	DashboardId: ctx.Rule.DashboardId,
	PanelId: ctx.Rule.PanelId,
	Type: annotations.AlertType,
	AlertId: ctx.Rule.Id,
	Title: ctx.Rule.Name,
	Text: ctx.GetStateModel().Text,
	NewState: string(ctx.Rule.State),
	PrevState: string(oldState),
	Epoch: time.Now().Unix(),
	Data: annotationData,
	}
	annotationRepo := annotations.GetRepository()
	if err := annotationRepo.Save(&item); err != nil {
	handler.log.Error("Failed to save annotation for new alert state", "error", err)
	}
	handler.notifier.Notify(ctx)
	}

I'm not sure if this is on the roadmap/radar but it would be great to be able to send the notification as long as the alert evaluates to true, regardless of the previous state. For reference, this is a configurable option of Kapacitor:

https://github.com/influxdata/kapacitor/blob/master/etc/kapacitor/kapacitor.conf#L274

[bergquist]

Seems like a reasonable thing to configure.

[pdf]

You would certainly need some form of minimum notification interval before enabling this.

[ghost]

I have another slightly different use case for this. I have multiple series in a graph: https://snapshot.raintank.io/dashboard/snapshot/V6cQW9mzHHfnDJHKgNKbyyCfzp4nZSBy

When the first reaches the alert level, everything goes smooth, but when the second reaches the alert level, it doesn't flip the alert state, and thus doesn't trigger anything. It'd be useful though :)

An idea how to go about this: a per-alert "notification id", which can use variables to distinguish between the desired alerts. In my graph above, I'd set the notification id to {{host}}.

[yannispanousis]

Prrety much same as #7330. +1

[mandeepsandhu]

Is there a status update on this feature? Any upcoming release we can expect it in?

[bergquist]

Closing this since its very simular to #6685

By configuring frequency/min-delay to 0 should all the notifier to execute on every evaluation.

[chaipugao]

@lgierth I hava the same problem with you.
can you detail how to config a per-alert "notification id" what you mentioned above？
thank you

[danielney]

I have a metric that I want to evaluate every 24 hours and if it crosses a threshold generate the alert regardless of whether the alert was generated in the previous evaluation. I am confused about what it means to set frequency to 0. Is the feature to repeat an alert implemented or not? What version?

[kubaseai]

Workaround: generate zero/one periodic signal and multiply original query with this query. You'll get dummy clear.