Offer the option to pause all alert rules using a specific datasource

[Jonnymcc]

Feature Request: We would like to be able to pause/unpause all alerts related to a specific datasource. That way we can prevent alert execution for a datasource while performing maintenance on that data source without impacting other alerts that are unrelated to it.

I wouldn't mind working on a PR for this feature but I'm not sure where to start. What backend source files and what static resources are relevant to making a such a change?

The design I have in mind is when viewing a list of datasources, each datasource would have a button that could be clicked on to pause all related alerts. The button would then change to an unpause button that can be clicked again to unpause all related alerts. Alternatively this button could appear in the datasource edit view... but the list view should still reflect whether or not alerts have been paused for a given datasource. Also, in the alert list view it should probably show in some way whether or not the alert was paused as a result of pausing all alerts for a datasource, which would indicate to the user that those (possibly many) paused alerts can be unpaused by clicking the unpause alerts button from the datasources list or edit view.

[bergquist]

Not sure about the UX for this feature.

Perhaps a first step is to create the API so people can pause alert for datasources from scripts.

[Jonnymcc]

@bergquist I saw a comment here https://github.com/grafana/grafana/pull/6996/files#r93044269 saying that pausing per datasource was once supported? How was it done? I noticed that the schema for alerts does not have a link to related metric query datasources. How can that relationship be identified? By inspecting the alert.settings field?

[bergquist]

It would require reading all alerts, parsing the model json and see if data source in the query model matches the data source you want to pause. The data source cannot be a part of the table schema since one alert can use multiple data sources. Unless we decide to create a relation table between alert and data source.

[ddhirajkumar]

We have the same need. But I realized we could accomplish this by scripting it using the REST API (as a workaround):-

• Get All Alerts and filter those which are associated to the datasource which is being maintained
• Pause each of those alerts

[orojina]

@Jonnymcc @ddhirajkumar In case you find this useful, and as a temporary solution, I created a script to list, pause/un-pause alerts with their statuses and the dashboard they belong to. Listing several alerts can be done with:

grafana-cli alerts | grep -i <your_filter_cliteria>

You can then use something like this to pause your alerts (assuming your alerts or dashboard titles contain the data source name)

~/grafana-cli.sh alerts | grep '<data_source>' | sort | awk '{print $1}' | grep '^[0-9]' | xargs | sed 's/ /,/g' | xargs ~/grafana-cli.sh alerts pause

https://github.com/orojina/grafana-cli

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!