Grafana 8.x new Alert Rule

[OscarVanL]

What does this do?

This builds on #495 to add a new Grafana 8.x alert rule.

In addition to that MR this does the following:

• Finishes off the AlertRule type by allowing you to have multiple queries and conditions per alert rule, this was blocking the previous MR.
• Reuses the existing Target and AlertCondition types.
• Adds better testing
• Adds example usage

It also includes some extras to make it more feature complete, but these can be separated out into a different PR if you prefer:

• Adds example script for uploading the AlertGroup to grafana.
• Adds console commands for generating alertgroup(s)

Why is it a good idea?

There was no way to produce alerts in new grafana versions using grafanalib.

Context

Grafana 8 added a new unified alerting system.

Questions

Limitations

Currently, there is no way to import/export grafana alerts as JSON via the Grafana user interface.

I have provided an example upload script that does the following:

1. Delete all alerts for the AlertGroup in a folder: DELETE https://<YOUR GRAFANA URL>/api/ruler/grafana/api/v1/rules/<FOLDER>/<ALERT GROUP NAME>

   • The reason for this delete stage is that it's not possible to POST alerts with the same title as an existing alert in that group (the request fails). It's easier to wipe the alerts clean and re-insert them, but I am open to suggestions if there's a better way.

2. Upload all alerts for the AlertGroup into the folder: POST https://<YOUR GRAFANA URL>/api/ruler/grafana/api/v1/rules/<FOLDER> with the generated JSON in the request body.

I recommend testing this in separate grafana deployment to your production environment with all your real alerts! I expect it would be good practice to have an alert group specifically for alerts-as-code so that you don't unintentionally delete alerts that colleagues created in the same alert group via the grafana website.

[lozhn]

I submitted a PR to you, @OscarVanL, to fix the failing flake8 👋

[OscarVanL]

I submitted a PR to you, @OscarVanL, to fix the failing flake8 👋

Thank you! I've merged it.

[OscarVanL]

It looks like I still need to rebase this to fix the conflicts.

[lozhn]

@OscarVanL could you please rebase and we'll check again? Maybe I could help

[OscarVanL]

I've rebased the branch. @JamesGibo it looks like you've reviewed some of the previous MRs, mind taking a look at this please? 😄

[ValentinaBaranova]

Hello, thank you for this job! We would like to use this class in out project too ) Please, take a look if possible.

[JamesGibo]

Thanks for this contribution, sorry i have not reviewed it so far, but as it is quite a big PR I want to fully tests its functionality.
Will try have a look at it today.

[JamesGibo]

We can ignore the doc failure as this is just some links that have moved in grafana

[OscarVanL]

Thanks! We've been using this internally in my company for a couple of months now and it seems to work for us (we wanted a way to template lots of common alert types for many microservices, which this was really useful for), but I also can't say I've tested every feasable use case.

[JamesGibo]

Just had a scan of the code which looks good.
I know you have added an example file, but would you also be able to add a quick example usage of the file to the getting started docs?

[JamesGibo]

@OscarVanL The PR looks good for adding alerts for Grafana v8, would you be able to add some basic documentation of the functionality? Once added I will merge this PR.

When tested I tested using it to generate alerts for Grafana v9, unfortunately the alert schema has change significantly.
I am just looking at how alerting for Grafana v9 could be supported, but it might lead to a breaking change where we have to change the class name to specify the schema required v8/v9, but will add this in a separate PR.

I was also looking at if we could use this code to create the required JSON file for file based Grafana alert provisioning
https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/
Which looks possible, I just need to add a new class to support which I will also do in a separate PR.

{
    "apiVersion": 1,
    "groups": [
    
                <Insert rule groups here>

    ]
}

[OscarVanL]

@JamesGibo I've added some documentation to the getting started page.

That's a shame to hear about the schema change in Grafana v9. I've not had a chance to try out Grafana v9 yet.

I've also been quite interested in Grafana provisioning, it would be cool if our dashboards could be generated as code and then provisioned by CI in a way that's well supported by Grafnaa.

[JamesGibo]

Thanks, for adding some documentation of the feature.
I will merger the PR now and will make a release once I have finished my work on alerts for Grafana v9.

File based provisioning for dashboards is quite stable and I have been using that for a while, but the alert provisioning has just been released and have already found a few bugs in it and its not very well documented.