Change in JSON payload in latest version of Grafana

[KiqoCode]

In new version of Grafana they have changed JSON payload format, therefor not compatible with grafana webhook in Alerta.
https://grafana.com/docs/grafana/latest/alerting/unified-alerting/contact-points/

[jjdmol]

You can use Grafana's 'Alertmanager' contact point type instead though to send to Alerta's 'Prometheus' webhook (f.e. http://alerta-server:8080/api/webhooks/prometheus?api-key=demo-key) , works like a charm!

[sushovan23]

So in this case we would need to run alertmanager ? or just use alertmanager and just webhook of alerta for prometheus instead of alertmanager url?

[jjdmol]

The latter. You don't need to run an actual Prometheus Alertmanager for this. "AlertManager" in this case describes the protocol for communicating the alarm messages, not the software.

Grafana is capable of emitting its alarms in the AlertManager format, and Alerta is capable of receiving alarms in the AlertManager format. It's just that the webhook in Alerta is called "prometheus", and we have Grafana mimicking Prometheus here, which makes things a bit confusing.

[sushovan23]

I tested out in grafana when I mention alertmanager as notification and try to put the alerta prometheus webhook it doesnt work even test alert fails
However in grafana if I mention webhook and put alerta/prometheus webhook it atleast works but not sure if grafana emits in alertmanager format then.

[mabed-fr]

Hi,
I confirm that with my grafana in version 10.4.1 the api target /api/webhooks/grafana?api-key=xxx&environment=Production&service=Web&timeout=3600

does not work but with the alertmanager / prometheus target it works almost 100%

Is there a fix planned?

[mabed-fr]

v9.0.3 not resolve problem

Payload(json) receive in logs

{
  "receiver": "test",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "TestAlert",
        "instance": "Grafana"
      },
      "annotations": {
        "summary": "Notification test"
      },
      "startsAt": "2024-04-12T19:53:04.088799693+02:00",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "",
      "fingerprint": "57c6d9296de2ad39",
      "silenceURL": "https://dash.exemple.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTestAlert&matcher=instance%3DGrafana",
      "dashboardURL": "",
      "panelURL": "",
      "values": null,
      "valueString": "[ metric='foo' labels={instance=bar} value=10 ]"
    }
  ],
  "groupLabels": {
    "alertname": "TestAlert",
    "instance": "Grafana"
  },
  "commonLabels": {
    "alertname": "TestAlert",
    "instance": "Grafana"
  },
  "commonAnnotations": {
    "summary": "Notification test"
  },
  "externalURL": "https://dash.exemple.com/",
  "version": "1",
  "groupKey": "test-57c6d9296de2ad39-1712944384",
  "truncatedAlerts": 0,
  "orgId": 1,
  "title": "[FIRING:1] TestAlert Grafana ",
  "state": "alerting",
  "message": "**Firing**\n\nValue: [no value]\nLabels:\n - alertname = TestAlert\n - instance = Grafana\nAnnotations:\n - summary = Notification test\nSilence: https://dash.exemple.com/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DTestAlert&matcher=instance%3DGrafana\n"
}

[hyberdk]

yes, there have been multiple changes to the grafana alerting (unfortunately), it seems like it has "somewhat" stabilized.. pull requests are welcome to fix the problem ;-)