You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary:
I ran into a situation recently where the Pagerduty-Grafana integration was failing to send 'alert resolved' notifications to a Pagerduty contact point after the alert resolved. The problem was due to a golang template for the 'summary' section of the contact point being wrapped in an {{ if .Alerts.Firing }} tag (which of course would result in an empty summary upon resolution of an alert). Pagerduty was responding with a 400 every time this empty summary came through, resulting in a failure to auto-resolve a page.
I was asked by @yuri-tceretian to generate an issue so ya'll could discuss the addition of possible "protect the user from their own misconfiguration" dummy text upon empty summary.
My own thoughts:
The addition of golang templating to grafana alerts in grafana 9 looks really powerful, but it's still pretty complex and not entirely transparent to the userbase yet... I'm not sure I (as a user) would appreciate the addition of an extra 'sane default' in an ecosystem where there's already a lot that can go wrong and not quite as much transparency on the alert compilation process...
... but you guys are the experts, go get 'em. :)
The text was updated successfully, but these errors were encountered:
Slack thread: https://grafana.slack.com/archives/C043CEX9MBK/p1689199648823409
Summary:
I ran into a situation recently where the Pagerduty-Grafana integration was failing to send 'alert resolved' notifications to a Pagerduty contact point after the alert resolved. The problem was due to a golang template for the 'summary' section of the contact point being wrapped in an {{ if .Alerts.Firing }} tag (which of course would result in an empty summary upon resolution of an alert). Pagerduty was responding with a 400 every time this empty summary came through, resulting in a failure to auto-resolve a page.
I was asked by @yuri-tceretian to generate an issue so ya'll could discuss the addition of possible "protect the user from their own misconfiguration" dummy text upon empty summary.
My own thoughts:
The addition of golang templating to grafana alerts in grafana 9 looks really powerful, but it's still pretty complex and not entirely transparent to the userbase yet... I'm not sure I (as a user) would appreciate the addition of an extra 'sane default' in an ecosystem where there's already a lot that can go wrong and not quite as much transparency on the alert compilation process...
... but you guys are the experts, go get 'em. :)
The text was updated successfully, but these errors were encountered: