[READY] adding alerts channels to the dashboards

[ya7ya]

This PR is for coupling alerts with charts and have these alerts trigger in alertmanager (and subsequently pagerDuty)

why?

• more visiblity on the dashboards, (the chart visually changes when an alert is triggering too)
• quickly compare other stats using the explore function

[iameli]

Awesome, I didn't really get how AlertManager and Grafana played together until now.

I want to make sure that the CPU and RAM alarms will trigger PagerDuty alerts. The previous things for that are here. Can we get the CPU/RAM alerts added there? Alternately, we could instead trigger PagerDuty alerts from the Grafana side if you think there's a way to do that.

[ya7ya]

@iameli that link you mentioned in the previous comment just leads me to the changed files, I'm not sure I follow .

If you mean why not just add it as a prometheus alert instead of grafana alert. if thats your point I can do that

[iameli]

Oh oops, I meant to link here. Those are the alerts that are currently capable of triggering PagerDuty.

If you mean why not just add it as a prometheus alert instead of grafana alert. if thats your point I can do that

I don't have any preference between Prometheus alerts and Grafana alerts really but any alerts should trigger our PagerDuty response and whatnot. So right now that means Prometheus, but I'd be also fine with figuring out how to integrate PagerDuty alerts from Grafana.

[ya7ya]

any alerts should trigger our PagerDuty response and whatnot. So right now that means Prometheus,

Yes , prometheus alertmanager is integrated with pagerDuty
the new grafana alerts is integrated by default with the alertmanager , so alerts would still trigger the pagerDuty because the alertmanager will forward these alerts.

The reason why use grafana alerts is because it makes it easier to look at the alert history on a chart and correlate that with other metrics, the alertmanager doesn't have that UX

I've added a direct pagerDuty integration too that we can activate that will allow grafana -> pagerDuty alerts without needing the alertmanager too. if thats more preferred I can activate it

[ya7ya]

@iameli is this good to go? let me know if you have any changes to add here before we merge this.

[iameli]

Yep, LGTM!