Setting up Grafana to work with Mimir-Alertmanager

[kotchaosu]

Currently comparing Cortex setup and feature set with Mimir (both installed via official helm charts):

• Grafana OSS 9.0.0-beta1
• Cortex 1.9.0
• Mimir 2.0.0

For Mimir I keep the setup minimal:

• 2 ingesters (retention to 24h and replication factor to 2)
• 1 distributor
• 1 ruler
• 1 alertmanager
• 1 querier and 1 query-frontend
• 1 storage-gateway
• 1 compactor (blocks retention to 7 days)
• 1 nginx

I enabled alpha plugins in Grafana to get the possibility to configure datasource of Alertmanager type.

Problem I: I cannot add successfully Mimir-Alertmanager as datasource

When trying to save the datasource (URL http://mimir-mimir-poc-nginx) I'm getting Health check failed. error in Grafana UI:

and in nginx logs I see:

10.233.86.193 - - [25/May/2022:16:59:57 +0000]  404 "GET /api/v2/status HTTP/1.1" 35 "-" "Grafana/8.5.0" "10.10.10.126, 10.233.86.221, 10.233.86.221"
10.233.86.193 - - [25/May/2022:16:59:57 +0000]  404 "GET /alertmanager/api/v2/status HTTP/1.1" 35 "-" "Grafana/8.5.0" "10.10.10.126, 10.233.86.221, 10.233.86.221"

For Cortex first endpoint fails with 404 as well, but the second one succeeds. In Mimir documentation in section listing differences in API I can't find this particular status endpoint.

Problem II: cannot send alerts

Created BogusCritical alert of Mimir/Cortex/Loki type (evaluated and firing):

And datasource for Prometheus Alertmanager (passing healthcheck):

In Grafana I added Slack receiver. Grafana reads Alertmanager configuration without problems (all applied AM changes are visible in the Grafana). However, I cannot get the notifications for my custom alerting rule.

I see the following error entries in Grafana logs, so I assume Grafana tries to send alerts to the Mimir Alertmanager (even it it's not explicitly created as a datasource) anyway:

logger=sender level=error alertmanager=http://mimir-mimir-poc-alertmanager:8080/api/v2/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"

Has anyone encountered similar issue? Where else should I check to investigate these problems?

[kotchaosu]

Ok, I was finally able to do it right. Missing parts are:

1. Fallback config for Mimir Aletmanager (example below).
2. Mimir Alertmanager must be set as Prometheus implementation in the Alertmanager (type) datasource configuration (which means its configuration can't be modified from Grafana UI as of now).

fallback.yml:

global:
  resolve_timeout: 5m
route:
  receiver: default
  group_by:
    - alertname
    - instance
  group_wait: 1m
  group_interval: 5m
receivers:
  - name: default
templates: []

[pracucci]

Mimir Alertmanager must be set as Prometheus implementation in the Alertmanager (type) datasource configuration (which means its configuration can't be modified from Grafana UI as of now).

Grafana supports it but before 9.0.0 it was hidden behind a feature flag. I've just opened the PR #2115 to update our tutorial and get Grafana Alerting UI working with Mimir Alertmanager too.

What I suggest you to do:

1. Upgrade Grafana to 9.0.0 (or above, when will be released)
2. Add a "Alertmanager" datasource:
   • Select "Cortex" implementation (we'll soon add "Mimir" too)
   • The URL should have no /alertmanager suffix
3. Edit the "Prometheus" datasource and select the "Alertmanager" datasource you just created in the related dropdown menu

Does it solve your issue?

[kotchaosu]

Works like charm! I'm able to configure notification receiver and templates from Grafana UI. Alerts are being sent properly. Thanks a lot for helping out!