Configuring Mimir Ruler with Grafana

[spyder007]

I have been running into some issues trying to manage Alerts in Mimir using Grafana.

Overall, my configuration is pretty simple: I'm using the mimir-distributed Helm chart (as a subchart) with the following values:

global:
  extraEnvFrom:
    - secretRef:
        name: mimir-minio-secret
minio: 
  enabled: false
alertmanager:
  enabled: true
  persistentVolume:
    storageClass: nfs-client
ingester:
  persistentVolume:
    storageClass: nfs-client
store_gateway:
  persistentVolume:
    storageClass: nfs-client
compactor:
  persistentVolume:
    storageClass: nfs-client
ruler:
  enabled: true
nginx:
  ingress:
    enabled: true
    hosts:
      - host: "https://my.host.net"
        paths:
          - path: /
            pathType: Prefix
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/service-upstream: "true"
    tls: []
mimir:
  structuredConfig:
    alertmanager_storage:
      backend: s3
      s3:
        access_key_id: ${MINIO_USERNAME}
        bucket_name: mimir-ruler
        endpoint: minio.local:9000
        insecure: true
        secret_access_key: ${MINIO_PASSWORD}
    blocks_storage:
      s3:
        access_key_id: ${MINIO_USERNAME}
        bucket_name: mimir-tsdb
        endpoint: minio.local:9000
        insecure: true
        secret_access_key: ${MINIO_PASSWORD}
    ruler_storage:
      backend: s3
      s3:
        access_key_id: ${MINIO_USERNAME}
        bucket_name: mimir-ruler
        endpoint: minio.local:9000
        insecure: true
        secret_access_key: ${MINIO_PASSWORD}
    limits:
      max_global_series_per_user: 0
      ingestion_rate: 40000
      ingestion_burst_size: 100000
      out_of_order_time_window: 
metaMonitoring:
  serviceMonitor:
    enabled: true
    interval: 1m
    scrapeTimeout: 30s

With this configuration, Mimir works fine, and it's a remote-write target for my other Prometheus instances (one in each cluster).

Within Grafana, I have configured a Prometheus datasource using this url: https://my.host.net/prometheus

Again, I can get data out of Mimir and my dashboards seem to work fine. However, when I go to manage alerts in Grafana (using Alerting -> + new alert rule, I cannot choose my Mimir data source for a Mimir or Loki alert. My Loki data source shows, but not Mimir.

The ruler components seems to be running correctly: calls to https://my.host.net/prometheus/api/v1/rules return success with no data (since I don't have groups setup yet)

{
"status":"success",
"data":{
  "groups":[]
},
"errorType":"",
"error":""}

and https://my.host.net/ruler/ring returns an active instance with a good heartbeat.

I had originally thought that I needed to change the prometheus_http_prefix, as suggested by This page, but, according to @dimitarvdimitrov, that documentation is out of date and the default should work.

I know, with the alert manager, I had to pre-configure my anonymous tenant in order to add the Alert Manager instance as a Grafana data source. Is this a case where i need to pre-configure the ruler for the anonymous tenant as well?

[pracucci]

Again, I can get data out of Mimir and my dashboards seem to work fine. However, when I go to manage alerts in Grafana (using Alerting -> + new alert rule, I cannot choose my Mimir data source for a Mimir or Loki alert. My Loki data source shows, but not Mimir.

Which Grafana version are you running? Can you try with the latest stable, if you're not running it yet? There were some issues in "old" Grafana versions.

[spyder007]

@dimitarvdimitrov That worked, but not in the way that you might think....

Taking your suggestion, I added the header in both spots (nginx.nginxConfig.serverSnippet and nginx.nginxConfig.httpSnippet). This successfully adds the header when making a GET call to /api/v1/status/buildinfo.

It looks like this get, though, is being proxied. Looking at my network log in Chrome, I see this:

GET https://grafana.mydomain.local/api/datasources/proxy/14/api/v1/status/buildinfo -> Returns a 301 redirect to https://mimir.mydomain.local/api/v1/status/buildinfo
GET https://mimir.mydomain.local/api/v1/status/buildinfo triggers a preflight call ->
OPTIONS https://mimir.mydomain.local/api/v1/status/buildinfo returns a 405 method not allowed

The error in chrome specifically mentions that the "Response to preflight request doesn't pass access control check," so I'm assuming that the 405 error when making the OPTIONS call is what's causing the issue.

I did a quick check within the cluster (stood up a test pod to try making calls directly to the nodes). Based on the nginx config file, that buildinfo path is forwarded to the mimir-query-frontend. When I hit that service directly (bypassing all NGINX routing), I get a 405 on the OPTIONS method.

It would seem that Chrome is treating that redirect as a "non-simple" call and doing all the necessary CORS checks, but since it cannot pre-flight that call to buildinfo, it's failing. But, applying Occam's Razor, I thought "Someone else has to have seen this if it's really a problem." So I examined further the network log, particularly the redirect. The redirect says it's cached. Where it built that cache from, well, I have NO idea.

So I disabled the cache in network tools, refreshed the page, and viola, it worked. Even weirder, looking specifically at the network log, there was not one call from the browser directly to mimir.mydomain.local: all the calls were proxied through grafana.mydomain.local. In looking at that, I wondered if I even needed those Access-Control-Allow-Origin headers, since it looks like the original problem was a Chrome caching issue.

So I removed those headers from both snippets sections, redeployed, and, well, everything works fine. It would seem that, at some point in time, Chrome got a 301 response from https://grafana.mydomain.local/api/datasources/proxy/14/api/v1/status/buildinfo, cached it, and decided it was never going to let go of that cache. Once I disabled the response cache, everything worked correctly.

I appreciate your help in this and, well, essentially being a rubber duck for me while I figured out it was my browser cache and nothing more.

[dimitarvdimitrov]

glad you resolved this. Thank you so much for writing it up in this level of detail!

I also forgot that grafana proxies requests through itself by default and doesn't use the user's browser.

[spyder007]

@dimitarvdimitrov are you able to mark this as answered?

[dimitarvdimitrov]

do you know why grafana returned a 301 in the first place - maybe you were using the browser access mode for the datasource instead of server?

[spyder007]

Honestly, I have no idea. I believe, at one point in time, I was getting other errors with accessing Mimir (and, specifically, the multi-tenant alert manager not being configured for anonymous). So, I probably clicked around a bunch, and, at some point, Chrome picked up that 301 and held on to it for dear life :)