mimir grafana dashboards with mixin

[ahsan-n]

Hi So we are using helm chart to deploy mimir-distributed, with the help of awesome contributors from slack channel I come to know that we have to first publish rules in order to make dashboards work properly. But getting stuck in publishing the right rules and metrics

enabled serviceMonitor:

serviceMonitor:
  enabled: true
  namespace: prometheus
  namespaceSelector:
    matchNames:
      - mimir

which creates service monitor crd like this:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2022-06-23T15:53:03Z"
  generation: 1
  labels:
    app.kubernetes.io/component: ingester
    app.kubernetes.io/instance: mimir
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mimir
    app.kubernetes.io/part-of: memberlist
    app.kubernetes.io/version: 2.1.0
    argocd.argoproj.io/instance: mimir
    helm.sh/chart: mimir-distributed-2.2.0-weekly.191
  name: mimir-ingester
  namespace: mimir
  resourceVersion: "5446326"
  uid: c43f70f6-eb89-4aa9-b950-199b6777fbf6
spec:
  endpoints:
  - port: http-metrics
    relabelings:
    - action: replace
      replacement: mimir/ingester
      sourceLabels:
      - job
      targetLabel: job
    - action: replace
      replacement: mimir
      targetLabel: cluster
    scheme: http
  namespaceSelector:
    matchNames:
    - mimir
  selector:
    matchExpressions:
    - key: prometheus.io/service-monitor
      operator: NotIn
      values:
      - "false"
    matchLabels:
      app.kubernetes.io/component: ingester
      app.kubernetes.io/instance: mimir
      app.kubernetes.io/name: mimir

By looking at this I think this is already handling the requirement of adding labels but I am unable to search any metric relvent to mimir in grafana.

Second challenge is I am not sure what to add in operations/mimir-mixin/config.libsonnet as per above config do I leave it as it is or have to do some changes:

    job_names: {
      ingester: '(ingester.*|cortex|mimir)',  // Match also custom and per-zone ingester deployments.
      distributor: '(distributor|cortex|mimir)',
      querier: '(querier.*|cortex|mimir)',  // Match also custom querier deployments.
      ruler_querier: '(ruler-querier.*)',  // Match also custom querier deployments.
      ruler: '(ruler|cortex|mimir)',
      query_frontend: '(query-frontend.*|cortex|mimir)',  // Match also custom query-frontend deployments.
      ruler_query_frontend: '(ruler-query-frontend.*)',  // Match also custom ruler-query-frontend deployments.
      query_scheduler: 'query-scheduler.*',  // Not part of single-binary. Match also custom query-scheduler deployments.
      ruler_query_scheduler: 'ruler-query-scheduler.*',  // Not part of single-binary. Match also custom query-scheduler deployments.
      ring_members: ['alertmanager', 'compactor', 'distributor', 'ingester.*', 'querier.*', 'ruler', 'ruler-querier.*', 'store-gateway.*', 'cortex', 'mimir'],
      store_gateway: '(store-gateway.*|cortex|mimir)',  // Match also per-zone store-gateway deployments.
      gateway: '(gateway|cortex-gw|cortex-gw-internal)',
      compactor: 'compactor.*|cortex|mimir',  // Match also custom compactor deployments.
      alertmanager: 'alertmanager|cortex|mimir',
      overrides_exporter: 'overrides-exporter',
    },

    // The label used to differentiate between different Kubernetes clusters.
    per_cluster_label: 'cluster',

    // Grouping labels, to uniquely identify and group by {jobs, clusters}
    job_labels: [$._config.per_cluster_label, 'namespace', 'job'],
    cluster_labels: [$._config.per_cluster_label, 'namespace'],

If not mistaking here I think I have to replace cluster with mimir ? not sure how and what to replace job with? In order to publish the right rules and metrics? as for I am not getting any metrics .

Thanks.

[dhbrojas]

Hi, unexperienced Mimir user here but I recently deployed the dashboards and monitored Mimir using CRDs just like you a few days ago. I don't think you have to make any changes to the config because for me the dashboards worked out of the box with the default config. Just to be clear, when you say:

If not mistaking here I think I have to replace cluster with mimir

Do you mean doing this?

-    per_cluster_label: 'cluster',
+    per_cluster_label: 'mimir',

If that's the case, no you don't need to change the value here. Have you tried uploading the rules to see if it works as is?

To check if your ruler is working properly you can check how many rules were recently evaluated using the cortex_prometheus_rule_evaluations_total metric.

[dimitarvdimitrov]

The monitoring mixin should work almost out of the box with the default service monitors from the chart.

per_cluster_label is the name of a label on mimir metrics. If your prometheus operator deployemnt is already adding the cluster label (with the value of the kubernetes cluster for example), then you don't need to do anything. If you have a similar label that contains similar info, then change per_cluster_label to that label (e.g. per_cluster_label: 'kubernetes_cluster'). If you don't have such a label, it is best if you change relabelling config in the prometheus to add a cluster label.

If you aren't seeing metrics, I would check the config of the prometheus from the prometheus operator. Check that it is indeed discovering the ServiceMonitros as targets.

In order to see all dashbaords populated, you need to install the recording rules yourself. The helm chart does not yet deploy any PrometheusRule CRD (this may change with #2134 ). The monitoring mixin renders dashbaords as well as a yaml file with alerts and recording rules. You can install the recording rules in the datasource which contains the mimir metrics.

[pracucci]

per_cluster_label is the name of a label on mimir metrics

You can find more information about all required labels here (and how to customize them) here:
https://grafana.com/docs/mimir/next/operators-guide/monitoring-grafana-mimir/requirements/

[ahsan-n]

Hi thanks for the tips, So Initially I was not adding mimir in serviceMonitorSelector

prometheusOperator:
  prometheusSpec:
    serviceMonitorSelector:
      matchLabels:
        app.kubernetes.io/instance: mimir

Once I added I am getting metrics now. But problem is I am not getting metrics which come from rules.

I published rules without any change in mixing config with this command:

mimirtool rules load ./operations/mimir-mixin-compiled/rules.yaml --id=fake --address http://127.0.0.1:9090

rule metric example which I am not getting in grafana:

cluster_namespace_job:cortex_distributor_received_samples:rate5m{}

[dhbrojas]

Weird! Looks like rules are being evaluated but not written 🤔

[krajorama]

Hi,

unfortunately we haven't got around to installing rules&alerts by default, see: grafana/helm-charts#1240

We did move the helm chart source code (from grafana/helm-charts) next to the mimir and jsonnet source (grafana/mimir) recently, giving us a way to generate the required rules relatively easily with some automation.

regards, krajo

[dhbrojas]

What happens when you run this query in Grafana?

sum by (cluster, namespace, job) (rate(cortex_distributor_received_samples_total[5m]))

Do you get any results back? Make sure you're running the query as the tenant fake.

[ahsan-n]

Thanks @rojas-diego for slack chat setting --id=anonymous fix the issue

[ahsan-n]

thanks for the amazing support and help closing this discussion