[Q] Autodiscovery of roadrunner pods with prometheus

[victor-sudakov]

Dear Colleagues,

If you use Prometheus to autodiscover and monitor RoadRunner pods, can you please share you config? I've configured annotations on the RoadRunner pods:

annotations:
  prometheus.io/scrape: "true"

and put the following into prometheus.yml:

- job_name: "kubernetes-pods"
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_scrape ]
      action: keep
      regex: true

I can view the metrics statically by running wget -O- roadrunner.dev.svc.cluster.local:9090/metrics from the prometheus pod (where "roadrunner" is a Service behind which the roadrunner pods are), but no discovery seems to be happening.

I'd be very grateful for a good example.

And yes, the service account the prometheus pod is running as seems to have all the necessary permissions to list, get and watch pods clusterwide.

[victor-sudakov]

The seemingly working config snippet is below.

scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

It relies on the roadrunner pods being annotated with

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9090"