kubernetes prometheus.io/path is not changing metric path in service endpoint.

[vishksaj]

What did you do?

Annotations configured for my service

  "prometheus.io/path": "/exporter",
  "prometheus.io/port": "9913",
  "prometheus.io/scrape": "true"

What did you expect to see?

As per relabel config the service endpoint should be /exporter

- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metric_path]
  action: replace
  target_label: __metrics_path__
  regex: (.+)

What did you see instead?

metrics_path is still /metrics (default)

Environment
Prometheus 2.4.2
c305ffa inside kubernetes.

I can see my cutom path in __meta_kubernetes_service_annotation_prometheus_io_path

Prometheus.yml

#my global config
global:
external_labels:
monitor: k8s-incluster

scrape_configs:

• job_name: 'kubernetes-service-endpoints'
  scrape_interval: 30s
  scrape_timeout: 30s
  tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  • role: endpoints
    relabel_configs:
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_metric_path]
    action: replace
    target_label: metrics_path
    regex: (.+)
  • source_labels: [address, __meta_kubernetes_service_annotation_prometheus_io_scrape_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: address
  • source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape_scheme]
    action: replace
    target_label: scheme
    regex: (https?)
  • action: labelmap
    regex: _meta_kubernetes_service_label(.+)
  • source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  • source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
    metric_relabel_configs:
  • source_labels: [container]
    replacement: '$1'
    action: replace
    target_label: container_name
  • source_labels: [pod]
    replacement: '$1'
    action: replace
    target_label: pod_name
  • source_labels: [namespace]
    regex: '([^-]+)-([^-]+)-(.*)'
    replacement: '$1'
    target_label: app
  • source_labels: [namespace]
    regex: '([^-]+)-([^-]+)-(.*)'
    replacement: '$2'
    target_label: app_env
  • source_labels: [namespace]
    regex: '([^-]+)-([^-]+)-(.*)'
    replacement: '$3'
    target_label: cluster_id

[gouthamve]

Hi, it should be __meta_kubernetes_service_annotation_prometheus_io_path not __meta_kubernetes_service_annotation_prometheus_io_metric_path.

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[vishksaj]

Thanks @gouthamve