Custom metric path does not work

[dbluxo]

What did you do?

I've configured a custom service inside kubernetes, with the following annotations:

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/path: '/custom'
    prometheus.io/port:  '8080'

What did you expect to see?

I've configured the following relabel operation inside prometheus.yml, as of:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)

I'm expecting __metrics_path__ will be replaced with /custom

What did you see instead? Under which circumstances?

__metrics_path__ is still /metrics (default)

Environment

We are using prometheus 2.2.1 inside kubernetes.

• System information:

  Linux 4.9.75-rancher x86_64

• Prometheus version:

  prometheus, version 2.2.1 (branch: HEAD, revision: bc6058c)
  build user: root@149e5b3f0829
  build date: 20180314-14:15:45
  go version: go1.10

[brian-brazil]

Does the service discovery status page show that __meta_kubernetes_service_annotation_prometheus_io_path is set as you expect?

[dbluxo]

Yes, as of service discovery status page io_path is set to /custom:

__meta_kubernetes_service_annotation_prometheus_io_path="/custom"

[brian-brazil]

Can you share the full relabel configs and the relevant row on the discovery status page?

[dbluxo]

Yes of course:

[simonpasquier]

I can't reproduce it with v2.2.1, there may be an error with your configuration.
Can you share the targets page too?

[dbluxo]

Here is the job config (prometheus.yml):

    - job_name: 'kubernetes-endpoints'
      scheme: http
      kubernetes_sd_configs:
      - role: endpoints
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_service

Here are the targets for this job (kubernetes-endpoints):

Last target is down, because the metrics are under "/prometheus" not under "/metrics".

[simonpasquier]

you've got a typo: the label name should be __metrics_path__ (you're missing trailing __).

[dbluxo]

Omg, thank you!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.