Kubernetes service discovery can not be detect

[yinwenqin]

I want to use the prometheus SD to discovery the kube service status,so i found a template of kube service target job,then i add it to configmap. but after reload prometheus,i can't event found the title of the job in prometheus web page ,what's wrong i make ? the other jobs are normal running,except the 'kube service'

My kube version is v1.9,here is the target-job template:

- job_name: 'kubernetes-services'
  kubernetes_sd_configs:
  - role: service
  metrics_path: /probe
  params:
    module: [http_2xx]
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: true
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: blackbox-exporter.example.com:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name

On the prometheu web ui , targets page, i can't find the job name kubernetes-services in Targets list which i just add into the configmap, but the other jobs are normal .

Here is the jobs in targets page view:
"""
Targets
federate (0/1 up)
kubernetes-apiservers (1/1 up)
kubernetes-cadvisor (3/3 up)
kubernetes-node-exporter (3/3 up)
kubernetes-nodes (3/3 up)
kubernetes-service-endpoints (1/5 up)
mysql performance (1/1 up)
“”“
somebody can help ? If can't even show it,there is no way to customize the rules of kube service

[brancz]

Did you reload or redeploy the Prometheus server after changing the configmap?

[yinwenqin]

@brancz Yes i did,but the new target did not appear,i have try change the prome version,but it still didn't work

[brancz]

Can you see your configuration snippet in the /config page of the Prometheus web UI?

[yinwenqin]

@brancz Yes,it's already in prome web /config page , but not in /targets page,so strange that

- job_name: kubernetes-services
  params:
    module:
    - http_2xx
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /probe
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: service
    namespaces:
      names: []
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: blackbox-exporter.example.com:9115
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: __meta_kubernetes_service_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: kubernetes_namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: kubernetes_name
    replacement: $1
    action: replace

[brancz]

Do you actually have a Service that has the prometheus.io/probe annotation?

[yinwenqin]

@brancz Oh i get it ,the reason for this is that the label doesn't match. After i add prometheus.io/probe: true into a svc metadata.annotations,then the kubernetes-services job appears on the /targets page,thank you so much!

By the way, i want to get metrics form kube_svc_name:9104,then i try to modify the job replacement field :replacement: __meta_kubernetes_service_name:9104. But it doesn't convert automatically so it get an error ,how should I modify the configuration file?

[yinwenqin]

I solved it,below is the job configuration file config,as thus, i can get all of the kube_svc with prometheus.io/probe: true annotation and spec.ports.name: mysql-exporter label

    - job_name: 'kubernetes-services'
      kubernetes_sd_configs:
      - role: service
      metrics_path: /metrics
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_port_name]
        action: keep
        regex: "mysql-exporter"
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name

@brancz Thank you so much,best wish for you!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.