Cannot get other information targets scrapped except kubernetes-nodes (kubernetes_sd_config)

[sudhirpandey]

What did you do?
Upgrade the openshift origin to 1.3 and prometheus to 1.3.1
What did you expect to see?
different job-names targets should have been visible under status > targets
What did you see instead? Under which circumstances?

Environment
promethes v1.3.1

oc v1.3.0
kubernetes v1.3.0+52492b4

openshift v1.3.0

kubernetes v1.3.0+52492b4

• System information:
  uname -srm

Linux 3.10.0-327.36.3.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 1.3.1 (branch: master, revision: be476954e80349cb7ec3ba6a3247cd712189dfcb)
 build user:       root@37f0aa346b26
 build date:       20161104-20:24:03
 go version:       go1.7.3

• Prometheus configuration file:

scrape_configs:
- job_name: 'kubernetes-apiservers'
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    #insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

- job_name: 'kubernetes-nodes'
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    # insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

  kubernetes_sd_configs:
  - role: node

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints

  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: (.+)(?::\d+);(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_service_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

- job_name: 'kubernetes-services'

  metrics_path: /probe
  params:
    module: [http_2xx]

  kubernetes_sd_configs:
  - role: service

  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
    action: keep
    regex: true
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: blackbox
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_service_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name

- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod

  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

With the configuration in place i expect to see services and endpoints.. but also the cluster api as well ..but in web UI of prometheus i just get results from job name kubernetes-nodes

I have used promethes 1.2.1 with origin 1.2.0 and i was able to get api-endpoints metrics as well. Is there any debug level info i can look at to figure out what really going on.

[sudhirpandey]

i realised that i was missing kubernetes_sd_confius on job_name kubernetes-apiservers , so i added this under the job, which resulted the api end servers to be found.

job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints

But i still cannot the other service, endpoints and pod roles to work. I amd hoping they will also show up as different targets via their job name in prometheus UI.

[brancz]

The config looks ok, do the logs say anything? Also make sure your Service/Endpoints objects have the prometheus.io/scrape: true annotation?

[sudhirpandey]

Thanks .
I see now correct annotations need to be applied to the services etc to show up as targets by applying the the annonations. It was bit unclear for me looking into docs though.

Also seems to need to be running the blackbox exporter pod to get the probe part to work.

i applied this annonations in the service

prometheus.io/probe: "true"

and have following yaml to take care of the blackbox service to do probing

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Service
  metadata:
    name: blackbox
    labels:
      name: blackbox
  spec:
    ports:
      - port: 80
        targetPort: blackbox
        protocol: TCP
    selector:
      app: blackbox-exporter

- apiVersion: v1
  kind: ReplicationController
  metadata:
    name: blackbox-exporter
  spec:
    replicas: 1
    selector:
      app: blackbox-exporter
    template:
      metadata:
        name: blackbox-exporter
        labels:
          app: blackbox-exporter
      spec:
        containers:
        - image: quay.io/prometheus/blackbox-exporter:v0.3.0
          name: blackbox
          ports:
          - containerPort: 9115
            name: blackbox

And was able to get the desired service end point to be monitored.

[brancz]

We have and are compiling Kubernetes + Prometheus resources at kube-prometheus. The problem with the example config is that it is a bit limit and opinionated. But I understand that some more explanation would have helped. I'll come up with something to make it easier for the next to come. Thanks for the feedback!

[sudhirpandey]

Thanks the resources indeed seems to be helpful. Thanks for the help :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.