kubernetes discovery broken for upgrade to 1.3

[cameronbraid]

What did you do?

Upgrade to 1.3

I'm not sure which version I upgraded from, as I was using "FROM prom/prometheus" rather than a tagged version

What did you expect to see?

A single target in prometheus with endpoint http://10.128.1.5:8081/prometheus

What did you see instead? Under which circumstances?

Two scrape target with both having state = DOWN

The first target has

target address = http://10.128.1.5:80/prometheus
before relabeling __address__="10.128.1.5"
before relabeling __meta_kubernetes_pod_container_name="container-2"
down error = Get http://10.128.1.5:80/prometheus: dial tcp 10.128.1.5:80: getsockopt: connection refused

So it looks to me like prometheus has identified container-2 as a target - which it shouldn't. Nothing is listening on port 80 in the pod.

The second target has

target address = http://10.128.1.5:8081/prometheus
(this looks correct to me, as I would expect given the relabeler configuration)
before relabeling __address__="10.128.1.5:8080" 
before relabeling __meta_kubernetes_pod_container_name="webapp"
down error = server returned HTTP status 500 Server Error

It is actually scraping http://10.128.1.5:8080 as I can see this in my webapp logs rather than http://10.128.1.5:8081 as per the annotations

Environment

openshift origin 1.3

pod has three containers, the first has 4 ports exposed the other two containers don't expose any ports yaml extract :

        metadata:
          labels:
            name: webapp
          annotations:
            prometheus.io/scrape: 'true'
            prometheus.io/port: '8081'
            prometheus.io/path: '/prometheus'
...
        spec:
          containers:
            - name: webapp
              ports:

                - name: http
                  containerPort: 8080

                - name: port-8081
                  containerPort: 8081

                - containerPort: 47500
                  name: ignite-tcp

                - containerPort: 3000
                  name: port-3000

                - name: port-10008
                  containerPort: 10008
                  hostPort: 10008

            - name: container-2
              image: container-2-image

            - name: container-3
              image: container-3-image

• System information:

Linux 3.10.0-327.36.1.el7.x86_64 x86_64

• Prometheus version:

prometheus, version 1.4.1 (branch: master, revision: 2a89e87)
build user: root@e685d23d8809
build date: 20161128-09:59:22
go version: go1.7.3

• Prometheus configuration file:

global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:

  - job_name: 'kubernetes-pods'
    scheme: http
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
      - role: pod

    relabel_configs:

    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true

    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)

    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: (.+):(?:\d+);(\d+)
      replacement: ${1}:${2}
      target_label: __address__

    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)

    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace

    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

[brancz]

Prometheus v1.3.0 revamped the kubernetes integration entirely, unfortunately your relabelling rules don't apply in the same way anymore. By default all ports of a Pod will be scraped and you need to keep/drop those you don't want, for example by introducing a practice where you call the appropriate port name metrics. Have a look at the new kubernetes sample config for a starter on the new config.

[cameronbraid]

thanks for your help, I have everything working again

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.