Kubernetes discovery failing with "Watch pods unexpectedly closed: EOF"

[guoshimin]

What did you do?
Using kubernetes discovery to discover services, nodes and pods.

What did you expect to see?
New pods discovered by prometheus

What did you see instead? Under which circumstances?
New pods not picked up by prometheus. Saw repeated messages like the following in the log:

time="2016-06-08T21:42:36Z" level=error msg="Watch pods unexpectedly closed: EOF" source="discovery.go:885" 
time="2016-06-08T21:43:59Z" level=error msg="Watch service endpoints unexpectedly closed: EOF" source="discovery.go:553" 

Environment

• System information:

Linux 4.3.3-coreos x86_64

• Prometheus version:

  0.19.2

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

insert configuration here (if relevant to the issue)

• Logs:

time="2016-06-08T21:42:36Z" level=error msg="Watch pods unexpectedly closed: EOF" source="discovery.go:885" 
time="2016-06-08T21:43:59Z" level=error msg="Watch service endpoints unexpectedly closed: EOF" source="discovery.go:553" 

[matthiasr]

How are you connecting to the API server? Is Prometheus running inside the cluster? Please paste the relevant bits of the Prometheus configuration you are using.

Another thing to try – if you scale a few pods around after Prometheus has started, do they start showing up? in that case this would be #1702

[guoshimin]

I forgot to mention that it did manage to discover one pod, but not others.

The relevant part of the config: (mostly copied from https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml)

                  - job_name: 'kubernetes-pods'
                    kubernetes_sd_configs:
                    - api_servers:
                      - https://10.3.0.1
                      in_cluster: true
                      tls_config:
                        insecure_skip_verify: true

                    relabel_configs:
                    - source_labels: [__meta_kubernetes_role, __meta_kubernetes_pod_annotation_prometheus_io_scrape]
                      action: keep
                      regex: pod;true
                    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                      action: replace
                      regex: (.+):(?:\\d+);(\\d+)
                      replacement: ${1}:${2}
                      target_label: __address__
                    - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                    - source_labels: [__meta_kubernetes_pod_namespace]
                      action: replace
                      target_label: kubernetes_namespace
                    - source_labels: [__meta_kubernetes_pod_name]
                      action: replace
                      target_label: kubernetes_pod_name

[guoshimin]

OK, I figured out the problem. Prometheus requires containers to declare their ports. In the absence of the port relabeling, it will scrape the lowest port. The error message in the log is just red herring.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.