Could not use kubernetes-pods job

[rvrignaud]

I'm using prometheus 0.19.2 in a Google Container Cluster (GKE) (Kubernetes 1.2.4).
'kubernetes-service-endpoints' discovery is working fine.
I'm trying to use 'kubernetes-pods' job but I don't get any target discovered.

My prometheus job definition is a copy/paste of 'kubernetes-pods' as defined in https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

I have a deployment object defined with:

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Deployment",
  "metadata": {
    "name": "telegraf",
    "namespace": "prd"
  },
  "spec": {
    "replicas": 1,
    "template": {
      "metadata": {
        "labels": {
          "name": "telegraf"
        },
        "annotations": {
          "prometheus.io/scrape": "true",
          "prometheus.io/port": "9126"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "telegraf",
            "image": "xxx",
            "resources": {
              "limits": {
                "memory": "100Mi",
                "cpu": "10m"
              }
            },
            "livenessProbe": {
              "httpGet": {
                "path": "/metrics",
                "port": 9126
              },
              "initialDelaySeconds": 30
            },
            "readinessProbe": {
              "httpGet": {
                "path": "/metrics",
                "port": 9126
              }
            },
            "env": [
             ....
            ]
          }
        ]
      }
    }
  }
}

I have pods scheduled and running. But no target is discovered. I'm not sure how to debug this.
I see nothing in prometheus logs relevant.

Am I doing anything wrong ?

[matthiasr]

I'm not 100% sure but I think you need to declare named ports in your pod spec. At least port 9126 – so just adding "ports": { "name": "telemetry", "containerPort": 9126 } should do.

[rvrignaud]

@matthiasr : thanks that fixed the problem. this may need to be specified in documentation ?
(ports must be an array not a map though)

[matthiasr]

Indeed! I filed prometheus/docs#455 for that.

[pdbogen]

FWIW, the issue here is not the name aspect, but that the port appears to not be declared at all.

Edit: On further consideration, the issue is that no port is declared; thus the pod discovery code declined to create a target for pod container a container that claims to provide no network services. I'm discussing in prometheus/docs#455 how to best address this.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.