Scrape multiple containers in a kubernets pod

[JorritSalverda]

With the prometheus.io/scrape and prometheus.io/port annotation you can indicate which port to scrape for metrics in a Kubernetes pod.

I'd like to be able to scrape multiple containers in a single pod. Is that possible?

For TLS termination I run HAProxy as a side container for which I export the stats using the exporter. But if the application itself is instrumented as well I'd like to be able to scrape both.

[brian-brazil]

Try the container role.

[JorritSalverda]

For my pods I use the following relabeling to use annotations to set a non-default port in the target.

- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  action: replace
  regex: (.+):(?:\d+);(\d+)
  replacement: ${1}:${2}
  target_label: __address__

But for role: containers this doesn't work in case you want to target multiple containers in a single pod . I don't think kubernetes allows you to do annotations on the containers inside a pod, does it? Any idea on how to select another than the default lowest numerical port?

[fabxc]

Containers are un-labeled. However, all meta labels you get with pod discovery should remain available for each container.

[JorritSalverda]

In https://coreos.com/blog/prometheus-and-kubernetes-up-and-running.html they use the endpoint to get to all the containers that expose a certain service. So it might make sense to label pods that expose a certain functionality like the haproxy exporter and access them through a headless service instead of using the container role. Any place I can find documentation on best practices for this?

[fabxc]

I used endpoint discovery in that case as it has been around longer. It's a more straightforward use-case and won't change much as we try to figure out more optimal approaches for container/pod discovery.

For endpoint discovery, there's no access to pod-level labels though.

[brian-brazil]

Is there any reason we couldn't add pod-level labels to endpoint discovery?

[fabxc]

Yes, endpoints in the Kubernetes API have no notion of origin. It's just a list that is written to as pods belonging to a service come and go. They are not necessarily backed by a pod and can also be maintained by third party software.

[brian-brazil]

Does that stop us from having pod labels where they are available? I'd presume the vast majority will be pods.

[fabxc]

Yes, because the API gives us a plain list of addresses and nothing else, because that's what it is.

[brian-brazil]

As part of the change to not ignore unhealthy endpoints, we'll likely be working from the service definition rather than the endpoints provided. We'll have access to all the labels then.

[fabxc]

This enters the whole discussion how the next iteration of SD would look like and won't help this issue.

[brian-brazil]

That's merely how we have to fix the existing SD to have the right semantics, and would help here if the user were to add a headless service is the container role doesn't work for some reason.

[tomwilkie]

@fabxc this can be close now, right?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.