Discover multiple (but not all) Ports in a Kubernetes Pod

[tux21b]

There are lots of examples around, that use the "kubernetes.io/port" annotation to specify the correct port to use to scrape metrics:

  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)

Unfortunately, this solution doesn't support pods running multiple prometheus exporters on different ports within the same pod. The official KubeDNS is such a pod (running SkyDNS with Prometheus metrics and a sidecar container collecting overall health metrics including dnsmasqd). But to make things worse, this container also listen on some ports that do not expose prometheus metrics. GitLab is another example of a pod containing multiple exporters.

A better solution would be to have a kuberntes.io/ports annotation containing a list of ports (maybe even named ports?), but I guess that's not possible with regular expressions alone.

How are other users dealing with this issue? Whats the proposed design? Is a pull request that filters ports based on a kubernetes.io/ports annotation programmatically a wanted addition?

[brian-brazil]

The way to handle it is to have relabel rules in Prometheus keeping only the targets with exactly the ports you need.

[tux21b]

We have already discussed several solutions. In a different issue, drewhemm suggested filtering by a) container names b) container port names or c) container port numbers. This solutions might work, but are not suitable for the common case. A annotation like "kubernetes.io/ports" that works without any side effects and other implications would be preferable.

[brian-brazil]

Unfortunately with complex setups we can only make them possible, not easy.

[grobie]

Unfortunately with complex setups we can only make them possible, not easy.

@brian-brazil This is a very common setup in Kubernetes. And I don't see what Prometheus makes possible here. It's impossible to achieve the right configuration with relabel rules at the moment.

[brian-brazil]

This is a very common setup in Kubernetes.

That something is common doesn't mean it is simple, or necessarily a good idea.

It's impossible to achieve the right configuration with relabel rules at the moment.

The feature as worded is impossible, and unlikely to be implemented for semantic reasons. Relabelling does not create targets.

There are a variety of other possibilities though, such as multiple annotations, monitoring all containers, or having a scrape config per port.

[grobie]

There are a variety of other possibilities though, such as multiple annotations, monitoring all containers, or having a scrape config per port.

The Kubernetes discovery returns one target per port by now. The issue to support configurations only defined in Kubernetes is that it's impossible to match the value of two annotations.

[brian-brazil]

it's impossible to match the value of two annotations.

That depends on what exactly you mean by matching the value of two annotations.

Don't think of this in terms of one particular potential solution, think more about the high level problem and ways it could be approached. Consider for example an annotation called "port2" in addition to "port".

[grobie]

That doesn't help here either. What we need is to keep a target if it's port label matches an annotation label. To my knowledge that's not possible with relabel rules, as RE2 doesn't support backreferences.

Coincidentally, @discordianfish just opened an issue with a proposal which would solve this issue at hand. I commented at #3117

[brian-brazil]

That doesn't help here either.

How doesn't it help exactly? As far as I can see it's perfectly doable with multiple scrape configs.

[matthiasr]

Given a pod with ports "http", "thrift", "telemetry", "mtail", and annotations "port=telemetry" and "port2=mtail", please demonstrate how I can filter targets to scrape the telemetry and mtail ports without prior knowledge of these port names.

[matthiasr]

Oh I see what you mean. The good ol' "shove the value of the annotation into the __address__ label". In that case, how can I validate that the port number given in the annotation is valid, as in: matches one of the declared ports?

[brian-brazil]

please demonstrate how I can filter targets to scrape the telemetry and mtail ports without prior knowledge of these port names.

That's not the feature requested in this issue, which is about port numbers. I'd have to think on that one a bit, but I think you'd have to use the port numbers or something like the operator.

how can I validate that the port number given in the annotation is valid, as in: matches one of the declared ports?

That's a problem with "kubernetes.io/port" generally, if you put in bad information you'll get a bad target. I don't think it's a problem in practice in this context.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.