[Enhancement]: Option of Createing Port Mapping for Prometheus HTTP Port (9404)

[hoptical]

Related problem

Regarding Kafka Connect CRD, It seems that the Strimzi operator only creates port mappings for port 8083 (Connect REST API) and 9999 (JMX port), and there isn't any option to create a mapping for Prometheus exporter (9404) for external Prometheus services. Hence, we have to create the service on our own.

Suggested solution

The operator should have the option to create a mapping for port 9404 (Prometheus exporter) in the service.

Alternatives

No response

Additional context

No response

[scholzj]

The issue with this was that when we did support it, it was causing issues to some users. So we actually removed this in the past. Not having the port there means that it does not break anyone's environment and you can still add your own service.

Also, please keep in mind that you cannot scrape the metrics over a service which points to multiple brokers => it would lead Prometheus every time to a different broker and the time series will make no sense. AFAIK Prometheus uses the services only as a discovery mechanism but normally connects directly to the Pods, which is why using the PodMonitor directly is much easier.

[hoptical]

@scholzj Thanks for your response.

I need to scrape Kafka Connect metrics via the Prometheus exporter and with an external Prometheus service. So normally, it's connecting to Kafka connect pods, not the broker pods. Am I right? Now according to what you said, does Prometheus scrape lead to a non-sense time series while having a cluster of Kafka connect pods?

[scholzj]

I need to scrape Kafka Connect metrics via the Prometheus exporter and with an external Prometheus service. So normally, it's connecting to Kafka connect pods, not the broker pods. Am I right? Now according to what you said, does Prometheus scrape lead to a non-sense time series while having a cluster of Kafka connect pods?

Right, my mistake. But the principle is the same. The Connect service points to all the pods belonging to the given Connect cluster. So if you try to use it to scrape metrics, it will not work ... it will do this:

• First connection: Connects to the service and is routed to pod A and scrapes metrics from pod A which cover only the Pod A
• Second connection 30 seconds later: Connects to the service which routes it to pod C and scrapes the metrics from Pod C which cover only Pod C
• Third connection 30 seconds later: Connects to the service which routes it to pod B and scrapes the metrics from Pod B which cover only Pod B

So basically every time, you will reach different pods and scrape their metrics. So your time series will have one data point from Pod A, one from Pod C and one from Pod B etc. Since each pod has its own metrics, they will not make sense.

I'm not a real Prometheus expert. But I think the right way to do this is to scrape locally and forward the data to your remote Prometheus. cluster. I think there are several tools doing this, for example https://prometheus.io/blog/2021/11/16/agent/.

[hoptical]

Right. Thanks for your rich response.

[scholzj]

Triaged on 20.4.2023: Should be closed due to the reasons explained above.