How to configure kafka connect metrics ?

[raghulp18]

Hi Team,

I want to know how to configure a kafka connect metrics using a jmx and export it to prometheus ?
I have already gone through a configuration file of kafka-connect-metrics over here - https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-connect-metrics.yaml, but after adding this in the kafka-connect.yaml where can i check the metrics and how to export the port ?

Below is the configuration file of the kafka-connect in my local.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: strimzi-kafka-connect-cluster
annotations:
strimzi.io/use-connector-resources: "true"
namespace: kafka
spec:
version: 3.6.1
replicas: 1
image: snowflake-jdbc-image
bootstrapServers: my-cluster-kafka-bootstrap:9092
metricsConfig:
type: jmxPrometheusExporter
valueFrom:
configMapKeyRef:
name: connect-metrics
key: metrics-config.yml
config:
group.id: my-cluster
offset.storage.topic: strimzi-connect-cluster-offsets
config.storage.topic: strimzi-connect-cluster-configs
status.storage.topic: strimzi-connect-cluster-status
config.storage.replication.factor: 3
offset.storage.replication.factor: 3
status.storage.replication.factor: 3
config.providers: directory
config.providers.file.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider

---

kind: ConfigMap
apiVersion: v1
metadata:
name: kafka-metrics
labels:
app: strimzi
data:
metrics-config.yml: |
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
#kafka.connect:type=connector-metrics,connector="{connector}"
- pattern: 'kafka.connect<type=connector-metrics, connector=(.+)><>(connector-class|connector-type|connector-version|status): (.+)'
name: kafka_connect_connector_$2
value: 1
labels:
connector: "$1"
$2: "$3"
help: "Kafka Connect $2 JMX metric type connector"
type: GAUGE

Please help here

Thanks
Raghul P

[scholzj]

You will have to format the YAMLs to make them readable. But in general, after you apply the YAML from the examples, the metrics will be exposed on every Connect Pod on port 9404 (path /metrics/).

[raghulp18]

Is there a way to change the metrics/prometheus metrics endpoint ?

[scholzj]

And I'm able to connect to metrics on curl http://localhost:9404/metrics

Right, so that shows the metrics are working. To expose only some metrics, you would need to edit the rules in the ConfigMap. But I'm afraid I do not have any guide for how to do it and it might be also different for each connector (assuming you want some metrics produced by the connector itself).

[raghulp18]

Yeah, thanks for the help. But I want to know is there a way to change/overwrite the metrics to a different port ?

[raghulp18]

And where can I check all the available links under localhost:9404/metrics/ ?

[scholzj]

It is just the metrics, there is nothing else as far as I know.