Is there a good way to manage Strimzi Kafka Connect Cluster and Connector in production?

[Haebuk]

hello. I am no longer using MSK Connect and would like to deploy Connect Cluster and Connetor on EKS using Strimzi.

I've deployed the Connect Cluster and Connector in EKS and built a dashboard in Grafana, and now I'm one step away from using it in production: How to effectively manage the resources of the Kafka Connect Cluster.

For example, here's a scenario .

1. I have multiple source and sink connectors, each with different amounts of resources they occupy (CPU and Memory).
2. As the usage of a particular connector increases, I need to increase the resources of the Kafka Connect Cluster that holds that connector. (Conversely, if the usage decreases, I need to decrease it).

This article explains that it is possible to deploy multiple Kafka Connect Clusters. Does this mean that I should assign the appropriate Kafka Connect Cluster based on the resource usage of the Connector? Or is it better to deploy and assign one Kafka Connect Cluster per connector (I think this is related to scenario 1)?

For scenario 2, can I simply change the resource in the yaml file and re-appy it? If I do this, will the connectors not be missing data during the brief downtime (or will there be any other impact), or is there a better way?

The problem I'm currently encountering is in the above two cases. If there are more things to consider when using Strimzi Kafka Connect Cluster and Kafka Connector in production, please let me know :)

[scholzj]

The resources are configured in the KafkaConnect custom resource. You can change them by editing it. Whether to run multiple connects with one or few connectors each or one Connect cluster with many connectors - that is usually a complicated discussion where the outcome is more about your opinion and not facts and everyone prefers something else.

For example, one Connect cluster with many connectors is usually slightly more efficient as the Connect overhead (JVM, running the orchestration layer etc.) is shared between many connectors. On the other hand, there is no proper isolation between the connectors, no advanced scheduling or per-connector resources (the resources of the Connect cluster are just all shared) which is why some prefer to run a different Connect cluster for each connector. But when you sum it up, it usually consumes more resources overall (not just CPU and memory, but it is obviously also harder to manage etc.). It is up to you which of these things matter more to you and which way you choose.

[Haebuk]

cool, how about logging? Currently, I don't have a separate logging configuration, so it's being written to the connect cluster pod log. The more connectors we have in the cluster, the longer the time period, the harder it seems to see the logs. Is it recommended to use an external service to separate the logs (e.g. s3, datadog, etc.)?

[scholzj]

I'm not aware of any special recommendation for logging. If you collect the logs into some logging stack such as OpenSearch or ElasticSearch, you should be able to easily filter it out there.