-
hello. I am no longer using MSK Connect and would like to deploy Connect Cluster and Connetor on EKS using Strimzi. I've deployed the Connect Cluster and Connector in EKS and built a dashboard in Grafana, and now I'm one step away from using it in production: How to effectively manage the resources of the Kafka Connect Cluster. For example, here's a scenario .
This article explains that it is possible to deploy multiple Kafka Connect Clusters. Does this mean that I should assign the appropriate Kafka Connect Cluster based on the resource usage of the Connector? Or is it better to deploy and assign one Kafka Connect Cluster per connector (I think this is related to scenario 1)? For scenario 2, can I simply change the resource in the yaml file and re-appy it? If I do this, will the connectors not be missing data during the brief downtime (or will there be any other impact), or is there a better way? The problem I'm currently encountering is in the above two cases. If there are more things to consider when using Strimzi Kafka Connect Cluster and Kafka Connector in production, please let me know :) |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
The resources are configured in the For example, one Connect cluster with many connectors is usually slightly more efficient as the Connect overhead (JVM, running the orchestration layer etc.) is shared between many connectors. On the other hand, there is no proper isolation between the connectors, no advanced scheduling or per-connector resources (the resources of the Connect cluster are just all shared) which is why some prefer to run a different Connect cluster for each connector. But when you sum it up, it usually consumes more resources overall (not just CPU and memory, but it is obviously also harder to manage etc.). It is up to you which of these things matter more to you and which way you choose. |
Beta Was this translation helpful? Give feedback.
The resources are configured in the
KafkaConnect
custom resource. You can change them by editing it. Whether to run multiple connects with one or few connectors each or one Connect cluster with many connectors - that is usually a complicated discussion where the outcome is more about your opinion and not facts and everyone prefers something else.For example, one Connect cluster with many connectors is usually slightly more efficient as the Connect overhead (JVM, running the orchestration layer etc.) is shared between many connectors. On the other hand, there is no proper isolation between the connectors, no advanced scheduling or per-connector resources (the resources of the Connect clus…