Is there anyway to activate auto scaling or some form of auto scaling with Strimzi?

[cstmgl]

Hi all,

Maybe this is something basic and clear in the documentation but is there any way I can use the operators to preform auto-scaling on the application?

Say for example you would like your replicas to fluctuate between 5 and 10 pods depending on the consumption.

I use Strimzi in the context of Openshift and I know they have some sort of auto scaling capabilities, which I have not explored much, but at first glance I did not find anything in Strimzi about it so I wanted to ask about that.

https://docs.openshift.com/container-platform/4.9/machine_management/applying-autoscaling.html

If it's not possible, is there any plan of having that in the roadmap, or maybe it's really something that does not make sense in the context of Kafka, not sure...

Thanks for any feedback, best regards,

[scholzj]

Auto-scaling Kafka is complicated. It usually cannot be done just based on some CPU utilization etc.

• If you want to scale consumers, you need to understand their consumer group membership and which topics are they consuming. Because the maximum number of replicas is for example limited with number of partitions from which they are consuming. You need to use tools such as for example KEDA to autoscale them which have some additional logic to take these things into account.
• If you want to auto-scale components such as Connect, Connectors, Bridge etc., Strimzi gives you the scale subresources to plug it into Kubernetes HPA and tools like KEDA. These are basically consumers and producers in a special packaging. So the same rules as described above apply for them.
• For Kafka brokers, auto-scaling is complicated because of their architecture. Adding or removing brokers is simple. But directing some load to them is complicated because they are in a way form of data storage. And moving the whole partitions between brokers is expensive. The partitions often contain huge amounts of data which need to be shifted from one broker to another - that will take time, it will have a performance penalty on the other traffic and possibly cost even real money for the data transfers. Plus it still might not work because if your bottleneck is for example a topic with 5 partitions, it might not matter whether you have 5 or 10 brokers. So from my experience, only rarely autoscaling of Kafka brokers makes sense.

That said, the autoscaling you linked to is about autoscaling the Kubernetes cluster which is a different story. It is about adding or removing Kubernetes worker nodes and not about adding or removing pods which is what your question refers to at the beginning. This does not necessarily involve changing number of Kafka pods but depending on other applications, they might need to be moved between nodes to for example scale down the number of worker nodes. Strimzi in general does no have an issue with it. You should use the Strimzi Drain Cleaner to ensure the availability of the Kafka cluster during these events. But otherwise you should be fine.

[cstmgl]

Thank you for the feedback, I may have underestimated the complexity of auto-scaling Kafka.
For sure what I was looking was for ways to increase and decrease the tasks and replicas for KafkaConnect and KafkaConnector I'll take a look at your suggestion about the KEDA.

Thank you for the feedback.

And yes the auto-scaling I put from Openshift was maybe a bit away too fast google on the feature I did not notice it was about the cluster and not the applications themself which I guess is a bit too generic to assume that for all applications you can just increase or decrease pods based on the load.
Anyway thanks for the feedback