How to scale down to zero a KRaft cluster?

[mossroy]

The need is to (usually temporarily) stop the kafka pods, while keeping the data in their PVs untouched. For example to safely backup the PVs, or to save resources when you don't actually use the cluster.

Without KRaft, you can simply set spec.kafka.replicas to 0 in the Kafka k8s object
With KRaft, the actual number of kafka pods is in the NodePool.
But if I set spec.replicas to 0 in the KafkaNodePool k8s object, the operator does not stop the pods, and throws this exception:

2024-05-27 09:17:31 WARN  AbstractOperator:556 - Reconciliation #3987(watch) Kafka(mynamespace/my-kafka-cluster): Failed to reconcile
io.strimzi.operator.common.model.InvalidResourceException: KafkaNodePools are enabled, but KafkaNodePools for Kafka cluster my-kafka-cluster either don't exist or have 0 replicas. Please make sure at least one KafkaNodePool resource exists, is in the same namespace as the Kafka resource, has at least one replica, and has the strimzi.io/cluster label set to the name of the Kafka resource.

How could I do that properly?

A workaround would probably be to scale down the strimzi operator to zero, then delete the pods, but it would stop reconciliation for all the k8s cluster

[scholzj]

You cannot scale Kafka to 0 the way you describe it because it holds the data. What you seem to want is to stop / shutdown the cluster. To do that, you can:

• Pause the reconciliation of the Kafka CR
• Delete the StrimziPodSet resources and Pods belonging to it
• Do whatever you need to do to the storage

And to start the cluster again, you can just unpause the Kafka CR again.

There are some previous discussions about this that might contain some more details.

[mossroy]

Thanks, this works.
The annotation to add to Kafka CR is strimzi.io/pause-reconciliation: "true"

I would have preferred to be able to set spec.replicas to 0 in the KafkaNodePool, if it worked. It would avoid the post-upgrade Helm hook I had to implement to delete the StrimziPodSet.

However, thanks again!

[scholzj]

I would have preferred to be able to set spec.replicas to 0 in the KafkaNodePool, if it worked. It would avoid the post-upgrade Helm hook I had to implement to delete the StrimziPodSet.

I'm not sure how does anything in this relate to Helm.

But once again, you have to keep in mind that scaling something is not the same as shutting it down temporarily - especially for stateful applications - and it might not behave the same way as you expect.

[mossroy]

Yes, sorry, I did not explain that I needed to stop or restart the cluster if a parameter is set in the values.yaml of Helm (and do it when the user runs helm install or helm upgrade). It was intentional to not create confusion.

Using Helm templating (to set the replica count to zero in a manifest) would have been a better fit than having to implement StrimziPodSet deletion in a helm hook. But at least it's working properly.

Many thanks