Multiple Kafka Clusters for resiliency

[jonsnowseven]

Hello.

I am investigating a way of avoiding having Kafka brokers down when changing Kafka Clusters.

One idea that came to mind was to have 2 Kafka Clusters in an active-passive way to avoid having brokers down. Is this possible with Strimzi Kafka? Maybe using MirrorMaker?

This is the behaviour we were looking for:

1. Kafka Cluster A and B (A active, B passive). Both fully synced. Cluster A is constantly getting requests because we have a real-time service producing events to it.
2. For instance, if we want to update Kafka Cluster's version
   1. Update version on cluster B, which triggers broker restarts
   2. Change cluster B to active and cluster A to passive (not sure how the traffic would be handled to be seamless transition for the producers and consumers)

(I might me oversimplifying this behaviour but just to outline what I am looking for)

Thank you in advance!

[scholzj]

I'm not sure why would you need such a thing. Active-passive mirroring is used for disaster recovery purposes. But if you want, you should probaly start by trying it out to see how easy it is to switch the passive cluster to active and the other way around.

[jonsnowseven]

Because we want to avoid broker restarts since they have a negative impact on the producers.

But if you want, you should probaly start by trying it out to see how easy it is to switch the passive cluster to active and the other way around.

Are you saying from the producers side or in the Kafka Strimzi itself?

[scholzj]

Because we want to avoid broker restarts since they have a negative impact on the producers.

Broker restarts are a common part of the Kafka cluster lifecycle. There should be no problem in handling them. If you have such problems, they can happen anytime in case of worker node restarts, crashes, evictions etc.

Are you saying from the producers side or in the Kafka Strimzi itself?

Not sure what you mean with this, sorry.

[jonsnowseven]

You said to start by trying to see how easy it is to switch the passive cluster to active (and vice-versa):

1. In the sense that the producers should update their bootstrap servers?
2. Or in the sense that we could switch the traffic from the active cluster to the passive cluster (like in a load balancer)? (not even sure if this is possible)

[scholzj]

It is not about just reconfiguring the producers. You have to also reconfigure consumers and if you are sensitive to duplicates, you need to reconfigure every consumer group at the same time. You also have to sync the mirroring with the producers if you care about things such as message ordering. But that part is still simple. Once you switch to the passive node, you can basically throw away the old active node away and you can start from zero with a new cluster. It is really quite complicated even if you have to do it after some actual disaster. That is why I encouraged you to actually try it in some environment to properly understand the complexity.

[jonsnowseven]

Oh got it!
Thank you for the input 👌