Usable for multiple separate/different Kafka instances?

[kinglite]

Hey,

I read here and there about Galapagos and it sounds very interesting.
But what I maybe missed, is a statement if it would be able to use Galapagos for multiple different and separate Kafka instances.
Meaning one instance is for example running on AWS and another is running on a VM somewhere on a non-cloud datacenter.

Would Galapagos be able to bring this two instances together?
One: In terms of self-servicing topic creation and access management on each of those instances?
Two: In terms of enabling communication between those instances? Meaning that topics from one instance could be "replicated" to the other instance if needed. And therefore accessible to applications running on the corresponding environment (assuming firewalling/networking between the two instances is established)

Kind regards

[albrechtflo-hg]

Hi @kinglite ,

Thanks for your question! Indeed did we discuss if Galapagos should support "multi-cluster mode" - and, to make a long story short: It does not, yet.

Galapagos does support "multi-stage" setups, e.g. having one cluster as Development stage, one for Integration Testing, and one for Production (or more stages, or less). Each stage may be a Confluent Cloud based cluster or a self-operated one (either in a cloud VM or self-hosted data center), so each combination is allowed.

But allowing a "multi-cluster" setup per stage would have introduced even more complexity into Galapagos, and things like replication between clusters is nothing which could be done "out of the box" with plain Kafka clusters, but would e.g. require integration with tools like Kafka Connect or with Confluent Cluster Linking.

With that said, this is not yet completely ruled out for the future. Currently, we are discussing which "migration scenarios" we want to support in the future (as we just had a not-so-easy migration from a Confluent Standard Cluster to a Dedicated one), and what additional features Galapagos might require for supporting this. Especially the "Cluster Linking" functionality of Confluent Cloud, which can be administered via REST API, could relatively easy be integrated.

As always with Open Source software, of course feel free to contribute, if you have some ideas on how these features could be integrated into Galapagos 😃