Use topic,user and connect CRDS in different namespace than kafka-cluster

[obbiondo]

Hi, I currently have a kafka cluster deployed with Strimzi in a specific namespace created with this scope. Now I would like to manage topic, user and connect CRDs in different tenant-specific namespace, all referencing to the same kafka-cluster.
Is it possible to do so?

[scholzj]

No, that is currently not supported.

[scholzj]

Yeah, it would be useful. But it is also very complicated to implement, especially for the Topic Operator.

[ErikEngerd]

I have two use cases:

1. production: kafka cluster deployed in one namespace and kafka users and kafka topics defined in other namespaces depending on the application components
2. test: kafka cluster deployed in a separate namespace with also kafka users and kafka topics in the same namespace.

In the clusters I am using the operator is installed with a configuration to watch all namespaces.

Now I am wondering what the behavior or strimzi is and if it supports my use cases.

On production, when I create a KafkaUser or KafkaTopic in another namespace, will the user/topic be created on the kafka cluster that is already running, or will it fail because there is no kafka cluster running in the same namespace as the KafkaUser and KafkaTopic?

For the test use case I can imagine it can be supported by having a separate install of the operator in a separate namespace configured to monitor only that single namespace. However, in the current setup (of which I have no control at the moment), the operator watches all namespaces. What will happen now when I deploy a KafkaUser or KafkaTopic? Will these be create on the cluster that is already running? Alternatively, should I create a separate install of the operator in my new namespace and configure KafkaUser and KafkaTopic in the same namespace?

Also, I am wondering what the difficulty that is mentioned in the previous post. If one would describe the object model of Kafka, then it is clear that a KafkaUser is linked to one or more kafka clusters and the same for a topic. So why not provide an explicit configuration to linka KafkaUser and KafkaTopic to a specific Kafka resource in a specific namespace? Is that already supported perhaps? Could not find any relevant settings for this when doing kubectl explain --recursive on KafkaUser and KafkaTopic or did I overlook something?

[scholzj]

@ErikEngerd I'm not sure I properly understand the use-cases you have. But maybe this helps you to answer them:

• The KafkaUser and KafkaTopic resources always belong to a single Kafka cluster. You cannot use a single resource to configure a user or topic in multiple clusters. The cluster to which they belong is defined by the strimzi.io/cluster label.
• When configuring the Kafka cluster, you can configure the namespace where the KafkaUser and KafkaTopic resources can be (check the watchedNamespace field in the Topic and USer operator configurations inside the Kafka custom resource). But it will be always a single namespace, not multiple namespaces.

[ErikEngerd]

It looks like that answers my question.

As an additional question. What will happen if the label is not unique? I.e. there are two Kafka resources in different namespaces with the same name? Anyway, seems like a good idea anyway to define kafka clusters with unique names.

[scholzj]

If you create a KafkaUser or KafkaTopic resource with a label for example as strimzi.io/cluster: my-cluster in a namespace (e.g. namespace1) that you configured as watched namespace from two different Kafka clusters named my-cluster running in other namespaces (e.g. namespace2 and namespace3) you might get very unpredictable result. The operators might fight over the resource both updating it constantly all the time as they are overwriting each other, you might have various random conflict errors, etc. So you definitely want to avoid that.

In general, I'm not aware of many users using the watchedNamespace to have the user and topic resources in some other namespace. The best solution is usually to have:

• One Kafka cluster per namespace
• Have the user and topic resources in the same namespace
• If needed, use various GitOps approaches to allow the users of that cluster to manage the users and topics in the cluster namespace
• If needed, use tools such as this to sync the certificate or password secrets into the namespaces where the applications are running.