[Enhancement] The user operator to create secrets in other namespaces which contains both user and cluster CA certs

[allanmoso]

Is your feature request related to a problem? Please describe.
We are using TLS authentication for our Kafka clients. Because strimzi's user operator is only able to create user secrets (and CA secrets) in one namespace we are not able to mount those secrets on our Kafka client pods that reside in other namespaces without having to duplicate the secret to the other namespace by other means.

This issue is also the same for the cluster's CA cert which the client needs for its truststore.

Describe the solution you'd like
It would be great if the user-operator is able to watch all namespaces and create the secret in the namespace where the KafkaUser was reconciled from. Because the kafka client also needs the cluster CA cert to connect to the cluster, it would also be nice if the user secret also contains that. This would make it so that everything that a Kafka client application needs to connect to the Kafka cluster is available as a secret in the namespace where it would be deployed, if a KafkaUser object is created in that namespace.

Describe alternatives you've considered

• The simplest way would be to have the admins move the secret between namespaces. But this would put too much operational burden on the admins, especially because of auto-renewals. We did not go with this approach.
• We created a k8s controller that watches secrets with an annotation that defines a source namespace. We would then create these empty placeholder secrets. The k8s controller will then load the secret with data from the source secret based on the source namespace annotation and that matches its name of the placeholder secret.

Additional context
The current behavior of the user operator is inconvenient for us based on how we want to manage our k8s cluster. We are pushing for self-service for our developers when it comes to deploying to the kubernetes cluster. However, developers are not able to interact with secrets directly, for security reasons. This prevents them from moving strimzi secrets between namespaces for their applications to use.

We are doing "GitOps" as a way to manage our cluster using Weaveworks' Flux

We are also using Stakater's Reloader tool that would allow us to restart pods when secrets or configmaps are changed, e.g., during CA auto-renewals.

[Fryuni]

This would definitely would help us a lot.
We are in a similar scenario, for now we are putting all the applications that interact with Kafka in the same namespace as the cluster. That works, but it's not ideal by any means. Organizing those applications in namespaces would be much better for managing policies, permissions, quotas and priorities.

[Bonehead5338]

+1, would be very helpful

[MarkusTiede]

Thanks for sharing all of the information - @allanmoso: would you be interested in maybe sharing / open-sourcing your solution? We're atm building a very similar solution and are also thinking about doing that; really appreciate your comments / feedback!

[jutley]

You can allow Strimzi to manage users in other namespaces by specifying STRIMZI_NAMESPACE="*" and STRIMZI_CA_NAMESPACE=<whereever strimzi is running>. Unfortunately, this doesn't at all address the issue of the truststore not being available from other namespaces.

[alok87]

+1
It is a very important thing from the renewal point of view. Strimzi takes care of renewing the certificates based on its renewal flags. So if at present we manually duplicate the secret, later it would again be a problem when we need to renew those certificates.

If the strimzi operator takes care of keeping the secrets created in the right namespace it would be really helpful,

[dweber019]

We have written a Quarkus Operator to distribute secrets based on a label called dist-namespace-1: <namespace>. If a secret has this label it will distribute the secret and if the secret has in addition a label strimzi.io/kind: KafkaUser it will distribute the CA too.

Sorry but currently the source is not open source.
@MarkusTiede what do you think?

[MarkusTiede]

@dweber019 : Thanks for sharing the information! IMHO there is no reason why we shouldn't / couldn't share this as an open-source project at ours. Depending on the level of maturity I'd either place it within our top-level organization /baloise - or otherwise the /baloise-incubator. But iirc this is in productive use @ ours which is why I'd place it directly top-level.

[dweber019]

Nice will check how I can open source it ;)

[dweber019]

Here the link to our Quarkus app https://github.com/baloise/kube-secret-watcher as a first draft release.

[rossbachp]

A more general tool to copy secrets and cm can help:

https://github.com/kubeops-tools/kubed

[veerendra2]

+1
Right now our requirement is share secrets to different stages(namespace) in a cluster

[veerendra2]

I think, it is available from 0.24
https://strimzi.io/blog/2021/07/22/using-kubernetes-config-provider-to-load-data-from-secrets-and-config-maps/

[scholzj]

@veerendra2 This was one of the use-cases I thought about when designing the configuration provider. It might not be perfect since you need to do the RBAC setup etc., so it is maybe not as easy as using the local secret. But you should definitely give it a try.

[dweber019]

If anyone inserted we built an operator which can copy stuff by providing coordinates in a CRD yaml.
We needed this to copy secrets created by Strimzi and RHSSO (Keycloak) into other namespaces. Both of them have different implementations on handling/templating we made it declarative and resource based not operator capability based.

https://github.com/baloise/os3-copier

[shaakaud]

This would definitely would help us a lot. We are in a similar scenario, for now we are putting all the applications that interact with Kafka in the same namespace as the cluster. That works, but it's not ideal by any means. Organizing those applications in namespaces would be much better for managing policies, permissions, quotas and priorities.

Did you find a better way to organize applications in a different namespace which can access the Cluster ?

[scholzj]

Triaged on 17.3.2022: This can be addressed by some of our existing utilities:

• The Kubernetes Config Provider (already released, works with Java clients)
• The Kafka Access operator (work in progress) which will allow to copy the cluster credentials into other namespaces.

[ltson4121994]

Triaged on 17.3.2022: This can be addressed by some of our existing utilities:

* The Kubernetes Config Provider (already released, works with Java clients)

* The Kafka Access operator (work in progress) which will allow to copy the cluster credentials into other namespaces.

I have a use case where each kafka connect cluster represents a user and is deployed in different namespaces. I can see that there is an example on using secrets in different namespace with KafkaConnector, but is there any way we can reference that secret in KafkaConnect .spec.tls.trustedCertificates? RBAC would not work here I think because kafka connect cannot be deployed.