Topic management via CRD

[pbalogh-sa]

Describe the solution you'd like to see
It would be useful if topic management could be configurable via CRD

[prune998]

Strimzi is doing this well... not sure if something more complicated is needed ?
Been using Kafka for years and never had to use more tweaking that what is presented in https://strimzi.io/docs/latest/#type-KafkaTopic-reference

At least it would be a starting point

[prune998]

Anyone working on this ?

[baluchicken]

Hi @prune998, thanks for asking. Currently we have higher priority features please check the roadmap for more details.

[tinyzimmer]

@baluchicken FWIW I am working on a kafka on Kubernetes implementation right now just using helm's incubator/kafka hacked up to properly support SSL (and disable PLAINTEXT). I wanted topic/user mgt via CRDs and wrote a sidecar operator that can handle them using operator-sdk and talking to the brokers via sarama.ClusterAdmin.

It could probably be easily plugged into this project either also as a sidecar to the clusters, or since it's already go scaffolding for the most part, it could also be lifted into the main operator I think. It currently gets its broker URL and TLS config from environment variables and a mounted secret, but if it were to be brought into the main operator it would probably make more sense to get auth information per event directly from the cluster CRDs and secrets (using a clusterRef from the KafkaUser/KafkaTopic CR specs or something).

The operator assumes a cert-manager issuer backing the SSL for users but it could be refactored to point at some other TLS provider. I have another tiny operator that can inject a cert-manager certificate-secret with JKS format (I use it for the brokers), but I can't decide if that's really needed or where it's final home would be. I haven't paid any attention to SASL as an option.

Maybe we can help each other out and then I could move to using a cluster that is a lot more impressive than my current setup, but still have my CRDs for CI/CD 😛 . I can put the controller on my Github, but I'd probably want to rename the apigroups and stuff first. If you'd like me to submit a PR I'd be open to any design changes you all recommend.

[tinyzimmer]

To follow up, I decided to just fork and give a go at merging my stuff into this operator. I managed to get it working and if you'd be open to a PR I can definitely get one started. The fork is here: https://github.com/tinyzimmer/kafka-operator

A few main things to note:

• I added support for the user to specify KafkaCluster.Spec.ListenersConfig.SSLSecrets.Create. This will create a full PKI for the cluster and clients using cert-manager.
• A cert-manager ClusterIssuer is made from the user-provided secret if the above is false. I understand this would make cert-manager a core dependency and I was possibly going to look into giving the controller the logic to generate and sign certificates instead.

In regards to the additions, here are a few sample CRs for the two new resources I add:

apiVersion: banzaicloud.banzaicloud.io/v1alpha1
kind: KafkaTopic
metadata:
  name: test-topic
  namespace: kafka
spec:
  clusterRef:
    name: kafka
  name: test-topic
  partitions: 6
  replicationFactor: 2
  config:
    "retention.ms": "800000"
    "cleanup.policy": "delete"
    # ... etc, etc
---
apiVersion: banzaicloud.banzaicloud.io/v1alpha1
kind: KafkaUser
metadata:
  name: test-kafkauser
  namespace: kafka
spec:
  clusterRef:
    name: kafka
  name: test-kafkauser
  secretName: test-kafkauser-secret
  includeJKS: true
  topicGrants:
    - topicName: test-topic
      accessType: read
    - topicName: test-topic
      accessType: write

If you would be interested in a PR, I'd be happy to make any fixes and changes y'all recommend. I do know I'd want to clean up a few places first. For example, the cruise control resource having it's own topic logic is no longer necessary, and there is some code left over from when I originally ran it as a sidecar to a cluster.

[prune998]

You should make a PR right away (when you are ready, or create it as WIP). I know this subject is not the top priority right now, but I'm sure there will be someone to review your code (and I will do too).
At least it will give an option for others (like me) until it is reworked or merged into master.

Thanks for your work !

[tinyzimmer]

I'll clean it up a bit more then open a PR as a WIP. I know at the very least I'll probably need to create some user documentation and stuff to go along with it.

[baluchicken]

Thanks for deciding to fork and contribute this.

[baluchicken]

Thanks for doing this @tinyzimmer