Addon Kafka-operator

[MdSahil-oss]

Description of your changes

Adds kafka-operator as an addon

How has this code been tested?

Checklist

I have:

• Title of the PR starts with type (e.g. [Addon] , [example] or [Doc]).
• Updated/Added any relevant documentation and examples.
• New addon should be put in experimental.
• Update addon should modify the version in metadata.yaml to generate a new version.

Verified Addon promotion rules

If this pr wants to promote an experimental addon to verified, you must check whether meet these conditions too:

• This addon must be tested by addon's e2e-test to guarantee this addon can be enabled successfully.
• This addon must have some basic but necessary information.
  • An accessible icon url and source url defined in addon's metadata.yaml.
  • A detail introduction include a basic example about how to use and what's the benefit of this addon in README.md.
  • Also provide an introduction in KubeVela documentation.
  • It's more likely to be accepted if useful examples provided in example dir.

[MdSahil-oss]

@wonderflow @wangyikewxgm when I'm running below yaml after enabling the addon with definition kafka-cluster

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: kafka-cluster-sample
spec:
  components:
    - type: "kafka-cluster"
      name: "kafka"
      properties:
        kafka:
          version: "3.3.2"
          replicas: 3
          listeners:
            - name: plain
              port: 9092
              type: internal
              tls: false
            - name: tls
              port: 9093
              type: internal
              tls: true
          config:
            offsets.topic.replication.factor: 3
            transaction.state.log.replication.factor: 3
            transaction.state.log.min.isr: 2
            default.replication.factor: 3
            min.insync.replicas: 2
            inter.broker.protocol.version: '3.3'
          storage:
            type: ephemeral
        zookeeper:
          replicas: 3
          storage:
            type: ephemeral
        entityOperator:
          topicOperator: {}
          userOperator: {}

no pods are getting created and no error occurs. Do you see anything I'm missing or making any mistake.

[wangyikewxgm]

You can check the status of applied CustomResource.

[MdSahil-oss]

@wangyikewxgm I've checked and found, resource name kafka gets created with the CRD Kafka for the expected replica set but pods doesn't get created. The same thing is happening with the addon redis-operator in my machine (Pods doesn't get created) can you test the redis-operator in your machine with the YAML:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: redis-operator-sample
spec:
  components:
    # This component is provided by redis-operator addon.
    # In this example, 2 redis instance and 2 sentinel instance
    # will be created.
    - type: redis-failover
      name: ha-redis
      properties:
        # You can increase/decrease this later to add/remove instances.
        replicas: 3

so that we can address the issue with the kafka-operator.

[wangyikewxgm]

@wangyikewxgm I've checked and found, resource name kafka gets created with the CRD Kafka for the expected replica set but pods doesn't get created. The same thing is happening with the addon redis-operator in my machine (Pods doesn't get created) can you test the redis-operator in your machine with the YAML:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: redis-operator-sample
spec:
  components:
    # This component is provided by redis-operator addon.
    # In this example, 2 redis instance and 2 sentinel instance
    # will be created.
    - type: redis-failover
      name: ha-redis
      properties:
        # You can increase/decrease this later to add/remove instances.
        replicas: 3

so that we can address the issue with the kafka-operator.

Ok, will have a try.My slack ID is YikeWong, feel free to discuss on slack.

[MdSahil-oss]

@wonderflow @wangyikewxgm I've made change like removed definitions MirrorMaker, MirrorMaker2 and kafka-user and updated README. can you have a look on that ?

[MdSahil-oss]

@wonderflow I've updated Readme, Now Readme explains how kafka can be used as a Producer/Consumer using kafka-topic, Please review it.

[MdSahil-oss]

@wonderflow we can't create a vela Application because the image producer/consumer uses, runs a shell application that must be connected in interactive mode to interact with Topic otherwise producer/consumer finishes running instantly, So we can create YAMLs to create producer/consumer pods but can't connect to them as they aren't stable to connect.
So, To run producer/consumer we must connect to them in interactive mode at the time of running producer/consumer.

[MdSahil-oss]

And all the messages that're generated through producer can be consumed using consumer from Topic

[MdSahil-oss]

@wonderflow I've added instructions in Readme to create a NodePort Service to access kafka-bridge & replaced uses of kubectl with vela Please Review it.

[wonderflow]

great job, please fix the CI

[MdSahil-oss]

@wonderflow I've fixed CI test.