Replace Zookeeper Dependency in Docker Compose for Modern Kafka

[cubxxw]

We have recently upgraded to Kafka version 3.5.1 in our Docker Compose setup. Notably, recent versions of Kafka no longer have a dependency on Zookeeper. However, our current Docker Compose file still contains configurations for Zookeeper.

  zookeeper:
   image: bitnami/zookeeper:3.8
   container_name: zookeeper
   ports:
    - "${ZOOKEEPER_PORT}:2181"
   volumes:
     - "/etc/localtime:/etc/localtime"
   environment:
     - ALLOW_ANONYMOUS_LOGIN=yes
     - TZ="Asia/Shanghai"
   restart: always
   networks:
      server:
        ipv4_address: ${ZOOKEEPER_NETWORK_ADDRESS}

  kafka:
    image: 'bitnami/kafka:3.5.1'
    container_name: kafka
    user: root
    restart: always
    ports:
    - "${KAFKA_PORT}:9094"
    volumes:
      - ./scripts/create_topic.sh:/opt/bitnami/kafka/create_topic.sh
      - ${DATA_DIR}/components/kafka:/bitnami/kafka
    command: >
      bash -c "
      /opt/bitnami/scripts/kafka/run.sh & sleep 5; /opt/bitnami/kafka/create_topic.sh; wait
      "
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
       - TZ=Asia/Shanghai
       - KAFKA_CFG_NODE_ID=0
       - KAFKA_CFG_PROCESS_ROLES=controller,broker
       - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@<your_host>:9093
       - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
       - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://${DOCKER_BRIDGE_GATEWAY}:${KAFKA_PORT}
       - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
       - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
    networks:
      server:
        ipv4_address: ${KAFKA_NETWORK_ADDRESS}

Given this change, we should consider replacing Zookeeper to modernize our Kafka setup and possibly reduce resource overhead. One known alternative is Kraft. However, we are open to suggestions on other alternatives.

Requirements:

1. Identify potential alternatives to Zookeeper for Kafka.
2. Compare the pros and cons of each alternative.
3. Update Docker Compose file to reflect the new setup.

Potential Alternatives:

1. Kraft:
   • Pros:
     • Native to Kafka: Simplifies Kafka setup and reduces external dependencies.
     • Improved stability and fewer moving parts.
   • Cons:
     • Might require some changes to our current scripts and configuration.
     • Need to validate its stability and performance in our specific use case.

table：

Features	Nacos	Eureka	Consul	CoreDNS	Zookeeper
Consistency Protocol	CP+AP	AP	CP	—	CP
Health Check	TCP/HTTP/MYSQL/Client Beat	Client Beat	TCP/HTTP/gRPC/Cmd	—	Keep Alive
Load Balancing Strategy	Weight/metadata/Selector	Ribbon	Fabio	RoundRobin	—
Avalanche Protection	Yes	Yes	No	No	No
Auto Deregister	Supported	Supported	Supported	Not Supported	Supported
Protocol	HTTP/DNS	HTTP	HTTP/DNS	DNS	TCP
Listeners Support	Supported	Supported	Supported	Not Supported	Supported
Multi-datacenter	Supported	Supported	Supported	Not Supported	Not Supported
Cross Registry Sync	Supported	Not Supported	Supported	Not Supported	Not Supported
SpringCloud Integration	Supported	Supported	Supported	Not Supported	Supported
Dubbo Integration	Supported	Not Supported	Supported	Not Supported	Supported
K8S Integration	Supported	Not Supported	Supported	Supported	Not Supported
Programming Language	Java	Java	Go	Go	Java
Data Storage Backend	MySQL	In-memory	LevelDB	—	File System

中文:

功能特性	Nacos	Eureka	Consul	CoreDNS	Zookeeper
一致性协议	CP+AP	AP	CP	—	CP
健康检查	TCP/HTTP/MYSQL/Client Beat	Client Beat	TCP/HTTP/gRPC/Cmd	—	Keep Alive
负载均衡策略	权重/metadata/Selector	Ribbon	Fabio	RoundRobin	—
雪崩保护	有	有	无	无	无
自动注销实例	支持	支持	支持	不支持	支持
访问协议	HTTP/DNS	HTTP	HTTP/DNS	DNS	TCP
监听支持	支持	支持	支持	不支持	支持
多数据中心	支持	支持	支持	不支持	不支持
跨注册中心同步	支持	不支持	支持	不支持	不支持
SpringCloud集成	支持	支持	支持	不支持	支持
Dubbo集成	支持	不支持	支持	不支持	支持
K8S集成	支持	不支持	支持	支持	不支持
编程语言	Java	Java	Go	Go	Java
数据存储后端	MySQL	内存	LevelDB	—	文件系统

Note:

• "一致性协议" is translated to "Consistency Protocol".
• "健康检查" is translated to "Health Check".
• "负载均衡策略" is translated to "Load Balancing Strategy".
• "雪崩保护" is translated to "Avalanche Protection".
• "自动注销实例" is translated to "Auto Deregister".
• "访问协议" is translated to "Protocol".
• "监听支持" is translated to "Listeners Support".
• "多数据中心" is translated to "Multi-datacenter".
• "跨注册中心同步" is translated to "Cross Registry Sync".
• "SpringCloud集成" is translated to "SpringCloud Integration".
• "Dubbo集成" is translated to "Dubbo Integration".
• "K8S集成" is translated to "K8S Integration".

Action items:

1. Conduct a feasibility study on the proposed alternatives.
2. Test the selected alternative in a sandbox environment.
3. Update Docker Compose and related configurations.
4. Document the changes and migration steps.

[fi-ecavc]

Native to Kafka: Simplifies Kafka setup and reduces external dependencies.
Improved stability and fewer moving parts.

It is recommended to use the native kafka, fewer components means more robust.

[xiaobeiy]

It is also good to be able to reuse Kafka's kraft mechanism, since Kafka can use kraft, the consistency mechanism should be reliable. The main thing is whether it meets the consistency and performance requirements of OpenIM.

[kubbot]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[cubxxw]

/restart

[fuyuku]

请问可以提供一个去掉zookeeper的选项吗？

[kubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Can you please provide an option to remove zookeeper?

[kubbot]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[cubxxw]

@fuyuku

请问可以提供一个去掉zookeeper的选项吗？

We are implementing this part of the logic, which is necessary

[kubbot]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.