ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)

[YuriyGavrilov]

Describe the bug
There is an error during datahub docker QuickStart

2023-11-02 20:36:42 kafka.common.InconsistentClusterIdException: The Cluster ID Xzl3J7H5R9KFPqhJwJjpww doesn't match stored clusterId Some(SWoJzy2JR922qyhMit2IzA) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.

To Reproduce

1. run 'daatahub docker QuickStart'
2. Wait 'until broker can't start'
3. Go to docker logs
4. See error

2023-11-02 20:36:42 [2023-11-02 17:36:42,044] INFO Client environment:java.class.path=/usr/bin/../share/java/kafka/plexus-utils-3.3.0.jar:/usr/bin/../share/java/kafka/connect-transforms-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.13.4.jar:/usr/bin/../share/java/kafka/jersey-server-2.34.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.86.Final.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/kafka-storage-api-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/connect-json-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.8.4.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.13.4.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/connect-api-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/hk2-api-2.6.1.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.13.4.jar:/usr/bin/../share/java/kafka/javassist-3.27.0-GA.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/kafka-group-coordinator-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/kafka_2.13-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/hk2-locator-2.6.1.jar:/usr/bin/../share/java/kafka/kafka-shell-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.13-1.0.2.jar:/usr/bin/../share/java/kafka/connect-mirror-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/kafka-storage-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/rocksdbjni-7.1.2.jar:/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.6.1.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.34.jar:/usr/bin/../share/java/kafka/kafka-metadata-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.13-2.6.0.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.6.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.86.Final.jar:/usr/bin/../share/java/kafka/connect-runtime-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/netty-transport-classes-epoll-4.1.86.Final.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.34.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.13.4.jar:/usr/bin/../share/java/kafka/scala-reflect-2.13.10.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.13-2.13.4.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/zookeeper-3.6.4.jar:/usr/bin/../share/java/kafka/jersey-client-2.34.jar:/usr/bin/../share/java/kafka/jose4j-0.7.9.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.13-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/maven-artifact-3.8.4.jar:/usr/bin/../share/java/kafka/kafka-raft-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/jersey-common-2.34.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.3.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/lz4-java-1.8.0.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.6.1.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.86.Final.jar:/usr/bin/../share/java/kafka/reload4j-1.2.19.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.86.Final.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/kafka-clients-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/scala-logging_2.13-3.9.4.jar:/usr/bin/../share/java/kafka/javax.annotation-api-1.3.2.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jetty-util-ajax-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/jakarta.validation-api-2.0.2.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.86.Final.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.13.4.jar:/usr/bin/../share/java/kafka/slf4j-reload4j-1.7.36.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.13.4.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.36.jar:/usr/bin/../share/java/kafka/zstd-jni-1.5.2-1.jar:/usr/bin/../share/java/kafka/jackson-core-2.13.4.jar:/usr/bin/../share/java/kafka/jline-3.21.0.jar:/usr/bin/../share/java/kafka/netty-common-4.1.86.Final.jar:/usr/bin/../share/java/kafka/hk2-utils-2.6.1.jar:/usr/bin/../share/java/kafka/kafka-tools-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.6.4.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.48.v20220622.jar:/usr/bin/../share/java/kafka/audience-annotations-0.13.0.jar:/usr/bin/../share/java/kafka/trogdor-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/metrics-core-4.1.12.1.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.86.Final.jar:/usr/bin/../share/java/kafka/scala-library-2.13.10.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.2.jar:/usr/bin/../share/java/kafka/jackson-databind-2.13.4.2.jar:/usr/bin/../share/java/kafka/kafka-server-common-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.3.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/swagger-annotations-2.2.0.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-7.4.0-ccs.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.34.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.86.Final.jar:/usr/bin/../share/java/confluent-telemetry/* (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:os.version=6.4.16-linuxkit (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:user.name=appuser (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:user.home=/home/appuser (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:user.dir=/home/appuser (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:os.memory.free=241MB (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:os.memory.max=256MB (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,045] INFO Client environment:os.memory.total=256MB (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,051] INFO Initiating client connection, connectString=zookeeper:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@5d5f10b2 (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,060] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2023-11-02 20:36:42 [2023-11-02 17:36:42,070] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
2023-11-02 20:36:42 [2023-11-02 17:36:42,075] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2023-11-02 20:36:42 [2023-11-02 17:36:42,080] INFO Opening socket connection to server zookeeper/172.25.0.2:2181. (org.apache.zookeeper.ClientCnxn)
2023-11-02 20:36:42 [2023-11-02 17:36:42,092] INFO Socket connection established, initiating session, client: /172.25.0.7:59094, server: zookeeper/172.25.0.2:2181 (org.apache.zookeeper.ClientCnxn)
2023-11-02 20:36:42 [2023-11-02 17:36:42,103] INFO Session establishment complete on server zookeeper/172.25.0.2:2181, session id = 0x100063596e9000d, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
2023-11-02 20:36:42 [2023-11-02 17:36:42,108] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
2023-11-02 20:36:42 [2023-11-02 17:36:42,452] INFO Cluster ID = Xzl3J7H5R9KFPqhJwJjpww (kafka.server.KafkaServer)
2023-11-02 20:36:42 [2023-11-02 17:36:42,462] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
2023-11-02 20:36:42 kafka.common.InconsistentClusterIdException: The Cluster ID Xzl3J7H5R9KFPqhJwJjpww doesn't match stored clusterId Some(SWoJzy2JR922qyhMit2IzA) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
2023-11-02 20:36:42     at kafka.server.KafkaServer.startup(KafkaServer.scala:234)
2023-11-02 20:36:42     at kafka.Kafka$.main(Kafka.scala:115)
2023-11-02 20:36:42     at kafka.Kafka.main(Kafka.scala)
2023-11-02 20:36:42 [2023-11-02 17:36:42,468] INFO shutting down (kafka.server.KafkaServer)
2023-11-02 20:36:42 [2023-11-02 17:36:42,472] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
2023-11-02 20:36:42 [2023-11-02 17:36:42,588] INFO Session: 0x100063596e9000d closed (org.apache.zookeeper.ZooKeeper)
2023-11-02 20:36:42 [2023-11-02 17:36:42,588] INFO EventThread shut down for session: 0x100063596e9000d (org.apache.zookeeper.ClientCnxn)
2023-11-02 20:36:42 [2023-11-02 17:36:42,592] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
2023-11-02 20:36:42 [2023-11-02 17:36:42,605] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
2023-11-02 20:36:42 [2023-11-02 17:36:42,605] INFO shut down completed (kafka.server.KafkaServer)
2023-11-02 20:36:42 [2023-11-02 17:36:42,605] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
2023-11-02 20:36:42 kafka.common.InconsistentClusterIdException: The Cluster ID Xzl3J7H5R9KFPqhJwJjpww doesn't match stored clusterId Some(SWoJzy2JR922qyhMit2IzA) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
2023-11-02 20:36:42     at kafka.server.KafkaServer.startup(KafkaServer.scala:234)
2023-11-02 20:36:42     at kafka.Kafka$.main(Kafka.scala:115)
2023-11-02 20:36:42     at kafka.Kafka.main(Kafka.scala)
2023-11-02 20:36:42 [2023-11-02 17:36:42,607] INFO shutting down (kafka.server.KafkaServer)

Expected behavior
Run successful

Screenshots

Desktop (please complete the following information):

• OS: [e.g. Mac OS X]
• Browser [e.g. chrome, safari]
• Version [e.g. 0.12]

Additional context

[ibrentlam]

same result on Ubuntu 22

[hsheth2]

This should've been fixed by #9148 - is this issue still happening on a fresh quickstart?

[github-actions]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[sirosc]

I'm still getting this on MacOS. Have been unable to successfully run the app yet. Images were downloaded as part of this run.

❯ datahub --version
acryl-datahub, version 0.12.1.1
❯ datahub docker quickstart

Results as above.

Docker logs:

2023-12-19 16:55:33 [2023-12-20 00:55:33,832] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
2023-12-19 16:55:33 [2023-12-20 00:55:33,945] INFO Cluster ID = EOMZ1IoTT8KwPfqZEMxe2g (kafka.server.KafkaServer)
2023-12-19 16:55:33 [2023-12-20 00:55:33,948] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
2023-12-19 16:55:33 kafka.common.InconsistentClusterIdException: The Cluster ID EOMZ1IoTT8KwPfqZEMxe2g doesn't match stored clusterId Some(EmfiZZf7SUSK-yft0Euz3w) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
2023-12-19 16:55:33     at kafka.server.KafkaServer.startup(KafkaServer.scala:234)
2023-12-19 16:55:33     at kafka.Kafka$.main(Kafka.scala:115)
2023-12-19 16:55:33     at kafka.Kafka.main(Kafka.scala)
2023-12-19 16:55:33 [2023-12-20 00:55:33,949] INFO shutting down (kafka.server.KafkaServer)
2023-12-19 16:55:33 [2023-12-20 00:55:33,950] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
2023-12-19 16:55:34 [2023-12-20 00:55:34,053] INFO Session: 0x1000757e42e0013 closed (org.apache.zookeeper.ZooKeeper)
2023-12-19 16:55:34 [2023-12-20 00:55:34,053] INFO EventThread shut down for session: 0x1000757e42e0013 (org.apache.zookeeper.ClientCnxn)
2023-12-19 16:55:34 [2023-12-20 00:55:34,053] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
2023-12-19 16:55:34 [2023-12-20 00:55:34,056] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
2023-12-19 16:55:34 [2023-12-20 00:55:34,056] INFO shut down completed (kafka.server.KafkaServer)
2023-12-19 16:55:34 [2023-12-20 00:55:34,056] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
2023-12-19 16:55:34 kafka.common.InconsistentClusterIdException: The Cluster ID EOMZ1IoTT8KwPfqZEMxe2g doesn't match stored clusterId Some(EmfiZZf7SUSK-yft0Euz3w) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
2023-12-19 16:55:34     at kafka.server.KafkaServer.startup(KafkaServer.scala:234)
2023-12-19 16:55:34     at kafka.Kafka$.main(Kafka.scala:115)
2023-12-19 16:55:34     at kafka.Kafka.main(Kafka.scala)
2023-12-19 16:55:34 [2023-12-20 00:55:34,056] INFO shutting down (kafka.server.KafkaServer)

[hsheth2]

@sirosc Can you try running a datahub docker nuke first, and then rerunning quickstart? If you have data stored in your instance, you can use a backup and restore (https://datahubproject.io/docs/next/quickstart#back-up-datahub).

It'd also be enough to remove the broker and zookeeper containers and remove their underlying volumes.

[sirosc]

THanks. That worked. I thought deleting the images within docker desktop would be enough, but clearly not.

[hsheth2]

@sirosc yup it was some of the zookeeper/kafka volumes that were the root cause of the problem.