Datahub fails to restart using docker compose (dependency failed to start)

[remisalmon]

Describe the bug

When using docker compose, datahub starts and stop normally but is not able to restart and throws a dependency failed to start: container broker exited (1) error.

To Reproduce
Steps to reproduce the behavior:

1. Run docker-compose -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml up
2. Stop with control+C or docker-compose down
3. Re-run command from step 1
4. Docker compose fails with dependency failed to start: container broker exited (1)

Doing a docker-compose down -v works but loses the data in the volume.

Expected behavior

Datahub to restart without error without having to remove the volume first.

Logs

broker                  | [2023-10-18 21:09:38,392] INFO Initiating client connection, connectString=zookeeper:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@74c79fa2 (org.apache.zookeeper.ZooKeeper)
broker                  | [2023-10-18 21:09:38,395] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
broker                  | [2023-10-18 21:09:38,399] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
broker                  | [2023-10-18 21:09:38,401] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
broker                  | [2023-10-18 21:09:38,402] INFO Opening socket connection to server zookeeper/172.19.0.3:2181. (org.apache.zookeeper.ClientCnxn)
broker                  | [2023-10-18 21:09:38,405] INFO Socket connection established, initiating session, client: /172.19.0.6:50376, server: zookeeper/172.19.0.3:2181 (org.apache.zookeeper.ClientCnxn)
broker                  | [2023-10-18 21:09:38,410] INFO Session establishment complete on server zookeeper/172.19.0.3:2181, session id = 0x1000021967b0001, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
broker                  | [2023-10-18 21:09:38,412] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
broker                  | [2023-10-18 21:09:38,565] INFO Cluster ID = hxwdM_xMQlu3tOUboh3Oew (kafka.server.KafkaServer)
broker                  | [2023-10-18 21:09:38,569] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
broker                  | kafka.common.InconsistentClusterIdException: The Cluster ID hxwdM_xMQlu3tOUboh3Oew doesn't match stored clusterId Some(UOnWEv1aRIWZ4TIOR46aCw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
broker                  | 	at kafka.server.KafkaServer.startup(KafkaServer.scala:234)
broker                  | 	at kafka.Kafka$.main(Kafka.scala:115)
broker                  | 	at kafka.Kafka.main(Kafka.scala)
broker                  | [2023-10-18 21:09:38,571] INFO shutting down (kafka.server.KafkaServer)
broker                  | [2023-10-18 21:09:38,572] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
broker                  | [2023-10-18 21:09:38,675] INFO Session: 0x1000021967b0001 closed (org.apache.zookeeper.ZooKeeper)
broker                  | [2023-10-18 21:09:38,675] INFO EventThread shut down for session: 0x1000021967b0001 (org.apache.zookeeper.ClientCnxn)
broker                  | [2023-10-18 21:09:38,676] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
broker                  | [2023-10-18 21:09:38,678] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
broker                  | [2023-10-18 21:09:38,678] INFO shut down completed (kafka.server.KafkaServer)
broker                  | [2023-10-18 21:09:38,678] ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$)
broker                  | kafka.common.InconsistentClusterIdException: The Cluster ID hxwdM_xMQlu3tOUboh3Oew doesn't match stored clusterId Some(UOnWEv1aRIWZ4TIOR46aCw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
broker                  | 	at kafka.server.KafkaServer.startup(KafkaServer.scala:234)
broker                  | 	at kafka.Kafka$.main(Kafka.scala:115)
broker                  | 	at kafka.Kafka.main(Kafka.scala)
broker                  | [2023-10-18 21:09:38,681] INFO shutting down (kafka.server.KafkaServer)
broker exited with code 1
zookeeper               | [2023-10-18 21:09:39,435] INFO Processing srvr command from /172.19.0.3:39002 (org.apache.zookeeper.server.NIOServerCnxn)
dependency failed to start: container broker exited (1)

Desktop (please complete the following information):

• OS: macOS 13.6
• Datahub: v0.11.0
• Docker compose: v2.22.0-desktop.2

[hsheth2]

@remisalmon have a fix for this coming here #9148

[remisalmon]

@remisalmon have a fix for this coming here #9148

Just tested and this worked, thank you!