docker-compose file does not contain volumes spec

[lkoniecz]

Description
https://github.com/confluentinc/cp-all-in-one/blob/7.2.1-post/cp-all-in-one-cloud/docker-compose.yml referred in official tutorial https://github.com/confluentinc/cp-all-in-one/blob/7.2.1-post/cp-all-in-one-cloud/docker-compose.yml does not contain volumes spec.

Troubleshooting
docker-compose up followed by docker-compose down clears all the data.

Environment

• any

[OneCricketeer]

AFAIK, no documentation says a volume will be mounted. You're welcome to add them on your own

[HarlemSquirrel]

Where should a volume or volumes be mounted when using the https://github.com/confluentinc/cp-all-in-one/blob/7.5.0-post/cp-all-in-one-kraft/docker-compose.yml kraft all-in-one compose configuration? I would like to persist topics and messages across container restarts for development.

[OneCricketeer]

The KAFKA_LOG_DIRS variable defines a folder for topic storage. Could you try mounting a volume there?

[HarlemSquirrel]

@OneCricketeer that doesn't seem to work

Setting

services:
  kafka:
    profiles:
      - kafka
    image: confluentinc/cp-kafka:7.6.1
    hostname: kafka
    restart: unless-stopped
    container_name: kafka
    ports:
      - "127.0.0.2:29092:29092"
      - "127.0.0.2:9997:9997"
    environment:
      JMX_PORT: 9997
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_NODE_ID: 1
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://kafka:9092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:29092
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LOG_DIRS: /tmp/kraft-combined-logs
      CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: true
    volumes:
      - kafka_logs:/tmp/kraft-combined-logs

volumes:
  kafka_logs:

results in logging this

kafka  | [2024-06-25 01:47:26,973] ERROR Failed to write meta.properties due to (kafka.server.BrokerMetadataCheckpoint) java.io.FileNotFoundException: /tmp/kraft-combined-logs/meta.properties.tmp (Permission denied) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187) at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:228) at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:226) at kafka.tools.StorageTool$.$anonfun$formatCommand$2(StorageTool.scala:434) at kafka.tools.StorageTool$.$anonfun$formatCommand$2$adapted(StorageTool.scala:425) at scala.collection.immutable.List.foreach(List.scala:333) at kafka.tools.StorageTool$.formatCommand(StorageTool.scala:425) at kafka.tools.StorageTool$.main(StorageTool.scala:81) at kafka.tools.StorageTool.main(StorageTool.scala) Exception in thread "main" java.io.FileNotFoundException: /tmp/kraft-combined-logs/meta.properties.tmp (Permission denied) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187) at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:228) at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:226) at kafka.tools.StorageTool$.$anonfun$formatCommand$2(StorageTool.scala:434) at kafka.tools.StorageTool$.$anonfun$formatCommand$2$adapted(StorageTool.scala:425) at scala.collection.immutable.List.foreach(List.scala:333) at kafka.tools.StorageTool$.formatCommand(StorageTool.scala:425) at kafka.tools.StorageTool$.main(StorageTool.scala:81) at kafka.tools.StorageTool.main(StorageTool.scala)

[HarlemSquirrel]

I think I got it working by mounting a local directory instead of using a docker volume since I couldn't get the volume to have the correct permissions for some reason.

I'm not sure if /var/lib/kafka is needed but I did see that mentioned in this doc for zookeeper so I added it. I see in the logs that it's checking for write permission on that but I don't see it change when I create topics or messages.

    volumes:
      - kafka_data:/var/lib/kafka
      - ./tmp/kraft-combined-logs:/tmp/kraft-combined-logs

volumes:
  kafka_data: