How to configure docker-compose.yml for Kafka local development?

[THPubs]

Hi, I'm trying to setup Kafka in a docker container for local development. My docker-compose.yml looks as follows:

version: '3'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181"
    hostname: zookeeper
  kafka:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    ports:
      - "9092"
    hostname: kafka
    environment:
      KAFKA_CREATE_TOPICS: "UploadFile:1:1,GetFile:1:1,TrackUpload:1:1,GetEmailContent:1:1" # topic:partition:replicas
      KAFKA_ADVERTISED_HOST_NAME: kafka # docker-machine ip
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_PORT: 9092
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - "zookeeper"

Unfortunately my node app running on my localhost (not in docker) cannot connect to it. I used the url 'kafka:9092' and even 'localhost:9092'. Nothing works. Any idea what's happening?

[OneCricketeer]

Does the wiki not address your question?

https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

And duplicate #352

[THPubs]

@Cricket007 Thanks, it worked. But now I can't produce from inside a Consumer. Simply a Consumer can't produce and call another consumer. Any idea why?

[OneCricketeer]

I'm not sure I understand.

Are you trying to use Kafka Streams and sending one topic to another?

[THPubs]

Not streams. Let's say I have topic A... In its consumer, when the work is complete, it will produce to topic B... For example, topic A gets a file from a remote server, topic B, uploads the file to a different storage server.

[OneCricketeer]

I see no reason why that wouldn't work if you are able to individually produce and consume.

I would suggest using Streams API, though, instead if the intention is sending data between topics. For example, consume topic "filenames", map each String name to some FileContent object (doing a lookup against remote server), send output into next topic with to("file-content")

[THPubs]

Solved. Not an issue with docker 😄 Some code refining I did while working on this issue. But here's how I solved the connectivity issue: https://stackoverflow.com/questions/51919584/how-to-configure-docker-compose-yml-for-kafka-local-development

I exposed the kafka port to the local machine:

kafka:
  image: wurstmeister/kafka
  command: [start-kafka.sh]
  ports:
    - "9092:9092"

[OneCricketeer]

Glad to hear!

As mentioned in the comments, there on your post, though, I linked to "recommended" way, using the listener mapping

[THPubs]

@Cricket007 Great thanks! I'll have a look at that now.

[ghost]

For some reason, it does not work for me. I have tried putting localhost or a static IP address for one broker, but my application can't connect to it on localhost:9092, it just connects on kafka:9092.

services:
  app:
    build:
      dockerfile: Dockerfile
      context: .
    ports:
      - '8081:8080'
    links:
      - kakfa:Kafka
    networks:
      - test
    depends_on:
      - zookeeper
      - kakfa
    volumes:
    - ~/test:/test
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    restart: always
    ports:
      - "2181:2181"
    networks:
      - test
  kakfa:
    image: wurstmeister/kafka
    container_name: kafka
    restart: always
    links:
      - zookeeper:zookeeper
    ports:
      - "9092:9092"
    networks:
      - test
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CREATE_TOPICS: "WAMITA_POC:1:1"
    depends_on:
      - zookeeper
networks:
  test:
    driver: bridge

My app has a producer but it can't recognize the Kafka broker on localhost.

[OneCricketeer]

it just connects on kafka:9092.

Because that's the only address you have advertised... What's the problem?

[ghost]

it just connects on kafka:9092.

Because that's the only address you have advertised... What's the problem?

So, when I change my

KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092

to

KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

along with changing the Producer config like this:
properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

It does not work for me. I am doing this in Java. My app container throws an error saying that the broker might be disconnected.
Am I missing something ?
I can provide the logs from docker container if you want.

[OneCricketeer]

1. Hostname setting is deprecated and will be overwritten if you define listeners

2. you cannot advertise localhost - that would make any client connect to itself, and not the broker

[OneCricketeer]

Also, links is a deprecated Docker feature, so I'm not sure I understand where you are copying that compose section from.

Have you read the Docker networking documentation?

1. Your app must connect to kafka service
2. Remove host and post and define KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS

https://stackoverflow.com/a/51634499

[ghost]

Also, links is a deprecated Docker feature, so I'm not sure I understand where you are copying that compose section from.

Have you read the Docker networking documentation?

1. Your app must connect to kafka service
2. Remove host and post and define KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS

https://stackoverflow.com/a/51634499

Thanks for the help. I have been going through several stack overflow questions ranging back 4 years so that's why deprecated features are there. Let me look at it and implement. For the broker, I am going to try and put a internal IP address and see if it works after that.

[OneCricketeer]

try and put a internal IP address and see if it works

You should never use IPs in Docker networks. Always service names (aka DNS)

[ghost]

try and put a internal IP address and see if it works

You should never use IPs in Docker networks. Always service names (aka DNS)

I have been doing that and it works great. The requirement from the team came was to either use localhost or internal IP address but based on our discussion and your awesome feedback, I can go back to team and say this is not the best practice. I am still new to docker so went with the team's requirement. Again, thanks for all your help, I really appreciate it.

[OneCricketeer]

In order to use localhost, your Java app needs to run outside of any container, as I refer to in my Stackoverflow answer there, but this also means you would be running code on a physical broker server, which should never happen in production, anyway