Vagrant Docker Kafka

⭐️ Local, automated and isolated environment for learning Kafka.

This project automates the creation of an Apache Kafka local environment, meant to be used for learning and testing purposes.

It creates a VM using Vagrant, that has Docker and Docker Compose installed.

Zookeeper and Kafka are installed and executed inside Docker containers, inside the VM.

With this, you don't need to install Docker or Kafka on your local computer.
You just need to install Vagrant and VirtualBox.

All the images and containers are created in an isolated environment that can be deleted at any time.

Architecture

Vagrant creates an Ubuntu VM that installs Docker, pulls Docker images from DockerHub, and runs containers with their corresponding port mappings.

Kafka and some of its related components will be accessible to the host machine through some ports:

Component	Port	Description
Kafka	9092	Kafka service (for connecting with CLI or SDKs)
Kafka UI	8080	Web UI (recommended over other Web UIs)
AKHQ	8081	Web UI

The Web UIs will be accessible to the host's web browser through the specified ports.

The automation process is specified using the following files:

1. Vagrantfile: Tells Vagrant how to create and configure the VM
2. docker-compose.yml: Tells Docker Compose which and how containers should be executed

The following diagram shows the architecture:

Prerequisites

• Install VirtualBox
• Install Vagrant

Verify installation

Note

Execute these steps only if it's the first time that you use Vagrant with VirtualBox.
If not, you can skip them. They only serve to test the Vagrant + VirtualBox installation.
If Vagrant and VirtualBox are installed and configured correctly, then the environment will work fine (it has already been tested, and is repeatable).

Check that the vagrant executable was added correctly to the PATH variable:

vagrant version

Check that vagrant is able to create a VM:

mkdir test-vagrant
cd test-vagrant
vagrant init ubuntu/jammy64
vagrant up
vagrant ssh
pwd
exit
vagrant destroy --force
cd ..
rm -rf test-vagrant

Warning

If the following error appears after executing vagrant up:
No usable default provider could be found for your system.

1. Verify that VirtualBox was installed correctly
2. Obtain more info about the error:

   vagrant up --provider=virtualbox
   

Warning

If the following error appears after executing vagrant up:
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005)

• Reinstall VirtualBox

Warning

If Vagrant gets stuck on the following line after executing vagrant up:
SSH auth method: private key

• Windows users: Open cmd as admin and execute:

  bcdedit /set hypervisorlaunchtype off
  

  This disables Hyper-V.

Warning

For other issues:

• Visit the VirtualBox Common Issues section of the Vagrant documentation

Kafka topics

This example will create 2 kafka topics:

• topic1: 1 partition, 1 replica
• topic2: 1 partition, 1 replica

This is specified in the docker-compose.yml file, in the following line:

KAFKA_CREATE_TOPICS: "topic1:1:1,topic2:2:1"

You can modify this by specifying a comma-separated list of <topic>:<partitions>:<replicaction-factor>.

These are the topics created at the creation of the Kafka cluster, but you can create more topics after that manually (the way to create them is explained below).

Steps to run the environment

All the vagrant commands must be executed in the host machine from the folder that contains the Vagrantfile (in this case, the project root folder).

Note

For Windows users:
If Vagrant doesn't show any output in the stdout for a Vagrant command after some time, press the Enter key or right click in the console window. See this post for more info about this problem.

1. Start the VM [host]

This will:

1. Install Docker inside the VM
2. Pull the Docker images from DockerHub
3. Run the containers All with the corresponding port mappings.

Note

Docker images/containers will only be downloaded/executed if the Docker Compose up line in the Vagrantfile is uncommented.

vagrant up

The following messages should appear at the end of the command stdout:

Creating vagrant-zookeeper-1 ... done
Creating vagrant-kafka-1     ... done

2. Check the status of the VM [host]

vagrant status

3. Connect to the VM [host]

This connection is done via SSH.

vagrant ssh

Tip

Some interesting commands to execute inside the VM:

Commmand	Description
free -h	Display amount of free and used memory in the VM
docker stats	Display a live stream of container(s) resource usage statistics. Useful to monitor Docker containers memory usage.
docker container ls --all	List all Docker containers (running or not). If both containers specify "Up" in the status column, everything is running fine.
docker logs <containerid>	Fetch the logs of a container. Really useful to troubleshoot Kafka or Zookeeper servers, or to simply see what's going on.
docker top <containerid>	Display the running processes of a container
docker exec -it <containerid> <command>	Run a command in a running container (in interactive mode)
docker images	List images
docker version	Show the Docker version information
docker info	Display system-wide information
netstat -tulpn | grep LISTEN	Display network connections (listening TCP or UDP). Useful to check that Kafka (9092) and Zookeeper (2181) ports are listening.

4. Create tmux session [vm]

The VM welcome message shows the command for connecting to the tmux session.

The tmux window is divided in panes with the following layout:

┌───────────┬───────┐
│ ZOOKEEPER │ KAFKA │
└───────────┴───────┘

Kafka commands

Inside the Kafka container:

Description	Commmand
List topics	/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
Describe topics	/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe
Create topic	/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --replication-factor <replicas> --partitions <partitions> --topic <topic>
Delete topic	/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic <topic>
Create Kafka Producer	/opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic <topic>
Create Kafka Consumer (from latest)	/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic <topic>
Create Kafka Consumer (from beginning)	/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic <topic> --from-beginning

Check connectivity from Kafka to Zookeeper

The following commands, executed inside the Kafka container, allow to check the connectivity with Zookeeper:

apk update

ping zookeeper

apk add bind-tools
dig zookeeper

nc -z -v zookeeper 2181

apk add openssl
openssl s_client -connect zookeeper:2181

5. Access Kafka UI in your web browser [host]

Write the following URL in your web browser: localhost:8080.

6. Access AKHQ in your web browser [host]

Write the following URL in your web browser: localhost:8081.

(Optional) Detach from tmux session [vm]

Ctrl-B + d

(Optional) Attach again to tmux session [vm]

tmux list-sessions
tmux attach-session -t <session-name>

If the tmux session is deleted (for example, using Ctrl-D several times), you may need to restart the tmux server in order to be able to connect again to the tmux session:

tmux kill-server
tmux attach-session -t <session-name>

(Optional) Remove and start containers to clean data [vm]

Note

Only if containers where executed using Docker Compose.

This is useful if you want to clean the data inside the containers.

cd /vagrant
docker compose rm --stop --force
docker compose up -d

(Optional) Connect to one of the Docker containers [vm]

Obtain the name of the container you want to connect to:

docker container ls --all

The name is the last column.

Execute the bash command in that container to connect to it:

docker exec -it <container-name> bash

Stop the VM (keeps data) [host]

Stopping the VM will stop the Docker containers and turn off the VM.
All the data is persisted inside the containers, and a subsequent turn on of the VM (and the containers) will have access to that data.

Stop the VM:

vagrant halt

Check the status of the VM:

vagrant status

Start the VM and the containers again:

vagrant up

Destroy the VM (removes data) [host]

Destroying the VM will remove all the VM data, and therefore, the containers inside it.

This should be the option used if you do not want to keep the data, and you want to have a "clean" environment in the next turn on of the VM (because the VM and the containers will be created from scratch).

vagrant destroy

Additional notes

Whenever you change the docker-compose.yml file, you need to run vagrant reload to redefine the Vagrant box.

References

• Vagrant
• Docker
• Vagrant Docker provisioner
• Kafka
• Kafka Docker image
• Kafka UI
• AKHQ
• Vagrant commands
• Docker commands
• Docker Compose commands