Skip to content
Configures a minikube kafka cluster with 1 zookeeper node and 3 kafka brokers
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

kafka-cluster with minikube

Table of Contents

Minikube configuration

Suggested minikube start configuration:

minikube start --memory=6144 --cpus=4

Instructions to build the cluster

Run the commands in the following order:

kubectl apply -f 00-namespace/
kubectl apply -f 01-zookeeper/
kubectl apply -f 02-kafka/
kubectl apply -f 03-yahoo-kafka-manager/
kubectl apply -f 04-kafka-monitor/

# depends on creating a new cluster: kafka-ca2
# replace in all previous files s/kafka-ca1/kafka-ca2 and run steps: 00, 01, and 02.
kubectl apply -f 05-kafka-mirrormaker/

Configure kafka-manager

Open kafka-manager:

open $(minikube service -n kafka-ca1 kafka-manager --url)

Add new cluster, and use the following data for Cluster Zookeeper Hosts:


Monitor cluster resources

watch -n 1 kubectl -n kafka-ca1 get deployments
watch -n 1 kubectl -n kafka-ca1 get statefulsets
watch -n 1 kubectl -n kafka-ca1 get services
watch -n 1 kubectl -n kafka-ca1 get pods
watch -n 1 minikube service list --namespace kafka-ca1


kubectl -n kafka-ca1 exec kafka-0 -- tail -f /opt/kafka/logs/state-change.log
kubectl -n kafka-ca1 exec kafka-0 -- tail -f /opt/kafka/logs/server.log
kubectl -n kafka-ca1 exec kafka-0 -- tail -f /opt/kafka/logs/controller.log


Uses kafka-monitor 1.1.x:

Connect to your local running image to run the test commands:

kubectl -n kafka-ca1 exec kafka-monitor-{hash} -i -t bash
cd kafka-monitor-1.1.0

Monitor end-to-end a single cluster + JMX

Run the following:

KAFKA_JMX_OPTS=" -Djava.rmi.server.hostname= " \
  exec ./bin/ --broker-list kafka-0.kafka.kafka-ca1.svc.cluster.local:9092,kafka-1.kafka.kafka-ca1.svc.cluster.local:9092,kafka-2.kafka.kafka-ca1.svc.cluster.local:9092 --zookeeper zookeeper-service.kafka-ca1.svc.cluster.local:2181 \
  --topic testtopic

Monitor a single cluster + JMX using more detailed configuration

KAFKA_JMX_OPTS=" -Djava.rmi.server.hostname= " \
  exec ./bin/ config/

Monitor end-to-end multi-cluster + JMX

Run the following command:

KAFKA_JMX_OPTS=" -Djava.rmi.server.hostname= " \
  exec ./bin/ config/

Connect to JMX metrics

Forward the port:

kubectl -n kafka-ca1 port-forward kafka-monitor-{hash} 9999

Connect using jconsole/jvisualvm:

jconsole localhost:9999


Cluster view

Kafka manager view


  • 2018-09-28, Added monitor configuration using config maps
  • 2018-09-25, Added kafka-monitor recent image for kafka-monitor 1.1.x
  • 2018-09-24, Big refactor, changed per broker config to a single StatefulSet
    • Changed zookeeper image to use the one provided by dockerhub
You can’t perform that action at this time.