Kafka on Kubernetes

Repository consists of:

• Kafka all-in-one docker-compose script (based on cp-all-in-one)

• Kafka all-in-one helm charts (based on cp-helm-charts)

• Kafka helm charts for AWS MSK (based on cp-helm-charts)

---

Deployment

Docker-Compose

Docker-Compose is a local dev environment. Including a postgres module to generate testing data in the docker-compose network.

./data-mock.py is an example script to generate demo data for testing.

Kubernetes / all-in-one

Kubernetes charts is in ./helm/kafka, which is based on cp-helm-charts.

It can easily setup by helm:

cd helm

kubectl create ns kafka
helm install -f kafka/values-dev.yaml -n kafka kafka kafka

# patching existing helm
helm upgrade -f kafka/values-dev.yaml -n kafka kafka kafka

Kubernetes / MSK or Production

For production deployment on MSK, we should dump the values-dev.yaml to values-prod.yaml for another set of configurations.

cd helm

kubectl create ns kafka
helm install -f kafka/values-prod.yaml -n kafka kafka kafka

# patching existing helm
helm upgrade -f kafka/values-prod.yaml -n kafka kafka kafka

Misc

Helm Chart

Here is the Kafka stack components and the sub-chart templates for corresponding kubernetes components:

Component	Templates	Description
Kafdrop	charts/kafdrop	Minimal UI for kafka brokers
Kafka Connect	charts/kafka-connect	Kafka connect distributed server
Kafka Connect UI	charts/kafka-connect-ui	Minimal UI for kafka-connect
Kafka Rest	charts/kafka-rest	Kafka Rest Proxy, good for frontend development ?
Schema Registry	charts/schema-registry	Metadata server, good-to-have
Schema Registry UI	charts/schema-regsitry-ui	Minimal UI for schema registry
Brokers	charts/kafka-broker	For all-in-one setup
Zookeeper	charts/zookeeper	For all-in-one setup

JMX / Prometheus

Confluent Kafka components enable Java Management Extensions (JMX) for monitoring. By enabling ${component}.prometheus.jmx.enabled, components in kubernetes open port for exporting application metrics.

One can easily setup Prometheus monitoring in kubernetes with Prometheus Operator.