Run

This script assumes you already have the Azure CLI and kubectl already installed.

If you wish to use an existing cluster, comment out the below lines in install.sh:

az group create -n $RG_NAME -l $LOCATION
az aks create -n $CLUSTER_NAME -g $RG_NAME -l $LOCATION --generate-ssh-keys
az aks get-credentials -n $CLUSTER_NAME -g $RG_NAME --overwrite-existing

Be sure to change the CLUSTER_NAME, RG_NAME and LOCATION in install.sh to corresponding values from your deployment.

export CLUSTER_NAME="kafka-k8-cluster"
export RG_NAME="kafka-k8"
export LOCATION="westus2"

Running the script

Run ./install.sh

This script will install a Kafka instance with three brokers and three Zookeeper instances on your AKS cluster, with external access via a loadbalancer. Currently it only supports PLAINTEXT authentication.

Test

Once the installation is complete, you can test the Kafka cluster.

To see the public IPs:

kubectl get svc

To list the topics on this Kafka cluster:

kubectl exec kafkaclient -- ./bin/kafka-topics.sh --zookeeper kafka-zookeeper:2181 --list

To create a topic test on the Kafka cluster with 4 partitions:

kubectl exec kafkaclient -- ./bin/kafka-topics.sh --create --zookeeper kafka-zookeeper:2181 --replication-factor 1 --partitions 4 --topic test

Using Kafkacat

You can easily test functionality with kafkacat

To install kafkacat:

sudo apt-get install kafkacat

To list topics, brokers and other metadata:

kafkacat -b <publicIP> -L

To create producer from command-line:

Enter below command. Just type (or paste) your message into the terminal and hit enter after that. Once you’re done just hit control+C

kafkacat -b </publicIP>/:31090 -t new_topic -P

To create consumers from command-line:

Enter below command. Once you’re done just hit control+C

kafkacat -b </publicIP>/:31090 -t new_topic -C