Example Kafka-KEDA App based on Python

Kafka App based on Python

Example app to read from a given Kafka topic and processing the message. Based on Python.

1. Install Kafka and KEDA on your cluster (e.g., via ./hack/requirements.sh).

2. Build and push the app's container image to a registry (change docker.io/aliok as you like):

   # Build and push the docker image
   docker build --platform linux/amd64 -t docker.io/aliok/python-keda .
   docker push docker.io/aliok/python-keda

3. Apply manifests (and optionally adjust the image name in config/kubernetes.yml before):

   kubectl apply -f config/kubernetes.yml

4. Test if messages get handled/logged:

   1. Produce some Kafka messages in the topic

      kubectl -n kafka run kafka-producer --rm -ti --image=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic

   2. Check the app logs

      kubectl logs -l app=python-app

Scale app with KEDA

You can scale the app with KEDA to distribute the message processing on multiple instances.

1. Apply the ScaledObject

   kubectl apply -f config/keda-scaler.yaml

2. Produce some load on the Kafka topic

   kubectl -n kafka run kafka-producer --rm -ti --image=quay.io/strimzi/kafka:0.47.0-kafka-4.0.0 --rm=true --restart=Never -- bin/kafka-producer-perf-test.sh --topic my-topic --num-records 100000 --record-size 10 --throughput -1 --producer-props bootstrap.servers=my-cluster-kafka-bootstrap:9092

3. Check how the python-app gets upscaled by KEDA:

   kubectl get po -l app=python-app -w