Skip to content
Sample KEDA deployment using a Go RabbitMQ consumer
Go Dockerfile
Branch: master
Clone or download
Latest commit 3e526b1 Jan 24, 2020
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
cmd initial Apr 24, 2019
deploy updating per new rabbitmq spec Sep 15, 2019
vendor/github.com/streadway/amqp
Dockerfile initial Apr 24, 2019
Gopkg.lock initial Apr 24, 2019
Gopkg.toml
README.md updating readme with helm 3 Jan 24, 2020

README.md

RabbitMQ consumer and sender

A simple docker container that will receive messages from a RabbitMQ queue and scale via KEDA. The reciever will receive a single message at a time (per instance), and sleep for 1 second to simulate performing work. When adding a massive amount of queue messages, KEDA will drive the container to scale out according to the event source (RabbitMQ).

Pre-requisites

Setup

This setup will go through creating a RabbitMQ queue on the cluster and deploying this consumer with the ScaledObject to scale via KEDA. If you already have RabbitMQ you can use your existing queues.

First you should clone the project:

git clone https://github.com/kedacore/sample-go-rabbitmq
cd sample-go-rabbitmq

Creating a RabbitMQ queue

Install Helm

Install RabbitMQ via Helm

Helm 3
helm install rabbitmq --set rabbitmq.username=user,rabbitmq.password=PASSWORD stable/rabbitmq

NOTE: if you are running the rabbitMQ image on KinD, you will run into permission issues unless you set volumePermissions.enabled=true

Use the following command if you are using KinD

helm install rabbitmq --set rabbitmq.username=user,rabbitmq.password=PASSWORD,volumePermissions.enabled=true stable/rabbitmq
Helm 2
helm install --name rabbitmq --set rabbitmq.username=user,rabbitmq.password=PASSWORD stable/rabbitmq

Wait for RabbitMQ to deploy

⚠️ Be sure to wait until the deployment has completed before continuing. ⚠️

kubectl get po

NAME         READY   STATUS    RESTARTS   AGE
rabbitmq-0   1/1     Running   0          3m3s

Deploying a RabbitMQ consumer

Deploy a consumer

kubectl apply -f deploy/deploy-consumer.yaml

Validate the consumer has deployed

kubectl get deploy

You should see rabbitmq-consumer deployment with 0 pods as there currently aren't any queue messages. It is scale to zero.

NAME                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
rabbitmq-consumer   0         0         0            0           3s

This consumer is set to consume one message per instance, sleep for 1 second, and then acknowledge completion of the message. This is used to simulate work. The ScaledObject included in the above deployment is set to scale to a minimum of 0 replicas on no events, and up to a maximum of 30 replicas on heavy events (optimizing for a queue length of 5 message per replica). After 30 seconds of no events the replicas will be scaled down (cooldown period). These settings can be changed on the ScaledObject as needed.

Publishing messages to the queue

Deploy the publisher job

The following job will publish 300 messages to the "hello" queue the deployment is listening to. As the queue builds up, KEDA will help the horizontal pod autoscaler add more and more pods until the queue is drained after about 2 minutes and up to 30 concurrent pods. You can modify the exact number of published messages in the deploy-publisher-job.yaml file.

kubectl apply -f deploy/deploy-publisher-job.yaml

Validate the deployment scales

kubectl get deploy -w

You can watch the pods spin up and start to process queue messages. As the message length continues to increase, more pods will be pro-actively added.

You can see the number of messages vs the target per pod as well:

kubectl get hpa

After the queue is empty and the specified cooldown period (a property of the ScaledObject, default of 300 seconds) the last replica will scale back down to zero.

Cleanup resources

kubectl delete job rabbitmq-publish
kubectl delete ScaledObject rabbitmq-consumer
kubectl delete deploy rabbitmq-consumer
helm delete rabbitmq
You can’t perform that action at this time.