Below you'll find instructions to start playing around with a Kafka cluster deployed on Kubernets, by using Strimzi operator.
On top of that we'll use Jaeger to create&display tracing info. Those tracing information is created with producers, consumers and streams applications.
As underlying Kubernetes cluster, we'll use GKE, but you can use whichever K8s flavour you prefer.
I will also show you how to apply GitOps approach of bringing applications/configuration to your K8s cluster by using Flux v2 to setup Kafka and Topics. You will find the step-by-step instructions in the section about deploying Kafka cluster
- Strimzi Kafka & Jaeger tracing first steps
- create new project within GCP console, name: strimzi-and-jaeger-eval
- create new configuration for gcloud cli:
gcloud config configuration create strimzi-jaeger - set project:
gcloud config set project strimzi-and-jaeger-eval - authenticate with your Google account:
gcloud auth login
For this playground let's create a single-zone cluster, consisting of 2 nodes. Please adjust the machine-type according to your needs, the "e2-standard-2" ships with 2CPUs and 8GB Ram.
gcloud container clusters create strimzi-jaeger-eval \
--num-nodes=2 \
--zone=europe-central2-a \
--machine-type=e2-standard-2We'll use plain Strimzi yaml files, to avoid installing Helm additionally. Alternative would be to install Strimzi operator via Helm. The operator will be installed into namespace kafka-op and the Kafka cluster itself into namespace kafka-cluster.
kubectl create ns kafka-op
kubectl create ns kafka-cluster
kubectl create ns testing
wget https://github.com/strimzi/strimzi-kafka-operator/releases/download/0.22.1/strimzi-0.22.1.tar.gz
tar -xvzf strimzi-0.22.1.tar.gz
cd strimzi-0.22.1sed -i 's/namespace: .*/namespace: kafka-op/' install/cluster-operator/*RoleBinding*.yamlOpen file 060-Deployment-strimzi-cluster-operator.yaml (under install/cluster-operator folder) and locate the STRIMZI_NAMESPACE environment variable to specify our namespace, as shown below:
env:
- name: STRIMZI_NAMESPACE
value: kafka-cluster,testingCreate rolebindings for our desired namespaces, kafka-cluster and testing:
kubectl apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n kafka-cluster
kubectl apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n kafka-cluster
kubectl apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n kafka-cluster
kubectl apply -f install/cluster-operator/020-RoleBinding-strimzi-cluster-operator.yaml -n testing
kubectl apply -f install/cluster-operator/031-RoleBinding-strimzi-cluster-operator-entity-operator-delegation.yaml -n testing
kubectl apply -f install/cluster-operator/032-RoleBinding-strimzi-cluster-operator-topic-operator-delegation.yaml -n testingkubectl apply -f install/cluster-operator -n kafka-op
kubectl create -f install/cluster-operator/040-Crd-kafka.yaml # because this file is too large for kubectl apply ......and verify it: kubectl get deployments -n kafka-op
until you see the deployment up and running, like e.g.:
NAME READY UP-TO-DATE AVAILABLE AGE
strimzi-cluster-operator 1/1 1 1 51sNext, let's setup a 3-node Kafka cluster with a single-node Zookeeper. Kafka will be reachable via port 9092 without authentication and encryption, as well as on port 9093 tls encrypted. Additionally we have an external facing listener on port 9094, which is created via a NodePort service.
To actually deploying the Kafka cluster I'll provide two different approaches. One is applying the plain yaml manifests by executing kubectl commands, and the other approach is to use GitOps-tool Flux, so that Kafka specs are being applied to the K8s cluster as soon as you push them to Git.
- manual kafka setup by running kubectl commands
- kafka setup, the GitOps way by using Flux
If you are done with setting up Kafka, you can list the available topics.
The property --bootstrap-server you have to set, according to the cluster you want to connect to, and it needs to resolve a service name in your desired namespace. To list those services, just run:
# for the 'prod' cluster:
kubectl get svc -n kafka-cluster | grep kafka-bootstrap
# for the 'testing' cluster:
kubectl get svc -n testing | grep kafka-bootstrapthen take this service name + namespace to create the value for --bootstrap-server property, in the form of <servicename>.<namespace>:9092 (because we want to quickly test via the unathenticated port 9092)
kubectl run kafka-producer -ti \
--image=strimzi/kafka:0.20.0-rc1-kafka-2.6.0 \
--rm=true \
--restart=Never \
-- bin/kafka-topics.sh --bootstrap-server testing-strimzi-cluster-kafka-bootstrap.testing:9092 --list...in -testing- Kafka cluster.
Start a Producer:
kubectl run kafka-producer -ti \
--image=strimzi/kafka:latest-kafka-2.4.0 \
--rm=true --restart=Never \
-- bin/kafka-console-producer.sh --broker-list testing-strimzi-cluster-kafka-bootstrap.testing:9092 --topic my-first-topicand in another terminal, start a Consumer:
kubectl run kafka-consumer -ti \
--image=strimzi/kafka:latest-kafka-2.4.0 \
--rm=true --restart=Never \
-- bin/kafka-console-consumer.sh --bootstrap-server testing-strimzi-cluster-kafka-bootstrap.testing:9092 --topic my-first-topic --from-beginningInstall Operator Lifecycle Manager (OLM), a tool to help manage the Operators running on your cluster.
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.18.1/install.sh | bash -s v0.18.1
Install the operator by running the following command:
kubectl create -f https://operatorhub.io/install/jaeger.yaml
This Operator will be installed in the "operators" namespace and will be usable from all namespaces in the cluster.
After install, watch your operator come up using next command: kubectl get csv -n operators
To use it, checkout the custom resource definitions (CRDs) introduced by this operator to start using it.
Now that the Jaeger Operator is in place, we can create our Jaeger CRD. In this demo we are using the All-in-one bundle of Jaeger, which includes all components within one binary. It will be installed in the default namespace:
cat <<EOF | kubectl apply -f -
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: my-jaeger
spec:
strategy: allInOne
allInOne:
image: jaegertracing/all-in-one:latest
options:
log-level: debug
storage:
type: memory
options:
memory:
max-traces: 100000
ingress:
enabled: false
agent:
strategy: DaemonSet
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
EOFPort forwarding from your local box to Jaeger-UI:kubectl port-forward service/my-jaeger-query 8080:16686
Now open a browser and navigate to http://localhost:8080
Let's deploy a sample application, consisting of a producer, consumer and a Kafka-streams application in between.
Let's distinguish also here between the manual approach and the GitOps approach.
- create two topics:
kubectl apply -f ./jaeger-example/1-topics.yaml - create Producer:
kubectl apply -f ./jaeger-example/2-producer.yaml - create Streams app:
kubectl apply -f ./jaeger-example/3-kafka-streams.yaml - create Consumer:
kubectl apply -f ./jaeger-example/4-consumer.yaml
This option requires that you finished the Flux setup & config as described in the chapter about deploying Kafka cluster the GitOps way. Means, you should have your Git repo flux-kafka-demo already cloned to your workstation.
The only thing we have to do, is to register a new "app" in Flux, to observe and apply the subfolder jaeger-example of our base Git repo strimzi-jaeger-eval.
- open a terminal and ensure you are within the directory of repo flux-kafka-demo
- register the "jaeger-example" in Flux and push the change
flux create kustomization jaeger-example \
--source=strimzi-jaeger-eval-main \
--path="./jaeger-example" \
--prune=true \
--validation=client \
--interval=5m \
--export > ./my-flux/jaeger-example-kustomization.yaml
# now push the change to Git and let the resources being created
git add -A && git commit -m "created Kustomization for Jaeger demo"
git push- check what Flux is doing
watch flux get kustomizationsWait a minute, or two ..... until you see the "jaeger-example" in state READY :
NAME READY MESSAGE REVISION SUSPENDED
flux-system True Applied revision: main/488748d800d3f5dc03daf23b1b5f87a8d239f195 main/488748d800d3f5dc03daf23b1b5f87a8d239f195 False
jaeger-example True Applied revision: main/b34f21917de4d392a9ff4b84b1bc88d152e44e23 main/b34f21917de4d392a9ff4b84b1bc88d152e44e23 False
strimzi-jaeger-eval True Applied revision: main/b34f21917de4d392a9ff4b84b1bc88d152e44e23 main/b34f21917de4d392a9ff4b84b1bc88d152e44e23 FalseIf you now check the Jaeger UI, searching for service "java-kafka-producer" and click on "Show traces", you'll see a list of traces which have been generated.
You can click on one of those traces to further dive into the details, e.g.:
Now that the base Kafka cluster and Jaeger is setup, let's enrich the platform by adding Kafka Connect....and a Connector to read data from Postgresql to Kafka. You'll find the instructions/docu on page KafkaConnect-Debezium-Postgresql