Skip to content

Latest commit

 

History

History
156 lines (109 loc) · 4 KB

kafka.adoc

File metadata and controls

156 lines (109 loc) · 4 KB
Raw

Core Kafka installation and operations

Create Kafka broker and topic

Add AMQ Streams operator to your OpenShift environment.

Create a Broker with metrics enabled:

  • consumer lag

  • consumer offsets

oc new-project my-kafka
oc apply -f k8s/01-kafka-metrics.yaml
oc apply -f k8s/02-kafkatopic.yaml

Enabling user workload monitoring

Note
 This demo relies on the cluster-admin role.

The following command:

  • adds a ConfigMap named cluster-monitoring-config in the openshift-monitoring namespace

  • creates a ClusterRoleBinding which grants permissions to grafana-serviceaccount

oc apply -f k8s/03-cluster-monitor.yaml

The ConfigMap triggers the deployment of Prometheus and Thanos: after a while it’s possible to check that the prometheus-operator, prometheus-user-workload and thanos-ruler-user-workload pods are running in the openshift-user-workload-monitoring project.

oc -n openshift-user-workload-monitoring get pod

Enable: - monitoring for kafka resources - prometheus rules

oc apply -f k8s/04-pod-monitor.yaml -n my-kafka
oc apply -f k8s/05-prometheus-rules.yaml -n my-kafka

Grafana uses the Thanos Querier which works as Prometheus proxy to scrape the metrics.

The following script:

  • creates a service account for Grafana with an access token

  • creates a datasource configuration for Grafana to scrape the metrics. It points to Thanos Querier which acts as a Prometheus proxy and uses the previous created access token to gain access.

k8s/07-create-datasource.sh

Deploy Grafana and expose it:

oc apply -f k8s/08-grafana.yaml
oc create route edge --service=grafana --namespace=my-kafka

Login with the default credentials (admin/admin) and then change the password.

Load the dashboard definitions from grafana-dashboards folder:

  • strimzi-kafka.json

  • strimzi-kafka-exporter.json

Install the consumer and producer applications

mvn -f kafka-consumer/pom.xml package -Dquarkus.kubernetes.deploy=true -DskipTests
mvn -f kafka-producer/pom.xml package -Dquarkus.kubernetes.deploy=true -DskipTests

Further information about the applications:

Demo routines

AMQ Streams High Availability

  1. Show consumer logs

    oc logs --tail=20 -f --selector="app.kubernetes.io/name=kafka-consumer"

  2. Show producer logs

    oc logs --tail=20 -f --selector="app.kubernetes.io/name=kafka-producer"

  3. Show the partitions distribution

    oc exec -it my-cluster-kafka-0 -- bin/kafka-topics.sh \
                                --bootstrap-server my-cluster-kafka-bootstrap:9092 \
                                --describe --topic event

  4. Test the Kafka’s resilience and consistency by forcefully shutting down one of brokers' pod.

    Tip
    		 Use the following command: oc delete --force pod <pod-name>
    Important
    		 Watching at the consumer log you should notice that it temporarily stops processing some messages (missing messages). This is expected! In fact, Kafka promotes consistency over availability, so until a new partition leader is elected you cannot write or consume messages on that partition. Eventually, the new leader will become available and the missing messages will be caught up.

  5. Show again the topic distribution on the cluster members

  6. Show the dashboard

Appendix

Refresh Grafana token

Prometheus access token lasts 7 days.

To refresh it:

oc delete serviceaccounts grafana-serviceaccount
oc delete configmap grafana-config
k8s/07-create-datasource.sh
oc delete pod --selector name=grafana

Full Grafana clean up

Delete Grafana deployment:

oc delete all --selector application=kafka-monitor
oc delete configmap grafana-config

Deploy Grafana running the scripts 07 and 08.