Notice of migration

This repo now lives at: https://codeberg.org/jessebot/kafka-k8s-example

Let's play with Kafka

Just a quick example of how to set up a test Kuberentes (k8s) environment with KIND, and then using ArgoCD as our Continuous Delivery to test deploy a Kafka helm chart. Uses Hashicorp's Vault for secret storage :)

Tech stack and Dependencies

Package Managers

• brew - Missing package manager for Mac (also supports Linux)
  • Also got some scripts for Debian distros
• helm2 installs k8s apps (mostly a bunch of k8s yamls)
• Docker - for the containers
• Kubernetes - THE CLUSTER we use to scale containers :3
  • KIND - Tool to spin up mini k8s cluster locally
• Vault - Open source secret management from Hashicorp
• ArgoCD - Continuous Delivery for k8s, from within k8s
• Argo CLI tool - this will let you use ArgoCD without the web interface
• ArgoCD Vault Plugin - ArgoCD with Vault
  • Kustomize - kubernetes native config managment to install argo with vault
• Kafka - Handles real-time data feeds at scale

Installation for Pre-Reqs

To get your enviornment set up, check out this repo to learn how to install kind and ArgoCD :)

Getting Started with Kafka

Kafka will also be installed via helm, but this time with Argo. You'll want to have a git repo for kafka for this next part. In this case, I'm using this repo, but only to keep this managable as a tutorial.

Add the Bitnami helm repo:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

Make sure you have the charts/kafka/Chart.yaml for this next part. Then update your chart:

$ helm dep update charts/kafka/
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading kafka from repo https://charts.bitnami.com/bitnami
Deleting outdated charts

That should put you to the point where you can now push this repo, and have argo sync it.

Prepare kafka repo

For sake of simplicity and future automation, you can use the github cli to create a repo:

$ gh repo create argo-and-kafka-example

Add kafka to ArgoCD

Create an SSH key pair, and then paste the public key into the deploy key section of your GitHub repo.

Make sure you have argocd cli installed, and then you can run (with ~/id_rsa_argo_deploy being your newly created SSH key pair from the previous step):

$ argocd repo add git@github.com:jessebot/argo-kafka-example/charts/kafka --insecure-ignore-host-key --ssh-private-key-path ~/id_rsa_argo_deploy