This project is an example of web application using event-driven services and distributed data stores. The frontend is Node.js with Express, Handlebars, and React. The backend is Java with Vertx, Kafka, Cassandra, Elasticsearch, and Minio. The application runs on Kubernetes. Helm charts for deploying the application on Kubernetes are provided and have been tested on Minikube. The application can also run on Docker. Docker compose files for running and debugging the application are provided.
Here is the list of the required tools and versions:
- Docker 24 (or later version)
- Minikube 1.32
- Kubectl 1.29
- Helm 3.11
- Maven 3.9
- Java JDK Temurin 21 (or other compatible JDK)
- jq
- nc
- curl
There are few certificates which are required to run the application. For the purpose of this example we use self-signed certificates.
Generate secrets:
./scripts/secrets.sh
Add self-signed CA certificate to trusted certificates (for Mac only):
security -v add-trusted-cert -r trustRoot -k ~/Library/Keychains/login.keychain secrets/ca_cert.pem
Docker is the recommended environment for developing and testing the services.
Build Docker images required for running platform and testing services:
./scripts/build-images.sh
Start Nexus and Pact Broker:
./scripts/docker-pipeline.sh --start
Wait until Nexus is ready (please be patient):
./scripts/wait-for.sh --timeout=60 --command="docker logs --tail=100 nexus | grep 'Started Sonatype Nexus'"
Export Nexus password:
export NEXUS_PASSWORD=$(./scripts/get-nexus-password.sh)
Create Maven repositories:
./scripts/create-nexus-repositories.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD}
Build services:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet
See results of Pact tests:
open http://localhost:9292
Build services with a verbose output:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD}
Build services without tests:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-tests
Build services without tests and set version:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-tests --version=1.0.0
Build services without tests but keep version:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-tests --keep-version
Build only two services without tests and keep version:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-tests --keep-version --services="frontend authentication"
Build services and run tests, but skip Pact tests:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-pact-tests --skip-pact-verify
Build services and run tests, but skip integration tests:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-integration-tests
Run tests without building Docker images:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-images --keep-version
Run Pact tests only:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-images --keep-version --skip-integration-tests
Run Pact verify only:
./scripts/build-services.sh --nexus-host=localhost --nexus-port=8082 --nexus-username=admin --nexus-password=${NEXUS_PASSWORD} --quiet --skip-images --keep-version --skip-integration-tests --skip-pact-tests
Update dependencies (only if you know what you are doing):
./scripts/update-dependencies.sh
Tail pipeline containers:
docker logs -f --tail=-1 nexus
docker logs -f --tail=-1 postgres
docker logs -f --tail=-1 pact-server
Start platform:
./scripts/docker-platform.sh --start
Wait until Kafka is ready:
./scripts/wait-for.sh --timeout=60 --command="docker logs --tail=-1 kafka | grep 'started (kafka.server.KafkaServer)'"
Create Kafka topics:
./scripts/docker-create-kafka-topics.sh
Create Minio bucket:
./scripts/docker-create-minio-bucket.sh
Create a GitHub OAuth application like:
Homepage URL:
http://localhost:8000
Authorization Callback URL:
http://localhost:8000/v1/auth/callback
then export the application secrets:
export GITHUB_ACCOUNT_ID=your-account-id
export GITHUB_CLIENT_ID=your-client-id
export GITHUB_CLIENT_SECRET=your-client-secret
Start services:
./scripts/docker-services.sh --start
Open application:
open http://localhost:8000/browse/designs.html
Follow login link and log into the GitHub account when asked in order to access the administration console.
Open Minio console:
open http://localhost:9091
Login with user 'admin' and password 'password'.
Open Jaeger console:
open http://localhost:16686
Restart services with debug logging level:
./scripts/docker-services.sh --start --debug
Tail services containers:
docker logs -f --tail=-1 authentication
docker logs -f --tail=-1 accounts
docker logs -f --tail=-1 designs-query
docker logs -f --tail=-1 designs-command
docker logs -f --tail=-1 designs-aggregate
docker logs -f --tail=-1 designs-watch
docker logs -f --tail=-1 designs-render1
docker logs -f --tail=-1 designs-render2
docker logs -f --tail=-1 designs-render3
docker logs -f --tail=-1 designs-render4
docker logs -f --tail=-1 frontend
Tail platform containers:
docker logs -f --tail=-1 kafka
docker logs -f --tail=-1 zookeeper
docker logs -f --tail=-1 elasticsearch
docker logs -f --tail=-1 cassandra
docker logs -f --tail=-1 minio
docker logs -f --tail=-1 mysql
docker logs -f --tail=-1 nginx
docker logs -f --tail=-1 jaeger
Stop services:
./scripts/docker-services.sh --stop
Stop platform:
./scripts/docker-platform.sh --stop
Remove platform volumes:
./scripts/docker-platform.sh --destroy
Stop pipeline:
./scripts/docker-pipeline.sh --stop
Remove pipeline volumes:
./scripts/docker-pipeline.sh --destroy
Remove Docker images:
docker image rm -f $(docker image ls 'integration/*' -q)
docker image rm $(docker image ls -f dangling=true -q)
Minikube is the recommended environment for testing the deployment of the services.
Setup Minikube:
minikube start --vm-driver=hyperkit --cpus 8 --memory 49152m --disk-size 128g --kubernetes-version=v1.29.0
Create alias for kubectl (unless you have installed kubectl 1.29 already):
alias kubectl="minikube kubectl --"
Install Minikube addons:
./scripts/minikube-install-addons.sh
Deploy Certificate Manager:
./scripts/helm-install-cert-manager.sh
Install Hostpath Provisioner:
./scripts/helm-install-hostpath-provisioner.sh
Create namespaces:
./scripts/kube-create-namespaces.sh
Create volumes:
kubectl apply -f scripts/volumes.yaml
Load Docker images required for running platform (it might take a while):
./scripts/minikube-load-platform-images.sh
Load Docker images required for running services (it might take a while):
./scripts/minikube-load-services-images.sh
Create a GitHub OAuth application like:
Homepage URL:
https://<minikube-ip>
Authorization Callback URL:
https://<minikube-ip>/v1/auth/callback
where minikube-ip is the address of Minikube:
minikube ip
then export the application secrets:
export GITHUB_ACCOUNT_ID=your-account-id
export GITHUB_CLIENT_ID=your-client-id
export GITHUB_CLIENT_SECRET=your-client-secret
Create secrets:
./scripts/kube-create-nginx-secrets.sh
./scripts/kube-create-services-secrets.sh
Deploy Kube Prometheus Stack, Jaeger, and Fluent Bit:
./scripts/helm-install-observability-tools.sh
Deploy observability services:
./scripts/helm-install-observability-services.sh
Expose observability services:
./scripts/kube-expose-observability-services.sh
Deploy platform:
./scripts/helm-install-platform.sh
Expose platform:
./scripts/kube-expose-platform.sh
Wait until Kafka is ready:
./scripts/wait-for.sh --timeout=60 --command="kubectl -n platform logs --tail=5000 -l component=kafka | grep 'started (kafka.server.KafkaServer)'"
Create Kafka topics:
./scripts/kube-create-kafka-topics.sh
Wait until Cassandra is ready:
./scripts/wait-for.sh --timeout=60 --command="kubectl -n platform logs --tail=1000 -l component=cassandra | grep 'Starting listening for CQL'"
Create Cassandra tables:
./scripts/kube-create-cassandra-tables.sh
Wait until Elasticsearch is ready:
./scripts/wait-for.sh --timeout=60 --command="kubectl -n platform logs --tail=1000 -l component=elasticsearch | grep '\"message\":\"started'"
Create Elasticsearch indices:
./scripts/kube-create-elasticsearch-indices.sh
Create Minio bucket:
./scripts/kube-create-minio-bucket.sh
Create Kibana index pattern:
./scripts/kube-create-kibana-pattern.sh
Deploy services:
./scripts/helm-install-services.sh
Create observability resources (jaeger might fail if not ready. please retry):
./scripts/kube-create-observability-resources.sh
Expose observability resources:
./scripts/kube-expose-observability-resources.sh
Scale services:
kubectl -n services scale deployment designs-render --replicas=4
kubectl -n services scale deployment designs-query --replicas=2
kubectl -n services scale deployment designs-watch --replicas=2
kubectl -n services scale deployment designs-aggregate --replicas=2
kubectl -n services scale deployment frontend --replicas=2
Alternatively configure autoscaling:
kubectl -n services apply -f scripts/services-autoscaling.yaml
Scale platform:
kubectl -n platform scale deployment nginx --replicas=2
Alternatively configure autoscaling:
kubectl -n platform apply -f scripts/platform-autoscaling.yaml
Open application:
open https://$(minikube ip)/browse/designs.html
Follow login link and log into the GitHub account when asked in order to access the administration console.
Open Jaeger console:
open http://$(minikube ip):16686
Open Kibana console:
open http://$(minikube ip):5601
Open Prometheus console:
open http://$(minikube ip):9090
Open Grafana console:
open http://$(minikube ip):3000
Login with user 'admin' and password 'password'.
Open Minio console:
open http://$(minikube ip):9001
Login with user 'admin' and password 'password'.
Open Minikube dashboard:
minikube dashboard
Redeploy services with debug logging level:
./scripts/helm-install-services.sh --version=$(./scripts/get-version.sh) --debug
Tail services containers:
kubectl -n services logs -f --tail=-1 -l component=authentication
kubectl -n services logs -f --tail=-1 -l component=accounts
kubectl -n services logs -f --tail=-1 -l component=designs-query
kubectl -n services logs -f --tail=-1 -l component=designs-command
kubectl -n services logs -f --tail=-1 -l component=designs-aggregate
kubectl -n services logs -f --tail=-1 -l component=designs-watch
kubectl -n services logs -f --tail=-1 -l component=designs-render
kubectl -n services logs -f --tail=-1 -l component=frontend
Tail platform containers:
kubectl -n platform logs -f --tail=-1 -l component=kafka
kubectl -n platform logs -f --tail=-1 -l component=zookeeper
kubectl -n platform logs -f --tail=-1 -l component=elastichsearch
kubectl -n platform logs -f --tail=-1 -l component=cassandra
kubectl -n platform logs -f --tail=-1 -l component=minio
kubectl -n platform logs -f --tail=-1 -l component=mysql
kubectl -n platform logs -f --tail=-1 -l component=nginx
Uninstall services:
./scripts/helm-uninstall-services.sh
Uninstall platform:
./scripts/helm-uninstall-platform.sh
Uninstall observability:
./scripts/kube-delete-observability-resources.sh
./scripts/helm-uninstall-observability-services.sh
./scripts/helm-uninstall-observability-tools.sh
Uninstall Nexus and Pact Broker:
./scripts/helm-uninstall-pipeline.sh
Stop Minikube:
minikube stop
Delete Minikube (all data saved in hostpath volumes will be lost):
minikube delete
Although it is not recommended, it is possible to use Minikube to run the pipeline required to build the services.
Deploy Nexus and Pact Broker:
./scripts/helm-install-pipeline.sh
Wait until Nexus is ready:
./scripts/wait-for.sh --timeout=60 --command="kubectl -n pipeline logs --tail=100 -l component=nexus | grep 'Started Sonatype Nexus'"
Expose Nexus and Pact Broker:
./scripts/kube-expose-pipeline.sh
Export variables:
export NEXUS_HOST=$(minikube ip)
export NEXUS_PORT=8081
export NEXUS_USERNAME=admin
export NEXUS_PASSWORD=$(kubectl -n pipeline exec $(kubectl -n pipeline get pod -l component=nexus -o json | jq -r '.items[0].metadata.name') -c nexus -- cat /opt/sonatype/sonatype-work/nexus3/admin.password)
Create Maven repositories:
./scripts/create-nexus-repositories.sh --nexus-host=${NEXUS_HOST} --nexus-port=${NEXUS_PORT} --nexus-username=${NEXUS_USERNAME} --nexus-password=${NEXUS_PASSWORD}
Select Docker engine running on Minikube:
eval $(minikube docker-env)
Build Docker images (and skip tests):
./scripts/build-services.sh --nexus-host=${NEXUS_HOST} --nexus-port=${NEXUS_PORT} --nexus-username=${NEXUS_USERNAME} --nexus-password=${NEXUS_PASSWORD} --docker-host=$(minikube ip) --skip-tests
Please note that integration tests or pact tests are not supported when using the Docker engine running on Minikube.
Deploy Nexus and Pact Broker:
./scripts/helm-install-pipeline.sh
Wait until Nexus is ready:
./scripts/wait-for.sh --timeout=60 --command="kubectl -n pipeline logs --tail=100 -l component=nexus | grep 'Started Sonatype Nexus'"
Expose Nexus and Pact Broker:
./scripts/kube-expose-pipeline.sh
Export variables:
export PACTBROKER_HOST=$(minikube ip)
export PACTBROKER_PORT=9292
export NEXUS_HOST=$(minikube ip)
export NEXUS_PORT=8081
export NEXUS_USERNAME=admin
export NEXUS_PASSWORD=$(kubectl -n pipeline exec $(kubectl -n pipeline get pod -l component=nexus -o json | jq -r '.items[0].metadata.name') -c nexus -- cat /opt/sonatype/sonatype-work/nexus3/admin.password)
Create Maven repositories:
./scripts/create-nexus-repositories.sh --nexus-host=${NEXUS_HOST} --nexus-port=${NEXUS_PORT} --nexus-username=${NEXUS_USERNAME} --nexus-password=${NEXUS_PASSWORD}
Reset Docker environment variables to use local Docker engine:
eval $(env | grep DOCKER_ | cut -f 1 -d "=" - | awk '{print "unset "$1}')
Build images and run tests:
./scripts/build-services.sh --pactbroker-host=${PACTBROKER_HOST} --pactbroker-port=${PACTBROKER_PORT} --nexus-host=${NEXUS_HOST} --nexus-port=${NEXUS_PORT} --nexus-username=${NEXUS_USERNAME} --nexus-password=${NEXUS_PASSWORD} --quiet
See results of Pact tests:
open http://$(minikube ip):9292
Load Docker images into Minikube:
./scripts/minikube-load-images.sh --version=$(./scripts/get-version.sh)