This project is an implementation of a distributed system for processing IoT captor data using Kubernetes, Kafka and Elastic Cloud.
kubectl create namespace kafka
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
kubectl apply -f kafka-base/kafka-persistent.yaml -n kafka
kubectl apply -f kafka-base/kafka-bridge -n kafka
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.5.1/deploy/static/provider/cloud/deploy.yaml
kubectl get svc --all-namespaces
Next you should lookup for the CLUSTER-IP of the kafka-bridge service and paste it the conf file on nginx upstream.
kubectl create configmap nginx-config --from-file=ingress/nginx.conf -n kafka
kubectl apply -f ingress/nginx.yaml -kafka
kubectl apply -f ingress/ingress-bridge.yaml -n kafka
kubectl get ingress -n kafka
You can now see the public IP that you can use to interact with your kafka cluster.
To test, use curl:
curl -v <INGRESS_IP>/healthy
You should get a status response of 200.
kubectl create -f https://download.elastic.co/downloads/eck/2.5.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.5.0/operator.yaml -n elastic-system
kubectl create namespace elastic
kubectl create -f elastic/elasticsearch.yaml -n elastic
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html https://strimzi.io/blog/2020/05/07/camel-kafka-connectors/ https://strimzi.io/blog/2019/11/05/exposing-http-bridge/