Skip to content

Event Driven Microservice architecture with kafka and kubernetes

Notifications You must be signed in to change notification settings

nileshkant/kafka-microservice

Repository files navigation

Project Name

This project is an example of an event-driven microservices architecture using Docker, Kubernetes, Kafka, Express.js, TypeScript, and PostgreSQL with TypeORM. It contains two services (service-1 and service-2) and demonstrates inter-service communication via Kafka topics.

Prerequisites

  • Docker
  • Kubernetes (Minikube or a managed Kubernetes cluster)
  • Skaffold (optional, for development with hot reloading)
  • An ingress controller (optional, for URL-based routing)
  • kafkacat (optional, for testing Kafka topics)

Setting Up the Project

  1. Clone the repository:

shCopy code

git clone https://github.com/your-repo-url/project-name.git cd project-name

  1. Set up the Docker environment to use the Minikube Docker daemon:

shCopy code

minikube start --driver=<your_preferred_driver> minikube docker-env | Invoke-Expression

(On macOS or Linux, use eval $(minikube docker-env) instead of minikube docker-env | Invoke-Expression.)

  1. Build and run the services in Docker:

shCopy code

docker-compose up --build

  1. (Optional) Install and enable the ingress controller for your Kubernetes cluster:

shCopy code

minikube addons enable ingress

  1. Deploy the services to your Kubernetes cluster:

shCopy code

kubectl apply -f k8s/

  1. (Optional) Add a mapping to your /etc/hosts file (or C:\Windows\System32\drivers\etc\hosts on Windows) to route requests to your ingress domain:

shCopy code

echo "$(minikube ip) myapp.local" | sudo tee -a /etc/hosts

Running the Project in Development Mode

  1. Set up your Docker environment to use the Minikube Docker daemon:

shCopy code

minikube start --driver=<your_preferred_driver> minikube docker-env | Invoke-Expression

(On macOS or Linux, use eval $(minikube docker-env) instead of minikube docker-env | Invoke-Expression.)

  1. Run Skaffold in development mode:

shCopy code

skaffold dev

Skaffold will watch for code changes, rebuild the Docker images, and redeploy your application to the Minikube cluster automatically.

Testing Kafka Topics

To test the Kafka topic communication, you can use the kafkacat utility. Install kafkacat following the instructions for your platform: https://github.com/edenhill/kafkacat

  1. Produce a message to a Kafka topic:

shCopy code

kafkacat -P -b localhost:9092 -t my-topic

Type a message, press Enter, and then press Ctrl+C to exit.

  1. Consume messages from a Kafka topic:

shCopy code

kafkacat -C -b localhost:9092 -t my-topic

Press Ctrl+C to exit.

License

MIT License

Acknowledgments

Troubleshooting

If you encounter issues while setting up or running the project, consider the following troubleshooting tips:

  1. Verify that all required software is installed and running correctly, including Docker, Kubernetes, Skaffold, and the ingress controller.

  2. Check the logs of your services and the ingress controller for any errors or issues:

shCopy code

kubectl logs <pod_name> -f

Replace <pod_name> with the name of the pod you want to check.

  1. Ensure your Kubernetes resources are correctly configured, including services, deployments, and the ingress resource.

  2. If you're using Minikube, ensure that your system is properly configured to use the Minikube Docker daemon and that the cluster is running.

  3. If you're using an ingress controller, make sure the ingress resource is properly configured, and the domain is mapped to the correct IP address in your /etc/hosts file.

  4. If you encounter issues with Kafka topics, double-check your Kafka and Zookeeper configurations, and ensure that the required ports are exposed and accessible.

  5. For development mode with hot reloading, ensure that Skaffold is set up correctly and that your project structure is compatible with the skaffold.yaml file.

If you continue to experience issues, feel free to open an issue in the project repository or seek help from the community.