Real-time Request Processing System with Kafka, gRPC, FastAPI, and React

Table of Contents

• Project Overview
• Key Features
• System Architecture
• Technologies Used
• Getting Started
  • Prerequisites
  • Installation
  • Running the System
• API Documentation
• Workflow Explanation
• Monitoring and Debugging
• Future Enhancements
• Contributing
• License

Project Overview

This project demonstrates a robust request processing system that showcases:

• Message Queuing with Apache Kafka for reliable request handling
• gRPC for efficient inter-service communication
• FastAPI as the backend service framework
• React with Tailwind CSS for the modern frontend interface

The system processes requests sequentially, showing the complete lifecycle from frontend submission to processed solution delivery.

Key Features

✅ End-to-end request processing pipeline
✅ Real-time status updates with polling mechanism
✅ Sequential processing demonstration (one request at a time)
✅ gRPC integration for efficient processing
✅ Kafka message queue for reliable delivery
✅ Modern UI with React and Tailwind CSS
✅ Containerized deployment with Docker
✅ Comprehensive logging at each processing stage

System Architecture

graph LR
    A[Frontend] -->|HTTP POST| B[FastAPI]
    B -->|Produce| C[(Kafka Requests Topic)]
    C --> D[gRPC Service]
    D -->|Produce| E[(Kafka Solutions Topic)]
    E --> B
    B -->|Polling| A

Loading

1. Frontend: React application with Tailwind CSS styling
2. API Layer: FastAPI backend with REST endpoints
3. Message Queue: Kafka with two topics (requests and solutions)
4. Processing Service: gRPC service for actual computation

Technologies Used

Backend:

• Python 3.9
• FastAPI
• gRPC
• Kafka-Python

Frontend:

• React 18
• Tailwind CSS 3
• Axios for HTTP requests

Infrastructure:

• Docker
• Docker Compose
• Apache Kafka
• Zookeeper

Getting Started

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+
• Node.js 16+ (for frontend development)
• Python 3.9+ (for backend development)

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/request-processing-system.git
   cd request-processing-system

2. Set up environment variables:

   cp backend/.env.example backend/.env
   cp frontend/.env.example frontend/.env

3. Build and start the services:

   docker-compose up --build

Running the System

After starting the services:

1. Access the frontend at: http://localhost:3000
2. FastAPI docs available at: http://localhost:8000/docs
3. gRPC service runs on: localhost:50051

API Documentation

REST API Endpoints

Submit a Request

POST /api/requests
Content-Type: application/json

{
  "data": 42,
  "problem_type": "square"
}

Check Solution Status

GET /api/solutions/{request_id}

gRPC Service Methods

ProcessRequest

rpc ProcessRequest (Request) returns (Response);

message Request {
  string request_id = 1;
  double input_value = 2;
  string problem_type = 3;
}

message Response {
  string request_id = 1;
  double result = 2;
  string status = 3;
}

Workflow Explanation

1. Request Submission

   • User submits request via React frontend
   • FastAPI receives request and produces Kafka message
   • Request gets unique ID for tracking

2. Queue Processing

   • gRPC service consumes messages one at a time
   • Each request processed sequentially (5s delay)
   • Results published to solutions topic

3. Solution Delivery

   • Frontend polls for solution every 3 seconds
   • FastAPI checks in-memory store for solutions
   • UI updates when solution becomes available

Monitoring and Debugging

View Kafka Topics:

docker-compose exec kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic requests \
  --from-beginning

Check gRPC Logs:

docker-compose logs grpc-service

Backend Logs:

docker-compose logs backend

Future Enhancements

• Add authentication layer
• Implement persistent storage for solutions
• Add more complex processing examples
• Implement WebSockets for push notifications
• Add Prometheus monitoring
• Implement dead-letter queue for failed requests

Contributing

1. Fork the project
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

---

Project Maintainer: [ZahinRakin]
Contact: bsse1447@iit.du.ac.bd
Project Link: https://github.com/ZahinRakin/Notification-System