🚀 Real-Time Fraud Detection ML Platform (Kafka + FastAPI + MLflow + Prometheus + Grafana)

A production-style machine learning system that detects fraudulent transactions in real-time using Kafka streaming, FastAPI inference service, MLflow experiment tracking, and Prometheus + Grafana monitoring.

This project demonstrates end-to-end ML system design used in companies like Stripe, Uber, and PayPal.

---

🧠 Problem Statement

Fraud detection systems must process thousands of transactions per second and instantly detect suspicious activity.

This project simulates a real-world fraud detection pipeline where:

1. Transactions are generated in real time
2. Data is streamed via Kafka
3. A trained ML model predicts fraud probability
4. Predictions are served via FastAPI
5. Metrics are monitored using Prometheus & Grafana

---

🏗 System Architecture

Transaction Producer
        │
        ▼
   Kafka Topic
        │
        ▼
Kafka Consumer
        │
        ▼
Feature Processing
        │
        ▼
FastAPI ML Model
        │
        ▼
Fraud Prediction
        │
        ▼
Monitoring & Tracking
   ├── MLflow
   ├── Prometheus
   └── Grafana

---

⚙️ Tech Stack

Machine Learning

• Python
• Scikit-learn
• Pandas

Streaming

• Apache Kafka
• Kafka Producer
• Kafka Consumer

API / Inference

• FastAPI
• Uvicorn

MLOps

• MLflow
• Docker
• Docker Compose

Monitoring

• Prometheus
• Grafana

---

📂 Project Structure

fraud-ml-platform-v2
│
├── api
│   └── app.py
│
├── training
│   └── train.py
│
├── streaming
│   ├── producer.py
│   └── consumer.py
│
├── monitoring
│   └── prometheus.yml
│
├── model
│   └── model.pkl
│
├── docker-compose.yml
└── README.md

---

🚀 Running the Project

1️⃣ Clone Repository

git clone https://github.com/yourusername/fraud-ml-platform.git
cd fraud-ml-platform

---

2️⃣ Train the Model

python training/train.py

This logs the experiment to MLflow.

---

3️⃣ Start the Platform

docker compose up -d

This launches:

• Kafka
• Zookeeper
• FastAPI
• Producer
• Consumer
• MLflow
• Prometheus
• Grafana

---

🔗 Service Links

Service	URL
FastAPI	http://localhost:8000
Swagger API Docs	http://localhost:8000/docs
MLflow	http://localhost:5000
Prometheus	http://localhost:9090
Grafana	http://localhost:3000

Grafana login:

username: admin
password: admin

---

🧪 Test Fraud Prediction

curl -X POST http://localhost:8000/predict \
-H "Content-Type: application/json" \
-d '{"amount":1000,"device_type":1,"location":2,"merchant":3}'

Example response:

{
  "prediction": 0,
  "fraud_probability": 0.03
}

---

📊 Monitoring

Metrics are exposed via:

http://localhost:8000/metrics

Prometheus scrapes metrics and Grafana visualizes:

• Prediction request count
• API latency
• System health

---

🧠 ML Model

The model uses:

• Transaction amount
• Device type
• Location risk
• Merchant category

Model used:

RandomForestClassifier

---

💡 Key Features

✔ Real-time streaming with Kafka ✔ Low-latency ML inference with FastAPI ✔ Experiment tracking with MLflow ✔ Monitoring with Prometheus & Grafana ✔ Containerized microservices using Docker

---

📈 Future Improvements

• Feature Store (Feast)
• Model Drift Detection
• Kubernetes deployment
• CI/CD pipeline
• AWS cloud deployment

---

👨‍💻 Author

Vimal Kumar

LinkedIn: https://www.linkedin.com/in/vimal-k-952745203/

GitHub: https://github.com/vim12345