⚡ TaskFlow — Distributed Task Queue & Real-Time Event System

A production-grade async backend system using FastAPI + Redis + RQ + WebSockets with a professional React dashboard.

---

🏗️ Architecture

User (Browser)
    │
    ├── POST /api/tasks        → FastAPI (submits job to Redis Queue)
    ├── GET  /api/tasks/{id}   → FastAPI (polls task status)
    └── WS   /ws/{client_id}  → WebSocket (real-time push notifications)
                                      ↑
                              RQ Worker (processes jobs, notifies via Redis PubSub)
                                      ↑
                              Redis (message broker + job store)

🚀 Features

• Distributed Task Queue via RQ (Redis Queue)
• Real-Time WebSocket Notifications — browser gets notified the instant a job finishes
• Multiple Task Types: Email Campaigns, Image Processing, Data Reports, File Exports
• Job Progress Tracking with percentage updates
• Retry & Failure Handling with exponential backoff
• Professional React Dashboard with live updates, job history, analytics
• Redis Pub/Sub bridge between worker and WebSocket server

---

🛠️ Tech Stack

Layer	Technology
API Server	FastAPI (Python)
Task Queue	RQ (Redis Queue)
Message Broker	Redis 7
Real-Time	WebSockets (FastAPI)
Frontend	React 18 + Vite
Styling	Tailwind CSS
Dev Tools	Docker Compose (optional)

---

📦 Prerequisites

• Python 3.11+
• Node.js 18+
• Redis (Windows: use Memurai or WSL2)

---

⚡ Quick Start (Windows / VS Code)

1. Clone & Setup

git clone https://github.com/YOUR_USERNAME/taskflow.git
cd taskflow

2. Backend Setup

# Create virtual environment
python -m venv venv
venv\Scripts\activate          # Windows
# source venv/bin/activate     # Mac/Linux

# Install dependencies
pip install -r requirements.txt

3. Start Redis (Windows)

Option A — Memurai (recommended for Windows): Download from https://www.memurai.com/ and install. It runs as a Windows service automatically.

Option B — WSL2:

wsl sudo service redis-server start

Option C — Docker:

docker run -d -p 6379:6379 redis:7-alpine

4. Run the System (3 terminals in VS Code)

Terminal 1 — API Server:

venv\Scripts\activate
uvicorn backend.api.main:app --reload --port 8000

Terminal 2 — RQ Worker:

venv\Scripts\activate
python -m backend.worker.run_worker

Terminal 3 — Frontend:

cd frontend
npm install
npm run dev

5. Open Dashboard

Visit http://localhost:5173

---

📁 Project Structure

taskflow/
├── backend/
│   ├── api/
│   │   ├── main.py          # FastAPI app + WebSocket server
│   │   ├── routes.py        # REST API endpoints
│   │   └── schemas.py       # Pydantic models
│   ├── worker/
│   │   ├── run_worker.py    # RQ worker entrypoint
│   │   └── tasks.py         # Task definitions (email, image, etc.)
│   └── core/
│       ├── config.py        # Settings & environment
│       ├── redis_client.py  # Redis connection
│       └── ws_manager.py    # WebSocket connection manager
├── frontend/
│   ├── src/
│   │   ├── components/      # React UI components
│   │   ├── hooks/           # Custom hooks (useWebSocket, useTasks)
│   │   └── pages/           # Dashboard, History pages
│   └── package.json
├── requirements.txt
├── .env.example
└── README.md

---

🔌 API Reference

Submit a Task

POST /api/tasks
Content-Type: application/json

{
  "task_type": "email_campaign",
  "payload": {
    "recipients": 5000,
    "template": "newsletter_v2"
  },
  "priority": "high"
}

Get Task Status

GET /api/tasks/{task_id}

List All Tasks

GET /api/tasks?status=pending&limit=20

WebSocket Connection

const ws = new WebSocket(`ws://localhost:8000/ws/${clientId}`);
ws.onmessage = (event) => {
  const { task_id, status, progress, result } = JSON.parse(event.data);
};

---

📊 Task Types

Type	Description	Simulated Duration
email_campaign	Bulk email sending simulation	10–30s
image_processing	Resize/compress image batch	5–15s
data_report	Generate analytics report	8–20s
file_export	Export large dataset to CSV	6–18s
notification_blast	Send push notifications	4–12s

---

🧪 Testing the System

# Submit a test task via curl
curl -X POST http://localhost:8000/api/tasks \
  -H "Content-Type: application/json" \
  -d '{"task_type": "email_campaign", "payload": {"recipients": 1000}}'

# Watch the terminal — worker picks it up, updates Redis, WebSocket pushes to browser

---

🐳 Docker Compose (Optional)

docker-compose up --build

All services start automatically (Redis, API, Worker, Frontend).

---

💡 Key Concepts Demonstrated

1. Message Broker Pattern — Redis decouples API from Worker
2. Pub/Sub Bridge — Worker publishes events; WebSocket server subscribes and pushes to clients
3. Horizontal Scalability — Multiple workers can consume from the same queue
4. Graceful Error Handling — Failed jobs are retried with backoff, then moved to failed queue
5. Real-Time UX — No polling; WebSocket delivers instant updates

---

📝 License

MIT