Distributed Task Queue

A production-ready distributed task queue system built with FastAPI, Redis, and RQ. Submit async jobs, track their progress, and scale workers independently.

Features

• ✅ Async Task Processing - Submit tasks to queue, execute asynchronously
• ✅ Real-time Status Tracking - Monitor task progress via REST API
• ✅ Multiple Task Types - Email, image processing, data export examples
• ✅ Worker Management - Scale workers independently from API
• ✅ Redis Integration - Persistent job queue with RQ
• ✅ React Dashboard - Submit tasks and monitor status
• ✅ Docker & Docker Compose - Easy local setup

Tech Stack

• Backend: FastAPI, Redis, RQ
• Frontend: React, Axios, Vite
• Infrastructure: Docker, Docker Compose

Quick Start

Prerequisites

• Docker & Docker Compose (OR Python 3.11+, Redis, Node 18+)

Using Docker Compose (Recommended)

docker-compose up

This starts:

• Redis on port 6379
• API on http://localhost:8000
• Worker processes jobs continuously
• Docs on http://localhost:8000/docs

Manual Setup

Backend:

cd backend
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt
uvicorn main:app --reload

Worker (separate terminal):

cd backend
python worker.py

Frontend:

cd frontend
npm install
npm run dev

Visit: http://localhost:3000

Demo & Screenshots

Dashboard Overview

The application provides an intuitive React-based dashboard for submitting tasks and monitoring their status:

Features shown:

• Task Selector - Choose from available task types (send_email, process_image, export_data)
• Arguments Editor - Input JSON arguments for selected task
• Submit Button - Queue the task for processing
• Active Tasks List - Real-time view of queued and processing tasks
• Status Tracking - Monitor task progress and results

API Endpoints

Submit Task

POST /tasks
Content-Type: application/json

{
  "task_name": "send_email",
  "arguments": {
    "recipient": "user@example.com",
    "subject": "Hello"
  }
}

Response: { "task_id": "abc123", "status": "queued", "task_name": "send_email" }

Get Task Status

GET /tasks/{task_id}

Response: {
  "task_id": "abc123",
  "status": "finished",
  "result": "Email sent to user@example.com: Hello",
  "error": null
}

List All Tasks

GET /tasks

Response: {
  "total": 5,
  "tasks": [
    { "task_id": "abc123", "status": "finished", "task_name": "send_email" },
    ...
  ]
}

Get Stats

GET /stats

Response: {
  "queued": 2,
  "finished": 10,
  "failed": 1,
  "total_jobs": 13
}

Available Tasks

1. send_email - Send email notification

   { "recipient": "user@example.com", "subject": "Subject Line" }

2. process_image - Process image file

   { "image_path": "/path/to/image.jpg", "operation": "resize" }

3. export_data - Export data to format

   { "format": "csv", "count": 1000 }

Project Structure

distributed-task-queue-app/
├── backend/
│   ├── main.py          # FastAPI app & task API
│   ├── worker.py        # RQ worker process
│   ├── requirements.txt  # Python dependencies
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── App.jsx      # React dashboard
│   │   ├── App.css
│   │   └── main.jsx
│   ├── package.json
│   ├── vite.config.js
│   └── index.html
├── docker-compose.yml
└── README.md

Key Concepts

Task Lifecycle

1. Queued - Task submitted, waiting for worker
2. Started - Worker processing the task
3. Finished - Task completed successfully
4. Failed - Task execution failed

Scaling

Add more workers to process jobs faster:

docker-compose up -d --scale worker=3

Testing

Submit a test task:

curl -X POST http://localhost:8000/tasks \
  -H "Content-Type: application/json" \
  -d '{"task_name": "send_email", "arguments": {"recipient": "test@example.com", "subject": "Test"}}'

Check status:

curl http://localhost:8000/tasks/{task_id}

Troubleshooting

Redis connection error?

• Ensure Redis is running: docker ps | grep redis
• Check REDIS_HOST and REDIS_PORT in .env

Task stuck in 'queued'?

• Worker process may have crashed
• Restart worker: python worker.py

Future Enhancements

• Database persistence (PostgreSQL)
• Task retries & dead letter queue
• Email notifications
• Admin dashboard
• Authentication & authorization

License

MIT