SmartQueue

SmartQueue is a distributed task processing portfolio project built to demonstrate backend engineering, queue architecture, worker orchestration, observability, and deployment readiness.

It presents a recruiter-ready SaaS operations dashboard backed by a FastAPI control plane, Redis priority queues, Python workers, and a small C++ compute simulation worker. The core demo runs locally with Docker Compose, while Kubernetes, Terraform, Prometheus, and Grafana files show how the system would evolve toward production.

Architecture

flowchart LR
  UI["React + TypeScript Dashboard"] --> API["FastAPI Control Plane"]
  API --> REDIS["Redis Queue Store"]
  REDIS --> PY["Python Worker Pool"]
  PY --> REDIS
  CPP["C++ Compute Simulator"] -. "CPU-heavy task model" .-> PY
  API --> METRICS["/metrics"]
  METRICS --> PROM["Prometheus"]
  PROM --> GRAF["Grafana"]

Loading

Tech Stack

• Frontend: React, TypeScript, Tailwind CSS, Vite
• Backend: FastAPI, Pydantic, structured JSON logging
• Queue: Redis hashes, sorted sets, streams, and priority lists
• Workers: Python queue worker plus C++ compute simulation folder
• Observability: Prometheus-compatible /metrics, Grafana dashboard JSON
• Delivery: Docker, Docker Compose, Kubernetes manifests, Terraform scaffold

Screenshots

Add hosted screenshots after deployment:

• docs/screenshots/dashboard.png
• docs/screenshots/jobs.png
• docs/screenshots/workers.png
• docs/screenshots/metrics.png
• docs/screenshots/health.png

Local Setup

Backend:

cd api
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000

Frontend:

cd frontend
npm install
npm run dev

Worker:

cd workers/python_worker
pip install -r requirements.txt
REDIS_URL=redis://localhost:6379/0 python worker.py

Docker Setup

docker compose up --build

Open:

• Frontend: http://localhost:3000
• API docs: http://localhost:8000/docs
• Metrics: http://localhost:8000/metrics

Kubernetes Deployment

kubectl apply -f k8s/namespace.yaml
kubectl apply -f k8s/configmap.yaml
kubectl apply -f k8s/redis.yaml
kubectl apply -f k8s/api.yaml
kubectl apply -f k8s/worker.yaml
kubectl apply -f k8s/frontend.yaml
kubectl apply -f k8s/hpa.yaml
kubectl apply -f k8s/ingress.yaml

The Kubernetes files are realistic scaffolding. Replace image placeholders, add resource requests/limits, configure TLS, and use managed Redis before production use.

API Endpoints

• GET /health
• GET /metrics
• POST /api/jobs
• GET /api/jobs
• GET /api/jobs/{job_id}
• POST /api/jobs/{job_id}/retry
• DELETE /api/jobs/{job_id}
• GET /api/queue/stats
• GET /api/workers
• GET /api/events

Engineering Highlights

• Multi-page SaaS dashboard with Dashboard, Jobs, Workers, Metrics, and System Health views.
• Realistic sample data when the backend is unavailable, so the hosted demo still presents well.
• Redis-backed job state with queued, running, retrying, completed, failed, and cancelled lifecycle states.
• Retry and cancel flows exposed through API and UI.
• Worker heartbeat model and live event stream panel.
• Prometheus text metrics and Grafana dashboard scaffold.
• Docker Compose for local demo and Kubernetes manifests for deployment narrative.
• Terraform scaffold for cloud registry/networking extension.

Known Limitations

• Redis is used as the demo state store; production deployments should use durable backing services and stronger persistence settings.
• C++ worker is a compute simulation folder rather than a full queue consumer.
• Auth, tenant isolation, and RBAC are intentionally omitted to keep the demo focused.
• Kubernetes and Terraform files are scaffolding and need environment-specific values before real deployment.