PyTaskFlow – Distributed Task Scheduler

PyTaskFlow is a distributed task scheduling and execution system built using Python , FastAPI, gRPC, PostgreSQL, React, and Docker.
It enables reliable scheduling, asynchronous execution, automatic retries, worker health monitoring, and real-time observability.
The system uses gRPC for internal coordination, and is deployed using Docker in a production-style cloud setup.

---

Demo Video: Click here to watch the demo

---

Key Features

• Distributed task scheduling and execution
• Time-based (future) task scheduling
• Asynchronous task execution
• Full task lifecycle tracking
• Automatic retries with exponential backoff
• Worker heartbeat and liveness detection
• Real-time monitoring dashboard
• Structured, color-coded logging
• Fully Dockerized microservices
• Cloud-deployable on AWS

---

Problem Statement

In real-world systems, background jobs must be:

• Scheduled reliably
• Executed asynchronously
• Retried on failure
• Recoverable after restarts
• Scalable across multiple workers
• Observable in real time

PyTaskFlow addresses these challenges by separating responsibilities into well-defined services that communicate using REST and gRPC, while persisting system state in a durable PostgreSQL database.

---

System Architecture

User
 ↓
React Dashboard
 ↓ (REST / SSE)
Scheduler (FastAPI)
 ↓ (Durable State)
PostgreSQL
 ↓ (Periodic Querying)
Coordinator
 ↓ (gRPC)
Worker(s)
 ↑ (Heartbeat)
Coordinator

---

Architectural Principles

• Separation of concerns
• Microservice-style architecture
• Control plane vs execution plane separation
• Async and non-blocking I/O
• Fault isolation and observability

---

Technology Stack

Backend

• Python 3.12
• FastAPI (REST API)
• SQLAlchemy (Async ORM)
• PostgreSQL
• gRPC with Protocol Buffers
• asyncio

Frontend

• React
• Vite
• Tailwind CSS
• Server-Sent Events (SSE)

DevOps / Infrastructure

• Docker
• Docker Compose
• Nginx
• AWS EC2
• Let’s Encrypt SSL (Certbot)
• DuckDNS

Observability

• structlog
• colorlog
• Health check endpoints
• Metrics endpoints (JSON and Prometheus-style)

---

Core Services

Scheduler (FastAPI – REST API)

• Accepts task submissions
• Validates input using Pydantic
• Persists tasks in PostgreSQL
• Exposes REST APIs for dashboard consumption
• Provides health and metrics endpoints
• Streams real-time updates via SSE

Coordinator (gRPC – Control Plane)

• Periodically queries the database for due tasks
• Dispatches tasks to workers via gRPC
• Tracks worker heartbeats
• Detects dead workers
• Handles retries with exponential backoff

Worker (gRPC – Execution Plane)

• Executes task commands asynchronously
• Limits concurrency using semaphores
• Reports execution results
• Sends periodic heartbeats

React Dashboard

• Schedule new tasks
• View task history
• Monitor worker health
• Observe live system metrics

---

Database Design

Task Table

• Full task lifecycle tracking
• created → scheduled → picked → running → completed / failed
• Execution timestamps
• Retry count and retry scheduling

Worker Table

• Worker identity
• Last heartbeat timestamp
• Alive / dead status

---

gRPC and Protocol Buffers

• Internal service communication via gRPC
• Strongly-typed contracts using Protocol Buffers
• Single source of truth: task.proto

---

Observability and Monitoring

• Structured, service-specific logging
• Color-coded log levels
• Health check endpoints
• Metrics APIs
• Real-time worker monitoring via SSE

---

AWS Deployment (Docker-based)

• AWS EC2
• Dockerized services:
  • Scheduler
  • Coordinator
  • Worker(s)
  • PostgreSQL
• Docker Compose for orchestration
• Nginx as reverse proxy and HTTPS termination

---