Distributed Task Scheduler (POC)

Overview

This project is a test task, consisting of:

• Scheduler API (FastAPI + PostgreSQL)
• Multiple stateless workers
• Background timeout reaper
• Priority-based job assignment
• Strategy pattern for task execution

The goal is to demonstrate clean architecture, concurrency handling, and production-oriented design without unnecessary complexity.

---

Architecture

Components

Scheduler

• Accepts jobs
• Stores them in PostgreSQL
• Assigns jobs to workers
• Tracks execution state
• Detects timeouts
• Retries failed jobs once

Worker

• Stateless container
• Polls scheduler for work
• Executes tasks using Strategy Pattern
• Reports completion or failure

Reaper

• Background scheduler process
• Runs periodically
• Finds jobs stuck in RUNNING state longer than 30 seconds
• Requeues or marks them as failed

---

High-level flow

1. Client submits job
2. Job stored in database
3. Worker polls scheduler
4. Scheduler assigns next available job
5. Worker executes job
6. Worker reports completion or failure
7. Reaper retries timed-out jobs if needed

---

Strategy Pattern

Worker uses Strategy Pattern for job execution.

Each job type has its own handler:

• print → PrintTaskStrategy
• http_call → HttpTaskStrategy

This allows:

• Easy extension with new job types
• No conditional if/else chains
• Clear separation of responsibilities

Strategies are intentionally simple (print and mock HTTP calls) to focus on architecture rather than business logic.

---

Concurrency Safety

Job claiming uses:

SELECT ... FOR UPDATE SKIP LOCKED

This guarantees:

• Multiple workers can run concurrently
• No duplicate job execution
• No distributed locks required

---

Reliability

• Jobs retry once automatically
• Timeout detection after 30 seconds
• Stateless workers allow horizontal scaling
• Scheduler is single source of truth

---

Requirements

• Docker
• Docker Compose
• Python 3.11+

---

Local development setup

Create virtual environment

make venv  
source .venv/bin/activate

For IDE support:

• Set interpreter to .venv
• Mark as Sources Root:
  • scheduler/src
  • worker/src

---

Run make help for available commands

make help

Running tests

Use the provided Makefile targets so PYTHONPATH is set correctly.

• Run all tests (scheduler + worker):

make test

---

Running the system

Run make help for available commands

make help

Start everything

make up

Scheduler API:

http://localhost:8000

Health check:

curl http://localhost:8000/health

---

Scaling workers

make scale-worker N=3

---

Stop everything

make down

---

Example usage

Create a print job

curl -X POST http://localhost:8000/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "print",
    "priority": 10,
    "payload": {
      "message": "Hello from scheduler"
    }
  }'

---

Create an HTTP job

curl -X POST http://localhost:8000/jobs \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "http_call",
    "priority": 5,
    "payload": {
      "method": "GET",
      "url": "https://example.com",
      "timeout": 5
    }
  }'

---

Get job status

curl http://localhost:8000/jobs/1

---

Summary

This project demonstrates:

• Distributed system fundamentals
• Clean layered architecture
• Dependency injection
• Strategy design pattern
• Concurrency safety
• Fault tolerance
• Horizontal scalability

while keeping the solution readable and test-task appropriate.