Go Job Scheduler

A distributed background job scheduler built in Go using goroutines, channels, and PostgreSQL. Supports concurrent job execution, automatic retries with exponential backoff, and graceful shutdown.

---

What It Does

You submit a job via HTTP. The system stores it in PostgreSQL, picks it up within 2 seconds, executes it using the right executor, and updates the status. If it fails, it retries automatically with increasing delays. Multiple instances of the scheduler can run simultaneously without processing the same job twice.

---

Architecture

Client (curl / Postman)
        │
        │  POST /jobs  {"type":"email", "payload":{...}}
        ▼
┌─────────────────────────────────────────┐
│           HTTP SERVER (:8080)           │
│   POST /jobs   → create job             │
│   GET  /jobs   → list jobs              │
│   GET  /jobs/:id → get job              │
│   GET  /stats  → counts by status       │
│   GET  /health → health check           │
└──────────────────┬──────────────────────┘
                   │ INSERT → status: pending
                   ▼
            PostgreSQL (jobs table)
                   │
                   │ SELECT FOR UPDATE SKIP LOCKED (every 2s)
                   ▼
┌─────────────────────────────────────────┐
│              SCHEDULER                  │
│  polls DB every 2s, claims ready jobs   │
└──────────────────┬──────────────────────┘
                   │ pushes into buffered channel (cap 100)
                   ▼
┌─────────────────────────────────────────┐
│           WORKER POOL (5 goroutines)    │
│                                         │
│  Worker-1 ◄── job from channel          │
│  Worker-2 ◄── job from channel          │
│  Worker-3 ◄── job from channel          │
│  Worker-4   (waiting)                   │
│  Worker-5   (waiting)                   │
└──────────────────┬──────────────────────┘
                   │
                   ▼
         Executor Registry
         "email"   → EmailExecutor
         "webhook" → WebhookExecutor
                   │
                   ▼
         UPDATE jobs SET status = 'completed'
         (or retry with exponential backoff)

---

Tech Stack

Technology	Purpose
Go 1.26	Core language — goroutines, channels, net/http
PostgreSQL 16	Job storage, status tracking
Redis 7	Distributed locking (defense-in-depth)
pgx/v5	PostgreSQL driver with connection pooling
go-redis/v9	Redis client
Docker + Colima	Local database infrastructure

No web framework — pure net/http from Go's standard library.

---

Project Structure

job-scheduler/
├── cmd/
│   └── server/
│       └── main.go              ← entry point, wires everything together
│
├── internal/
│   ├── models/
│   │   └── job.go               ← Job struct, JobStatus type
│   │
│   ├── executor/
│   │   ├── executor.go          ← Executor interface
│   │   ├── email.go             ← EmailExecutor
│   │   ├── webhook.go           ← WebhookExecutor
│   │   └── registry.go          ← maps type string → executor
│   │
│   ├── scheduler/
│   │   ├── worker.go            ← single worker goroutine
│   │   ├── worker_pool.go       ← manages N workers + job channel
│   │   └── scheduler.go         ← polls DB, feeds channel
│   │
│   ├── storage/
│   │   ├── postgres.go          ← all DB operations
│   │   └── redis.go             ← distributed lock
│   │
│   └── api/
│       ├── handlers.go          ← HTTP handlers
│       ├── middleware.go        ← logging, panic recovery, CORS
│       └── server.go            ← routes + middleware chain
│
├── migrations/
│   └── 001_create_jobs.sql      ← creates jobs table
│
├── docker-compose.yml           ← PostgreSQL + Redis
├── .env.example                 ← copy to .env and fill in values
├── go.mod
└── go.sum

---

Prerequisites

• Go 1.22+
• Docker (via Colima on Mac or Docker Desktop)

---

Running Locally

1. Clone and install dependencies

git clone https://github.com/manishv123/job-scheduler
cd job-scheduler
go mod download

2. Set up environment

cp .env.example .env
# Edit .env if needed — defaults work out of the box

3. Start databases

# Mac (using Colima)
colima start
docker-compose up -d

# Linux / Docker Desktop
docker compose up -d

PostgreSQL automatically runs migrations/001_create_jobs.sql on first start.

4. Run the server

go run cmd/server/main.go

You should see:

{"level":"INFO","msg":"connected to postgres","max_conns":25}
{"level":"INFO","msg":"starting worker pool","workers":5}
{"level":"INFO","msg":"worker started","worker_id":1}
{"level":"INFO","msg":"scheduler started","interval":2000000000,"batch_size":10}
{"level":"INFO","msg":"http server listening","addr":":8080"}

---

API Reference

Submit a Job

POST /jobs
Content-Type: application/json

{
  "type": "email",
  "payload": {
    "to": "user@example.com",
    "subject": "Hello",
    "body": "World"
  }
}

Optional fields:

• scheduled_at — RFC3339 timestamp, e.g. "2024-01-15T10:30:00Z" (default: now)
• max_attempts — integer (default: 3)

Response:

{"id": "63368de0-6efd-4ab8-85ad-05090bb2cd12", "status": "pending"}

---

Get a Job

GET /jobs/:id

Response:

{
  "id": "63368de0-6efd-4ab8-85ad-05090bb2cd12",
  "type": "email",
  "payload": {"to": "user@example.com", "subject": "Hello", "body": "World"},
  "status": "completed",
  "attempts": 1,
  "max_attempts": 3,
  "scheduled_at": "2024-01-15T10:30:00Z",
  "started_at": "2024-01-15T10:30:02Z",
  "completed_at": "2024-01-15T10:30:04Z",
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:30:04Z"
}

---

List Jobs

GET /jobs
GET /jobs?status=pending
GET /jobs?status=failed

---

Stats

GET /stats

Response:

{"completed": 47, "failed": 3, "pending": 2, "running": 1}

---

Health Check

GET /health

Response: ok with status 200

---

Supported Job Types

email

{
  "type": "email",
  "payload": {
    "to": "user@example.com",
    "subject": "Subject line",
    "body": "Email body text"
  }
}

webhook

{
  "type": "webhook",
  "payload": {
    "url": "https://api.example.com/hook",
    "method": "POST",
    "body": {"event": "user.signed_up", "user_id": 123},
    "headers": {"Authorization": "Bearer token123"}
  }
}

---

Key Design Decisions

Worker Pool Pattern

A fixed number of goroutines (default: 5) read from a shared buffered channel. This provides controlled concurrency — the system never spawns unlimited goroutines regardless of job volume.

Backpressure

The job channel has a capacity of 100. When all workers are busy and the channel is full, Submit blocks — the scheduler naturally slows down instead of building an unbounded in-memory queue.

Distributed Safety with FOR UPDATE SKIP LOCKED

Multiple instances of this scheduler can run against the same PostgreSQL database. The ClaimReadyJobs query uses SELECT ... FOR UPDATE SKIP LOCKED — each instance claims different rows, making duplicate execution impossible at the database level.

Exponential Backoff

Failed jobs are retried after 2^attempt seconds:

• Attempt 1 → retry in 2s
• Attempt 2 → retry in 4s
• Attempt 3 → retry in 8s

Graceful Shutdown

On SIGTERM or SIGINT:

1. HTTP server stops accepting new requests (30s drain window)
2. Scheduler stops polling for new jobs
3. Workers finish their current job before exiting
4. No in-flight jobs are abandoned

---

Stopping the Server

Press Ctrl+C. You will see:

{"level":"INFO","msg":"shutdown signal received","signal":"interrupt"}
{"level":"INFO","msg":"shutting down..."}
{"level":"INFO","msg":"worker stopped","worker_id":3}
{"level":"INFO","msg":"worker stopped","worker_id":1}
{"level":"INFO","msg":"worker pool stopped"}
{"level":"INFO","msg":"shutdown complete"}

Stop databases:

docker-compose down
colima stop   # Mac only

---

Adding a New Job Type

1. Create internal/executor/yourtype.go implementing the Executor interface:

type YourExecutor struct { ... }

func (e *YourExecutor) Type() string { return "yourtype" }

func (e *YourExecutor) Execute(ctx context.Context, payload json.RawMessage) error {
    // your logic here
    return nil
}

2. Register it in cmd/server/main.go:

registry.Register(NewYourExecutor(logger))

3. Submit jobs with "type": "yourtype" — no other changes needed.