Distributed Job Queue (Go + Redis)

A lightweight distributed job queue built in Go that demonstrates asynchronous task processing using a queue-based architecture.
The system decouples job submission from execution and supports horizontal scaling through independent worker processes.

This project focuses on real-world backend and systems engineering concepts rather than framework-heavy abstractions.

---

Overview

The Distributed Job Queue allows clients to submit jobs via an HTTP API.
Submitted jobs are pushed into a Redis-backed queue and processed asynchronously by one or more worker services.

The architecture mirrors patterns commonly used in production systems for background processing, task scheduling, and event-driven workloads.

---

Architecture

Client
  |
  v
Job API (Go)
  |
  v
Redis Queue
  |
  v
Worker Service(s)

Core Components

• Job API
  Accepts incoming job requests over HTTP and enqueues them into Redis.

• Redis
  Acts as a distributed, durable queue using blocking pop semantics.

• Worker Service
  Runs as an independent process that continuously consumes and processes jobs from the queue.

---

API Specification

Endpoint

POST /jobs

Request Body

{
  "type": "email",
  "payload": "hello world"
}

Response

{
  "id": "f2c3e9a1-7b4f-4b52-a0c0-2d8b2c9b8c9a",
  "status": "queued"
}

---

Job Lifecycle

1. Client submits a job to the API
2. API validates and enqueues the job in Redis
3. Worker blocks on the queue and retrieves the job
4. Job is processed asynchronously
5. On failure, the job is retried up to a configured limit

---

Project Structure

.
├── cmd/
│   ├── api/
│   │   └── main.go        # Job submission API
│   └── worker/
│       └── main.go        # Worker process
├── pkg/
│   ├── model/
│   │   └── job.go         # Job definition
│   └── queue/
│       └── redis.go       # Redis queue implementation
├── docker-compose.yml     # Redis setup
├── go.mod
├── go.sum
└── README.md

---

Running Locally

1. Start Redis

docker compose up -d

2. Start the Job API

go run ./cmd/api

3. Start the Worker

go run ./cmd/worker

4. Submit a Job

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"type":"email","payload":"hello world"}' \
  http://localhost:8080/jobs

---

Key Technical Learnings

• Designing asynchronous systems using queues
• Decoupling producers and consumers
• Blocking queue patterns with Redis
• Building concurrent worker processes in Go
• Retry handling and failure management
• Structuring Go projects for maintainability

---

Why a Job Queue?

Job queues are a fundamental building block for:

• Background processing
• Email and notification systems
• Event handling
• Batch and scheduled workloads
• Scalable backend services

This project demonstrates how such systems work internally without relying on heavy frameworks.

---

Possible Enhancements

• Job processing by type (email, webhook, etc.)
• Multiple concurrent workers for horizontal scaling
• Dead-letter queue for failed jobs
• Metrics and observability
• Persistence and visibility tooling
• Deployment using containers or cloud services

---

Status

This project is functional and demonstrates a production-style distributed job queue implemented using Go and Redis.
It was built to deepen understanding of backend systems, concurrency, and asynchronous processing.