Asynchronous Job Queue and Worker System (Node.js + Redis)

This is a simple asynchronous job queue and worker system built with Node.js and Redis, including:

• Producer (producer.js) that creates jobs
• Redis list queues for active jobs (LPUSH / BRPOP)
• Multiple workers (worker.js) that process jobs concurrently
• Retries and dead-letter queue handling
• Delayed jobs via Redis sorted set + scheduler (scheduler.js)
• Priority queues (high priority before normal)

The focus is educational: clear code and realistic infrastructure behavior without heavy frameworks.

Architecture

1. Producer

The producer builds a structured JSON job:

• id
• type
• payload
• retryCount
• timestamp
• priority

Then it either:

• pushes directly into active queue with LPUSH
• or schedules it in delayed set (ZADD) with a future timestamp score

2. Queue Layer (Redis)

Redis keys used:

• jobs:high - high-priority list queue
• jobs:normal - normal-priority list queue
• jobs:dead-letter - dead-letter queue
• jobs:delayed - delayed jobs sorted set

Workers consume with BRPOP in this order:

1. jobs:high
2. jobs:normal

This gives high-priority jobs preference.

3. Workers

Each worker process:

• waits for jobs via blocking pop (BRPOP)
• processes jobs asynchronously
• supports concurrency using an in-flight Promise set
• logs lifecycle events

When a job fails:

• increment retryCount
• if retry limit not reached: requeue job
• if retry limit exceeded: push to dead-letter queue

4. Scheduler

The scheduler runs every few seconds and:

• reads due jobs from jobs:delayed (score <= now)
• moves each ready job into high/normal active queue
• removes moved job from delayed set

Prerequisites

• Node.js 18+
• Redis server (local install) for local mode
• Docker + Docker Compose for containerized mode

Option A: Run Locally (Node + Local Redis)

1) Install dependencies

npm install

2) Start Redis

If Redis is installed locally, start it with your preferred command. Example:

redis-server

3) Start scheduler

Terminal 1:

npm run scheduler

4) Start one or more workers

Terminal 2 (and more terminals for additional workers):

npm run worker

Optional worker tuning:

WORKER_ID=worker-a WORKER_CONCURRENCY=5 MAX_RETRIES=3 npm run worker

5) Enqueue jobs from producer CLI

Normal job:

npm run producer -- send-email '{"to":"team@example.com","workMs":700}'

High-priority job:

npm run producer -- generate-report '{"reportId":42,"workMs":500}' --priority high

Delayed job (runs after 5 seconds):

npm run producer -- sync-data '{"source":"crm","workMs":900}' --delay 5000

Flaky job (random failures -> retries):

npm run producer -- resize-image '{"file":"photo.png","workMs":600}' --fail-rate 0.7

Always-fail job (moves to dead-letter after max retries):

npm run producer -- billing-charge '{"invoiceId":"inv-123","shouldFail":true}'

Option B: Run with Docker Compose

1) Build images

docker compose build

2) Start Redis + scheduler + worker(s)

docker compose up -d redis scheduler worker

Scale workers (example: 3 workers):

docker compose up -d --scale worker=3 worker

3) Enqueue jobs from the producer container

Normal job:

docker compose run --rm producer send-email '{"to":"team@example.com","workMs":700}'

High-priority job:

docker compose run --rm producer generate-report '{"reportId":42,"workMs":500}' --priority high

Delayed job:

docker compose run --rm producer sync-data '{"source":"crm","workMs":900}' --delay 5000

Failing job:

docker compose run --rm producer billing-charge '{"invoiceId":"inv-123","shouldFail":true}'

4) Watch logs

docker compose logs -f scheduler worker

5) Stop stack

docker compose down

Remove Redis volume too:

docker compose down -v

Environment Variables

• REDIS_URL (default: redis://127.0.0.1:6379)
• WORKER_ID (optional worker label)
• WORKER_CONCURRENCY (default: 3)
• MAX_RETRIES (default: 3)
• SCHEDULER_INTERVAL_MS (default: 2000)
• SCHEDULER_BATCH_SIZE (default: 100)

Example Flow

1. Producer creates a job and pushes to Redis.
2. Worker pops from high-priority queue first, then normal queue.
3. Worker logs start/completion or failure.
4. Failed jobs are retried with incremented retryCount.
5. Jobs that exceed max retries move to dead-letter queue.
6. Scheduler promotes delayed jobs into active queues when due.

Files

• redisClient.js - Redis connection setup
• job.js - job model and serialization helpers
• queue.js - queue operations, delayed jobs, retry/dead-letter logic
• producer.js - CLI producer
• worker.js - concurrent worker process
• scheduler.js - delayed job scheduler
• Dockerfile - app container image for worker/scheduler/producer
• docker-compose.yml - local multi-container orchestration