A robust, production-ready job queue system built with TypeScript, Redis, Express.js, and Bun. TinyQueue provides FIFO processing, concurrent worker pools, automatic retry mechanisms with exponential backoff, and real-time monitoring.
- FIFO Job Processing - First-in, first-out queue using Redis lists
- Concurrent Worker Pool - Configurable number of parallel workers
- Automatic Retry Logic - Exponential backoff for failed jobs
- Job State Management - Track jobs through pending → processing → completed/failed states
- Type-Safe Handler Registration - Strongly typed job handlers with TypeScript
- Real-Time Monitoring - REST API for job status and queue statistics
- Graceful Shutdown - Proper cleanup on process termination
- Atomic Operations - Redis atomic operations prevent job loss
Before you begin, ensure you have the following installed:
- Bun >= 1.0.0 (Installation Guide)
- Redis >= 6.0 (Installation Guide)
- Node.js >= 18.0 (if not using Bun exclusively)
git clone https://github.com/dexter-ifti/TinyQueue.git
cd tinyqueuebun install Docker
docker run -d --name redis-queue -p 6379:6379 redis:latestredis-cli ping
# Should return: PONGbun run devCreate .env file:
PORT=3000
REDIS_URL=redis://localhost:6379
WORKER_CONCURRENCY=5# Development mode with hot reload
bun --watch src/server.ts
# Production mode
bun run src/server.tsYou should see:
✓ TinyQueue started with 5 workers
Server running on http://localhost:3000
┌─────────────┐
│ Client │
└──────┬──────┘
│ HTTP POST /jobs
▼
┌─────────────────────────────────────┐
│ Express Server │
│ ┌───────────────────────────────┐ │
│ │ Queue Manager │ │
│ │ • enqueue() │ │
│ │ • dequeue() │ │
│ │ • complete() │ │
│ │ • fail() │ │
│ └───────────┬───────────────────┘ │
└──────────────┼──────────────────────┘
│
▼
┌────────────────┐
│ Redis │
│ • queue:jobs │ (Pending)
│ • queue:proc │ (Processing)
│ • queue:job:* │ (Job Data)
└────────┬───────┘
│
▼
┌──────────────────────────────────────┐
│ Worker Pool (N workers) │
│ ┌──────────┐ ┌──────────┐ │
│ │ Worker 1 │ │ Worker 2 │ ... │
│ └────┬─────┘ └────┬─────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────────────────┐ │
│ │ Job Handlers │ │
│ │ • send-email │ │
│ │ • process-image │ │
│ │ • export-data │ │
│ └─────────────────────────────┘ │
└──────────────────────────────────────┘
- Job Submission: Client sends job via POST
/jobs - Enqueue: Job stored in Redis with unique ID, added to pending queue
- Dequeue: Worker atomically moves job from pending to processing
- Execute: Handler processes job payload
- Complete/Fail: Job status updated, removed from processing queue
- Retry (if failed): Job re-queued with exponential backoff
Create a new job and add it to the queue.
Endpoint: POST /jobs
Request Body:
{
"type": "send-email",
"payload": {
"to": "user@example.com",
"subject": "Welcome!",
"body": "Thanks for signing up"
},
"maxAttempts": 3
}Response: 201 Created
{
"jobId": "550e8400-e29b-41d4-a716-446655440000",
"status": "pending"
}cURL Example:
curl -X POST http://localhost:3000/jobs \
-H "Content-Type: application/json" \
-d '{
"type": "send-email",
"payload": {
"to": "user@example.com",
"subject": "Hello World",
"body": "This is a test email"
},
"maxAttempts": 3
}'Retrieve current status and details of a specific job.
Endpoint: GET /jobs/:id
Response: 200 OK
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"type": "send-email",
"payload": {
"to": "user@example.com",
"subject": "Welcome!",
"body": "Thanks for signing up"
},
"status": "completed",
"attempts": 1,
"maxAttempts": 3,
"createdAt": 1700000000000,
"updatedAt": 1700000002000
}Job Statuses:
pending- Job is waiting in queueprocessing- Job is currently being processedcompleted- Job finished successfullyfailed- Job failed after all retry attempts
cURL Example:
curl http://localhost:3000/jobs/550e8400-e29b-41d4-a716-446655440000View current queue metrics.
Endpoint: GET /stats
Response: 200 OK
{
"pending": 42,
"processing": 5
}cURL Example:
curl http://localhost:3000/statsVerify the server is running.
Endpoint: GET /health
Response: 200 OK
{
"status": "ok"
}Dockerfile:
FROM oven/bun:1 as base
WORKDIR /app
# Install dependencies
COPY package.json bun.lockb ./
RUN bun install --frozen-lockfile
# Copy source
COPY . .
# Expose port
EXPOSE 3000
# Run
CMD ["bun", "run", "src/server.ts"]docker-compose.yml:
version: '3.8'
services:
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redis-data:/data
command: redis-server --appendonly yes
tinyqueue:
build: .
ports:
- "3000:3000"
environment:
- REDIS_URL=redis://redis:6379
- WORKER_CONCURRENCY=10
depends_on:
- redis
restart: unless-stopped
volumes:
redis-data:Deploy:
docker-compose up -d- Enable Redis persistence (AOF or RDB)
- Set up Redis authentication
- Configure Redis memory limits
- Enable TLS/SSL for Redis connection
- Set up log aggregation (ELK, Datadog)
- Configure application monitoring (Prometheus, Grafana)
- Set up alerts for queue size thresholds
- Implement rate limiting on API endpoints
- Add authentication to API endpoints
- Set up automated backups for Redis data
- Configure horizontal pod autoscaling (K8s)
- Document incident response procedures
Contributions are welcome! Please follow these guidelines:
- Fork the repository
- Create a feature branch:
git checkout -b feature/amazing-feature - Commit changes:
git commit -m 'Add amazing feature' - Push to branch:
git push origin feature/amazing-feature - Open a Pull Request
MIT License - see LICENSE file for details
- Documentation: GitHub Wiki
- Issues: GitHub Issues
- Discussions: GitHub Discussions
Happy queuing!