Skip to content

Queue and Jobs

Precious Lyson edited this page Nov 13, 2026 · 2 revisions

The Machinjiri Queue-Job System is a robust, feature-rich background job processing framework that allows you to defer time-consuming tasks to be processed asynchronously. Whether you're sending emails, processing images, generating reports, or handling any other resource-intensive operation, this system provides the tools you need.

Why Use Queues?

Queues solve several critical problems in web applications:

  • Improved Response Times: Offload slow operations so users don't wait
  • Reliability: Retry failed jobs automatically with exponential backoff
  • Scalability: Distribute work across multiple workers and servers
  • Resource Management: Control memory usage and prevent runaway processes
  • Observability: Track job status, failures, and performance metrics

Features

Core Features

Feature Description
Multiple Drivers Database, Redis, File, Memory, and Sync drivers
Job Retries Automatic retry with configurable backoff strategies
Delayed Jobs Schedule jobs to run after a specified delay
Priority Queues Process high-priority jobs first
Job Batching Process multiple jobs efficiently
Failure Handling Comprehensive failed job storage and retry
Payload Compression Optional compression for large payloads
Event System Hooks for job lifecycle events

Worker Management

Feature Description
Daemon Mode Long-running worker processes
Memory Limits Auto-restart workers exceeding memory limits
Job Limits Process a specific number of jobs then exit
Graceful Shutdown Handle SIGINT/SIGTERM for clean shutdown
Heartbeat Monitoring Track worker health status
Auto-Restart Supervisor restarts failed workers
Concurrency Control Run multiple worker instances

Step 1: Initialize Queue System

Run the initialization command to set up the queue infrastructure:

php artisan queue:init

This command will:

  • Create necessary queue driver files
  • Generate configuration files
  • Set up database tables (if using database driver)
  • Create the QueueServiceProvider

Step 2: Configure Your Queue Driver

Edit config/queue.php to configure your preferred driver:

<?php
return [
    'default' => env('QUEUE_DRIVER', 'database'),
    
    'drivers' => [
        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
            'failed_table' => 'failed_jobs',
            'processed_table' => 'processed_jobs',
        ],
        
        'redis' => [
            'driver' => 'redis',
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'port' => env('REDIS_PORT', 6379),
            'password' => env('REDIS_PASSWORD', null),
            'database' => env('REDIS_DATABASE', 0),
            'queue' => 'default',
            'retry_after' => 90,
        ],
    ],
];

Step 3: Set Environment Variables

Add to your .env file:

QUEUE_DRIVER=database

# Worker settings
QUEUE_WORKER_MAX_MEMORY=128
QUEUE_WORKER_MAX_JOBS=500
QUEUE_WORKER_CHECK_INTERVAL=5
QUEUE_WORKER_HEARTBEAT_TTL=15
QUEUE_WORKER_GRACE_PERIOD=10
QUEUE_WORKER_STOP_ON_EMPTY=false

Step 4: Verify Installation

Check that everything is set up correctly:

php artisan queue:health

Quick Start

1. Create Your First Job

php artisan queue:make-job SendWelcomeEmail --queue=emails

This creates app/Jobs/SendWelcomeEmailJob.php:

<?php

namespace App\Jobs;

use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseJob;
use Mlangeni\Machinjiri\Core\Exceptions\ContractsException\JobException;

class SendWelcomeEmailJob extends BaseJob
{
    public function __construct(
        \Mlangeni\Machinjiri\Core\Container $app,
        array $payload = [],
        array $options = []
    ) {
        $defaultOptions = [
            'maxAttempts' => 3,
            'queue' => 'emails',
            'timeout' => 60,
        ];
        
        parent::__construct($app, $payload, array_merge($defaultOptions, $options));
    }

    public function handle(): void
    {
        $data = $this->getPayload();
        $email = $data['email'] ?? null;
        $name = $data['name'] ?? 'User';
        
        // Send the welcome email
        // Your email sending logic here
        
        $this->addMetadata('sent_at', date('Y-m-d H:i:s'));
    }

    public function failed(JobException $exception): void
    {
        // Handle failure - log, notify, etc.
    }
}

2. Dispatch the Job

use App\Jobs\SendWelcomeEmailJob;

// Get the dispatcher
$dispatcher = $app->resolve('queue.dispatcher');

// Create and dispatch the job
$job = new SendWelcomeEmailJob($app, [
    'email' => 'user@example.com',
    'name' => 'John Doe'
]);

$jobId = $dispatcher->dispatch($job);

3. Start Processing Jobs

# Start a worker for the emails queue
php artisan queue:work --queue=emails

That's it! Your job will now be processed by the worker.


Core Concepts

Jobs

Jobs are the fundamental unit of work in the queue system. Each job is a PHP class that extends BaseJob and implements a handle() method.

Job Lifecycle:

Created → Queued → Reserved → Processing → Completed
                                    ↓
                                 Failed → Retry → ...
                                    ↓
                              Permanently Failed

Queues

Queues are named channels that hold jobs waiting to be processed. You can have multiple queues for different job types:

  • default - General purpose jobs
  • emails - Email sending jobs
  • images - Image processing jobs
  • reports - Report generation jobs
  • high - High-priority jobs

Workers

Workers are processes that pull jobs from queues and execute them. Workers can run:

  • Single-run mode: Process available jobs then exit
  • Daemon mode: Run continuously until stopped
  • With limits: Stop after N jobs or memory usage

Drivers

Drivers determine how jobs are stored and retrieved:

Driver Use Case Pros Cons
database General purpose Persistent, queryable Slower than Redis
redis High performance Very fast, feature-rich Requires Redis
file Simple setups No dependencies Not for high volume
memory Testing Instant Not persistent
sync Development Immediate execution Blocks request

Queue Drivers

Database Driver

The database driver stores jobs in SQL tables, providing persistence and queryability.

Setup:

php artisan queue:init

This creates three tables:

  • jobs - Pending jobs
  • failed_jobs - Failed jobs
  • processed_jobs - Completed jobs

Configuration:

'database' => [
    'driver' => 'database',
    'table' => 'jobs',
    'queue' => 'default',
    'retry_after' => 90,
    'failed_table' => 'failed_jobs',
    'processed_table' => 'processed_jobs',
],

Features:

  • Job reservation with timeout
  • Automatic retry of stale jobs
  • Full job history
  • Transaction support

Redis Driver

The Redis driver provides high-performance job storage using Redis lists and sorted sets.

Setup:

# Install Redis extension
pecl install redis

# Or use Predis
composer require predis/predis

Configuration:

'redis' => [
    'driver' => 'redis',
    'host' => env('REDIS_HOST', '127.0.0.1'),
    'port' => env('REDIS_PORT', 6379),
    'password' => env('REDIS_PASSWORD', null),
    'database' => env('REDIS_DATABASE', 0),
    'prefix' => 'queue:',
    'retry_after' => 90,
    'timeout' => 2.5,
],

Features:

  • Atomic operations
  • Delayed jobs with sorted sets
  • Reserved job tracking
  • Pub/sub notifications (optional)

File Driver

The file driver stores jobs as JSON files, suitable for simple deployments.

Configuration:

'file' => [
    'driver' => 'file',
    'path' => __DIR__ . '/../storage/queue',
    'retry_after' => 90,
],

Directory Structure:

storage/queue/
├── pending/     # Jobs waiting to be processed
├── processing/  # Jobs currently being processed
├── failed/      # Failed jobs
└── processed/   # Completed jobs

Memory Driver

The memory driver stores jobs in PHP arrays, useful for testing.

Configuration:

'memory' => [
    'driver' => 'memory',
    'retry_after' => 90,
],

Note: Jobs are lost when the process ends. Use only for testing.

Sync Driver

The sync driver executes jobs immediately, blocking the request.

Configuration:

'sync' => [
    'driver' => 'sync',
],

Use Cases:

  • Development environments
  • Testing
  • Jobs that must complete before response

Creating Jobs

Basic Job

php artisan queue:make-job ProcessOrder

Creates app/Jobs/ProcessOrderJob.php:

<?php

namespace App\Jobs;

use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseJob;

class ProcessOrderJob extends BaseJob
{
    public function handle(): void
    {
        $payload = $this->getPayload();
        $orderId = $payload['order_id'] ?? null;
        
        // Process the order
        $order = $this->app->resolve('db')->table('orders')->find($orderId);
        $order->status = 'processing';
        $order->save();
        
        // ... more processing
        
        $this->addMetadata('processed_at', date('Y-m-d H:i:s'));
    }
}

Job Options

When creating a job, you can specify various options:

$job = new ProcessOrderJob($app, $payload, [
    'queue' => 'orders',           // Queue name
    'maxAttempts' => 5,            // Maximum retry attempts
    'timeout' => 120,              // Timeout in seconds
    'delay' => 60,                 // Initial delay in seconds
    'retryDelay' => 30,            // Base retry delay
    'compressPayload' => true,     // Compress large payloads
    'metadata' => [                // Custom metadata
        'user_id' => 123,
        'source' => 'api',
    ],
]);

Job Types

The generator supports different job templates:

# Standard job
php artisan queue:make-job ProcessData

# Email job
php artisan queue:make-job SendNewsletter --type=email --queue=emails

# Model job with database
php artisan queue:make-job SyncUser --type=model --database

Email Job Example

<?php

namespace App\Jobs;

use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseJob;
use Mlangeni\Machinjiri\Core\Transport\Mail\MailMessage;
use Mlangeni\Machinjiri\Core\Transport\Mail\MailManager;

class SendNewsletterJob extends BaseJob
{
    public function handle(): void
    {
        $data = $this->getPayload();
        $messageData = $data['message'] ?? [];
        
        $message = MailMessage::fromArray($messageData);
        $mailManager = $this->app->make(MailManager::class);
        
        $response = $mailManager->send($message);
        
        $this->addMetadata('message_id', $response->getMessageId());
    }
}

Dispatching Jobs

Basic Dispatch

$dispatcher = $app->resolve('queue.dispatcher');
$job = new ProcessOrderJob($app, ['order_id' => 123]);
$jobId = $dispatcher->dispatch($job);

Dispatch to Specific Queue

$jobId = $dispatcher->dispatchToQueue($job, 'high-priority');

Dispatch with Delay

// Delay 60 seconds
$jobId = $dispatcher->dispatchWithDelay($job, 60);

Dispatch Immediately (Sync)

$result = $dispatcher->dispatchNow($job);

Bulk Dispatch

$jobs = [
    new SendEmailJob($app, ['to' => 'user1@example.com']),
    new SendEmailJob($app, ['to' => 'user2@example.com']),
    new SendEmailJob($app, ['to' => 'user3@example.com']),
];

$jobIds = $dispatcher->dispatchBulk($jobs);

Job Prioritization

// Via dispatcher
$queue = $app->resolve('queue');
$queue->pushWithPriority($job, 'default', 10); // Priority 10

// Via job metadata
$job->addMetadata('priority', 10);

Running Workers

Basic Worker

php artisan queue:work

Worker Options

Option Short Description Default
--queue Queue name default
--sleep -s Seconds to sleep when empty 3
--memory -m Memory limit in MB 128
--timeout -t Job timeout in seconds 60
--max-jobs Max jobs before exit unlimited
--stop-on-empty Stop when queue is empty false
--tries Max attempts per job 3
--daemon Run continuously false
--force -f Run in maintenance mode false

Examples

# Process emails queue
php artisan queue:work --queue=emails

# High-throughput worker
php artisan queue:work --queue=high --sleep=1 --memory=256

# Process specific number of jobs
php artisan queue:work --max-jobs=100

# One-time processing (stop when empty)
php artisan queue:work --stop-on-empty

# Daemon mode with PID file
php artisan queue:work --daemon --pid-file=/var/run/worker.pid

Worker Signals

Workers respond to Unix signals:

  • SIGTERM / SIGINT: Graceful shutdown (finish current job)
  • SIGKILL: Force kill (may leave job in reserved state)

Worker Management

Starting Workers

# Start single worker
php artisan queue:worker-start

# Start multiple workers
php artisan queue:worker-start --concurrency=4

# For specific queue
php artisan queue:worker-start --queue=emails --concurrency=2

Checking Worker Status

php artisan queue:worker-status --queue=default

Output:

+----------+-------+---------+---------+-------------+---------------------+
| Instance | PID   | Running | Healthy | Memory (MB) | Last Heartbeat      |
+----------+-------+---------+---------+-------------+---------------------+
| 1        | 12345 | Yes     | Yes     | 45.2        | 2024-01-15 10:30:00 |
| 2        | 12346 | Yes     | Yes     | 42.8        | 2024-01-15 10:30:00 |
+----------+-------+---------+---------+-------------+---------------------+

Stopping Workers

# Stop all workers for queue
php artisan queue:worker-stop --queue=default

# Stop specific instance
php artisan queue:worker-stop --queue=default --instance=1

Restarting Workers

php artisan queue:worker-restart --queue=default --concurrency=4

Cleanup Orphaned Workers

php artisan queue:worker-cleanup

Monitoring & Supervision

Supervisor Command

The supervisor monitors workers and restarts them if they fail:

# Run supervisor in daemon mode
php artisan queue:supervisor --queue=default --concurrency=4 --daemon

What the Supervisor Does:

  1. Checks worker status every N seconds
  2. Restarts dead workers
  3. Restarts unhealthy workers (stale heartbeat)
  4. Restarts workers exceeding memory limit

Using with Systemd

Create /etc/systemd/system/queue-worker.service:

[Unit]
Description=Machinjiri Queue Worker
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/var/www/app
ExecStart=/usr/bin/php artisan queue:supervisor --queue=default --concurrency=4 --daemon
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable queue-worker
sudo systemctl start queue-worker

Using with Supervisord

Create /etc/supervisor/conf.d/queue-worker.conf:

[program:queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work --daemon
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
stopwaitsecs=3600

Health Monitoring

# Check queue health
php artisan queue:health

# Check database connection
php artisan queue:db-info

Queue Statistics

php artisan queue:status

Output:

+---------+----------+---------+-------+---------+
| Pending | Reserved | Delayed | Total | Health  |
+---------+----------+---------+-------+---------+
| 45      | 4        | 12      | 61    | Healthy |
+---------+----------+---------+-------+---------+

Failed Jobs

Viewing Failed Jobs

php artisan queue:failed

Output:

+----+--------+---------+---------------------+-----------------+
| No | Job ID | Queue   | Failed At           | Error           |
+----+--------+---------+---------------------+-----------------+
| 1  | job_01 | emails  | 2024-01-15 10:30:00 | Connection ti...|
| 2  | job_02 | default | 2024-01-15 10:25:00 | Invalid argum...|
+----+--------+---------+---------------------+-----------------+

Retrying Failed Jobs

# Retry specific job
php artisan queue:retry job_01

# Retry all failed jobs
php artisan queue:retry all

# Retry jobs from specific queue
php artisan queue:retry all --queue=emails

Removing Failed Jobs

# Remove specific job
php artisan queue:forget job_01

# Flush all failed jobs
php artisan queue:flush

Automatic Retry Logic

Jobs are retried automatically with exponential backoff:

Attempt 1: Fail → Wait 60s
Attempt 2: Fail → Wait 120s (60 * 2^1)
Attempt 3: Fail → Wait 240s (60 * 2^2)
Attempt 4: Fail → Permanently failed

Configure retry delay:

$job = new MyJob($app, $payload, [
    'maxAttempts' => 5,
    'retryDelay' => 30, // Base delay in seconds
]);

Configuration

Queue Configuration File

config/queue.php:

<?php
return [
    'default' => env('QUEUE_DRIVER', 'database'),
    
    'drivers' => [
        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
            'failed_table' => 'failed_jobs',
            'processed_table' => 'processed_jobs',
        ],
        
        'redis' => [
            'driver' => 'redis',
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'port' => env('REDIS_PORT', 6379),
            'password' => env('REDIS_PASSWORD', null),
            'database' => env('REDIS_DATABASE', 0),
            'queue' => 'default',
            'retry_after' => 90,
            'block_for' => null,
        ],
        
        'file' => [
            'driver' => 'file',
            'path' => __DIR__ . '/../storage/queue',
            'retry_after' => 90,
        ],
        
        'memory' => [
            'driver' => 'memory',
            'retry_after' => 90,
        ],
        
        'sync' => [
            'driver' => 'sync',
        ],
    ],
    
    'failed' => [
        'driver' => 'database',
        'database' => env('DB_CONNECTION', 'mysql'),
        'table' => 'failed_jobs',
    ],
];

Jobs Configuration File

config/jobs.php:

<?php
return [
    'defaults' => [
        'queue' => 'default',
        'max_attempts' => 3,
        'timeout' => 60,
        'retry_delay' => 60,
    ],
    
    'jobs' => [
        'send_email' => [
            'queue' => 'emails',
            'max_attempts' => 5,
            'timeout' => 120,
            'class' => App\Jobs\SendEmailJob::class,
        ],
        'process_image' => [
            'queue' => 'images',
            'max_attempts' => 3,
            'timeout' => 180,
            'class' => App\Jobs\ProcessImageJob::class,
        ],
    ],
    
    'queue_priority' => [
        'high',
        'default',
        'low',
        'emails',
        'notifications',
    ],
];

Environment Variables

# Driver
QUEUE_DRIVER=database

# Worker Settings
QUEUE_WORKER_MAX_MEMORY=128
QUEUE_WORKER_MAX_JOBS=500
QUEUE_WORKER_CHECK_INTERVAL=5
QUEUE_WORKER_HEARTBEAT_TTL=15
QUEUE_WORKER_HEARTBEAT_INTERVAL=60
QUEUE_WORKER_GRACE_PERIOD=10
QUEUE_WORKER_STOP_ON_EMPTY=false

# Redis (if using Redis driver)
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DATABASE=0

Advanced Topics

Custom Queue Drivers

Create a custom driver by extending BaseQueue:

<?php

namespace App\Queue\Drivers;

use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseQueue;
use Mlangeni\Machinjiri\Core\Artisans\Contracts\JobInterface;

class CustomQueue extends BaseQueue
{
    protected array $config = [];
    
    public function __construct($app, string $name, array $config = [])
    {
        parent::__construct($app, $name, $config);
        $this->config = array_merge([
            // Default configuration
        ], $config);
    }
    
    public function push(JobInterface $job, string $queue = 'default', int $delay = 0): string
    {
        // Implement job storage
        $jobId = $job->getId();
        $serialized = json_encode($job->serialize());
        
        // Store in your backend
        
        return $jobId;
    }
    
    public function pop(string $queue = 'default'): ?JobInterface
    {
        // Implement job retrieval
        return null;
    }
    
    // Implement other required methods...
}

Job Middleware

Add middleware to jobs for cross-cutting concerns:

class MyJob extends BaseJob
{
    public function handle(): void
    {
        // Pre-processing
        $this->beforeHandle();
        
        try {
            // Main logic
            $this->process();
        } finally {
            // Post-processing
            $this->afterHandle();
        }
    }
    
    protected function beforeHandle(): void
    {
        $this->addMetadata('started_at', microtime(true));
    }
    
    protected function afterHandle(): void
    {
        $started = $this->getMetadata()['started_at'] ?? 0;
        $duration = microtime(true) - $started;
        $this->addMetadata('duration', $duration);
    }
}

Job Events

Listen to job lifecycle events:

use Mlangeni\Machinjiri\Core\Artisans\Events\EventListener;

$events = new EventListener();
$events->listen('job.completed', function($data) {
    // Log completed job
    error_log("Job {$data['job_id']} completed");
});

$events->listen('job.failed', function($data) {
    // Alert on failure
    error_log("Job {$data['job_id']} failed: {$data['exception']}");
});

Payload Compression

For jobs with large payloads:

$job = new ProcessLargeDataJob($app, $largeData, [
    'compressPayload' => true,
]);

Job Batching

Process jobs in batches for efficiency:

$worker = $app->resolve('queue.worker');
$processed = $worker->processBatch('default', 10);

Memory Management

For long-running workers, implement memory checks:

class MyJob extends BaseJob
{
    public function handle(): void
    {
        // Check memory before heavy operation
        if (memory_get_usage() > 100 * 1024 * 1024) {
            throw new JobException('Memory limit approaching, retrying later');
        }
        
        // Process...
    }
}

Command Reference

Queue Management

Command Description
queue:init Initialize queue system
queue:make Create queue driver
queue:make-job Create job class
queue:status Show queue status
queue:health Check queue health
queue:db-info Show database info

Worker Management

Command Description
queue:work Start worker
queue:worker-start Start managed workers
queue:worker-stop Stop managed workers
queue:worker-restart Restart workers
queue:worker-status Show worker status
queue:worker-cleanup Clean orphaned files
queue:supervisor Run supervisor

Failed Jobs

Command Description
queue:failed List failed jobs
queue:retry Retry failed job
queue:forget Remove failed job
queue:flush Clear failed jobs

Queue Operations

Command Description
queue:clear Clear queue
queue:job Process specific job type

Troubleshooting

Common Issues

Jobs Not Processing

  1. Check worker is running:

    php artisan queue:worker-status
  2. Check queue has jobs:

    php artisan queue:status
  3. Check worker logs:

    tail -f storage/logs/queue-*.log
  4. Verify queue name matches:

    // Job queue
    $job = new MyJob($app, $payload, ['queue' => 'emails']);
    
    // Worker queue
    php artisan queue:work --queue=emails

Database Connection Issues

php artisan queue:db-info

If you see "No such file or directory":

  • Change DB_HOST=localhost to DB_HOST=127.0.0.1
  • Or provide the correct socket path

Memory Limit Exceeded

Workers restart automatically when exceeding memory limit. To increase:

php artisan queue:work --memory=256

Or set in .env:

QUEUE_WORKER_MAX_MEMORY=256

Stuck Jobs

If jobs are stuck in "reserved" state:

  1. Check if worker is still running
  2. Wait for retry_after timeout
  3. Or manually clear and re-queue
// Check reserved jobs
$stats = $queue->getStats('default');
echo "Reserved: " . $stats['reserved'];

// Wait for retry_after or restart worker

Permission Denied

Ensure storage directories are writable:

chmod -R 755 storage/queue
chown -R www-data:www-data storage/queue

Debug Mode

Enable verbose output:

php artisan queue:work -v

Logging

Queue logs are stored in storage/logs/:

  • queue-supervisor-*.log - Supervisor logs
  • queue-worker-*.log - Worker logs
  • queue-processor-*.log - Processor logs
  • worker-{driver}-{queue}.log - Individual worker output

API Reference

BaseJob

abstract class BaseJob implements JobInterface
{
    // Identification
    public function getId(): string;
    public function getName(): string;
    
    // Payload
    public function getPayload(): array;
    
    // Attempts
    public function getAttempts(): int;
    public function incrementAttempts(): void;
    public function getMaxAttempts(): int;
    public function getNextRetryDelay(): int;
    
    // Queue
    public function getQueue(): string;
    public function getDelay(): int;
    public function getTimeout(): int;
    public function getRetryDelay(): int;
    
    // Metadata
    public function getMetadata(): array;
    public function setMetadata(array $metadata): void;
    public function addMetadata(string $key, $value): void;
    
    // Lifecycle
    abstract public function handle(): void;
    public function failed(JobException $exception): void;
    
    // Serialization
    public function serialize(): array;
    public static function unserialize(array $data, ?Container $app = null): self;
}

BaseQueue

abstract class BaseQueue implements QueueInterface
{
    // Push
    public function push(JobInterface $job, string $queue = 'default', int $delay = 0): string;
    public function bulk(array $jobs, string $queue = 'default', int $delay = 0): array;
    public function pushWithPriority(JobInterface $job, string $queue = 'default', int $priority = 0, int $delay = 0): string;
    
    // Pop
    public function pop(string $queue = 'default'): ?JobInterface;
    public function popBatch(string $queue = 'default', int $batchSize = 10): array;
    
    // Release/Delete
    public function release(JobInterface $job, string $queue = 'default', int $delay = 0): bool;
    public function delete(JobInterface $job, string $queue = 'default'): bool;
    
    // Status
    public function size(string $queue = 'default'): int;
    public function getQueues(): array;
    public function getStats(string $queue = 'default'): array;
    public function isHealthy(): bool;
    public function getHealthMetrics(string $queue = 'default'): array;
    
    // Failed Jobs
    public function getFailed(string $queue = 'default', int $limit = 50, int $offset = 0): array;
    public function retryFailed(string $jobId, string $queue = 'default'): bool;
    public function forgetFailed(string $jobId, string $queue = 'default'): bool;
    public function flushFailed(string $queue = 'default'): int;
    
    // Clear
    public function clear(string $queue = 'default'): int;
}

BaseWorker

class BaseWorker implements WorkerInterface
{
    // Lifecycle
    public function start(string $queue, array $options = []): void;
    public function stop(): void;
    public function pause(): void;
    public function resume(): void;
    public function isRunning(): bool;
    
    // Status
    public function getStatus(): array;
    
    // Processing
    public function processNextJob(string $queue): bool;
    public function run(string $queue, ?int $maxJobs = null): int;
    public function processBatch(string $queue = 'default', int $batchSize = 10): int;
    
    // Heartbeat
    public function setHeartbeatCallback(callable $callback, int $intervalSeconds = 60): void;
}

BaseJobDispatcher

class BaseJobDispatcher implements JobDispatcherInterface
{
    // Dispatch
    public function dispatch(JobInterface $job): string;
    public function dispatchToQueue(JobInterface $job, string $queue): string;
    public function dispatchWithDelay(JobInterface $job, int $delay): string;
    public function dispatchNow(JobInterface $job): mixed;
    public function dispatchBulk(array $jobs): array;
    
    // Configuration
    public function getDefaultQueue(): string;
    public function setDefaultQueue(string $queue): void;
}

BackgroundWorkerManager

class BackgroundWorkerManager
{
    // Configuration
    public function reloadConfig(): void;
    
    // Worker Management
    public function startWorker(string $queue, string $driver = 'default', int $concurrency = 1): int;
    public function stopWorker(string $queue, string $driver = 'default', ?int $instance = null);
    public function restartWorker(string $queue, string $driver = 'default', ?int $instance = null, int $concurrency = 1): int;
    public function workerStatus(string $queue, string $driver = 'default', ?int $instance = null): array;
    
    // Monitoring
    public function monitorWorker(string $queue, string $driver = 'default', int $concurrency = 1, ?callable $output = null): void;
    public function updateHeartbeat(string $queue, string $driver = 'default', ?int $instance = null): void;
    
    // Cleanup
    public function cleanupOrphanedPids(): void;
}

Examples

Sending Emails Asynchronously

// Create the job
class SendEmailJob extends BaseJob
{
    public function handle(): void
    {
        $data = $this->getPayload();
        
        $message = new MailMessage();
        $message->to($data['to'])
                ->subject($data['subject'])
                ->body($data['body']);
        
        $mailer = $this->app->make(MailManager::class);
        $mailer->send($message);
    }
}

// Dispatch the job
$dispatcher = $app->resolve('queue.dispatcher');
$job = new SendEmailJob($app, [
    'to' => 'user@example.com',
    'subject' => 'Welcome!',
    'body' => 'Thanks for signing up.',
], ['queue' => 'emails']);

$dispatcher->dispatch($job);

Processing Images

class ProcessImageJob extends BaseJob
{
    public function handle(): void
    {
        $data = $this->getPayload();
        $imagePath = $data['path'];
        
        // Load image
        $image = imagecreatefromjpeg($imagePath);
        
        // Resize
        $resized = imagescale($image, 800, 600);
        
        // Save
        imagejpeg($resized, $data['output_path'], 85);
        
        // Cleanup
        imagedestroy($image);
        imagedestroy($resized);
        
        $this->addMetadata('processed_at', date('Y-m-d H:i:s'));
    }
}

Generating Reports

class GenerateReportJob extends BaseJob
{
    public function handle(): void
    {
        $data = $this->getPayload();
        $reportType = $data['type'];
        $dateRange = $data['date_range'];
        
        // Generate report
        $report = $this->generateReport($reportType, $dateRange);
        
        // Save to file
        $path = storage_path("reports/{$reportType}_" . date('Ymd') . ".pdf");
        file_put_contents($path, $report);
        
        // Notify user
        $this->addMetadata('report_path', $path);
        $this->addMetadata('completed_at', date('Y-m-d H:i:s'));
    }
    
    protected function generateReport(string $type, array $range): string
    {
        // Report generation logic
        return $pdfContent;
    }
}

Chained Jobs

// Dispatch a chain of jobs
$dispatcher = $app->resolve('queue.dispatcher');

$job1 = new DownloadFileJob($app, ['url' => $url]);
$job2 = new ProcessFileJob($app, ['depends_on' => $job1->getId()]);
$job3 = new CleanupJob($app, ['depends_on' => $job2->getId()]);

$dispatcher->dispatch($job1);
$dispatcher->dispatchWithDelay($job2, 5);  // Wait 5s for job1
$dispatcher->dispatchWithDelay($job3, 10); // Wait 10s for job2

Best Practices

1. Keep Jobs Small and Focused

// Good: Single responsibility
class SendWelcomeEmailJob extends BaseJob { /* ... */ }
class SendVerificationEmailJob extends BaseJob { /* ... */ }

// Bad: Multiple responsibilities
class SendAllEmailsJob extends BaseJob { /* ... */ }

2. Make Jobs Idempotent

public function handle(): void
{
    $orderId = $this->getPayload()['order_id'];
    
    // Check if already processed
    if ($this->isAlreadyProcessed($orderId)) {
        return;
    }
    
    // Process...
}

3. Handle Failures Gracefully

public function failed(JobException $exception): void
{
    // Log with context
    $this->getLogger()->error('Job failed', [
        'job_id' => $this->getId(),
        'payload' => $this->getPayload(),
        'exception' => $exception->getMessage(),
    ]);
    
    // Notify relevant parties
    // Update database status
    // Move to dead letter queue
}

4. Set Appropriate Timeouts

// Quick job - short timeout
$job = new SendEmailJob($app, $payload, ['timeout' => 30]);

// Long job - longer timeout
$job = new GenerateReportJob($app, $payload, ['timeout' => 300]);

5. Use Queues Wisely

// High priority - process immediately
$dispatcher->dispatchToQueue($urgentJob, 'high');

// Normal priority
$dispatcher->dispatchToQueue($normalJob, 'default');

// Low priority - batch processing
$dispatcher->dispatchToQueue($bulkJob, 'low');

6. Monitor Your Queues

# Set up monitoring
* * * * * php artisan queue:status >> /var/log/queue-status.log
* * * * * php artisan queue:health || echo "Queue unhealthy!" | mail -s "Alert" admin@example.com

Security Considerations

1. Validate Payloads

public function handle(): void
{
    $data = $this->getPayload();
    
    // Validate required fields
    if (!isset($data['user_id']) || !is_int($data['user_id'])) {
        throw new JobException('Invalid user_id in payload');
    }
    
    // Sanitize inputs
    $email = filter_var($data['email'], FILTER_VALIDATE_EMAIL);
    if (!$email) {
        throw new JobException('Invalid email address');
    }
}

2. Don't Store Sensitive Data

// Bad: Password in payload
$job = new CreateUserJob($app, [
    'email' => $email,
    'password' => $password,  // Don't do this!
]);

// Good: Reference only
$job = new CreateUserJob($app, [
    'user_id' => $user->id,  // Fetch password from secure storage
]);

3. Secure Worker Processes

# Run workers as non-root user
sudo -u www-data php artisan queue:work

# Set appropriate file permissions
chmod 600 storage/queue/*.pid
chmod 700 storage/queue

4. Use Compression for Large Payloads

$job = new ProcessLargeDataJob($app, $largeData, [
    'compressPayload' => true,
]);

Performance Tuning

1. Optimize Database Driver

-- Add indexes for faster queries
CREATE INDEX idx_jobs_queue_available ON jobs(queue, available_at);
CREATE INDEX idx_jobs_reserved ON jobs(reserved_at) WHERE reserved_at > 0;

2. Tune Worker Settings

# High-throughput configuration
php artisan queue:work \
    --queue=high \
    --sleep=1 \
    --memory=256 \
    --timeout=120

3. Use Redis for High Volume

Redis is significantly faster than database for high-throughput scenarios:

QUEUE_DRIVER=redis

4. Batch Processing

// Process jobs in batches
$worker->processBatch('default', 25);

5. Connection Pooling

For database driver, use connection pooling:

$pool = new ConnectionPool();
$connection = $pool->getConnection('database', $config);

Migration Guide

From Sync to Queue

  1. Identify slow operations:
// Before: Blocks request
$mailer->send($message);

// After: Async
$dispatcher->dispatch(new SendEmailJob($app, ['message' => $message->toArray()]));
  1. Update code to use dispatcher

  2. Configure queue driver

  3. Start workers

  4. Monitor for failures

From Other Systems

If migrating from Laravel Queue or similar:

Laravel Machinjiri
dispatch(new Job()) $dispatcher->dispatch($job)
dispatch()->onQueue('emails') $dispatcher->dispatchToQueue($job, 'emails')
dispatch()->delay(60) $dispatcher->dispatchWithDelay($job, 60)
$this->job $this (BaseJob)
$this->attempts() $this->getAttempts()

Contributing

We welcome contributions! Please see our contributing guidelines for details.

Development Setup

# Clone the repository
git clone https://github.com/mlangeni/machinjiri.git
cd machinjiri

# Install dependencies
composer install

# Run tests
composer test

# Run queue tests specifically
composer test -- --group=queue

Running Tests

# Unit tests
./vendor/bin/phpunit tests/Unit/Queue

# Integration tests
./vendor/bin/phpunit tests/Integration/Queue

# All tests
./vendor/bin/phpunit

Clone this wiki locally