Repository navigation
Queue and Jobs
The Machinjiri Queue-Job System is a robust, feature-rich background job processing framework that allows you to defer time-consuming tasks to be processed asynchronously. Whether you're sending emails, processing images, generating reports, or handling any other resource-intensive operation, this system provides the tools you need.
Queues solve several critical problems in web applications:
- Improved Response Times: Offload slow operations so users don't wait
- Reliability: Retry failed jobs automatically with exponential backoff
- Scalability: Distribute work across multiple workers and servers
- Resource Management: Control memory usage and prevent runaway processes
- Observability: Track job status, failures, and performance metrics
| Feature | Description |
|---|---|
| Multiple Drivers | Database, Redis, File, Memory, and Sync drivers |
| Job Retries | Automatic retry with configurable backoff strategies |
| Delayed Jobs | Schedule jobs to run after a specified delay |
| Priority Queues | Process high-priority jobs first |
| Job Batching | Process multiple jobs efficiently |
| Failure Handling | Comprehensive failed job storage and retry |
| Payload Compression | Optional compression for large payloads |
| Event System | Hooks for job lifecycle events |
| Feature | Description |
|---|---|
| Daemon Mode | Long-running worker processes |
| Memory Limits | Auto-restart workers exceeding memory limits |
| Job Limits | Process a specific number of jobs then exit |
| Graceful Shutdown | Handle SIGINT/SIGTERM for clean shutdown |
| Heartbeat Monitoring | Track worker health status |
| Auto-Restart | Supervisor restarts failed workers |
| Concurrency Control | Run multiple worker instances |
Run the initialization command to set up the queue infrastructure:
php artisan queue:initThis command will:
- Create necessary queue driver files
- Generate configuration files
- Set up database tables (if using database driver)
- Create the QueueServiceProvider
Edit config/queue.php to configure your preferred driver:
<?php
return [
'default' => env('QUEUE_DRIVER', 'database'),
'drivers' => [
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'failed_table' => 'failed_jobs',
'processed_table' => 'processed_jobs',
],
'redis' => [
'driver' => 'redis',
'host' => env('REDIS_HOST', '127.0.0.1'),
'port' => env('REDIS_PORT', 6379),
'password' => env('REDIS_PASSWORD', null),
'database' => env('REDIS_DATABASE', 0),
'queue' => 'default',
'retry_after' => 90,
],
],
];Add to your .env file:
QUEUE_DRIVER=database
# Worker settings
QUEUE_WORKER_MAX_MEMORY=128
QUEUE_WORKER_MAX_JOBS=500
QUEUE_WORKER_CHECK_INTERVAL=5
QUEUE_WORKER_HEARTBEAT_TTL=15
QUEUE_WORKER_GRACE_PERIOD=10
QUEUE_WORKER_STOP_ON_EMPTY=falseCheck that everything is set up correctly:
php artisan queue:healthphp artisan queue:make-job SendWelcomeEmail --queue=emailsThis creates app/Jobs/SendWelcomeEmailJob.php:
<?php
namespace App\Jobs;
use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseJob;
use Mlangeni\Machinjiri\Core\Exceptions\ContractsException\JobException;
class SendWelcomeEmailJob extends BaseJob
{
public function __construct(
\Mlangeni\Machinjiri\Core\Container $app,
array $payload = [],
array $options = []
) {
$defaultOptions = [
'maxAttempts' => 3,
'queue' => 'emails',
'timeout' => 60,
];
parent::__construct($app, $payload, array_merge($defaultOptions, $options));
}
public function handle(): void
{
$data = $this->getPayload();
$email = $data['email'] ?? null;
$name = $data['name'] ?? 'User';
// Send the welcome email
// Your email sending logic here
$this->addMetadata('sent_at', date('Y-m-d H:i:s'));
}
public function failed(JobException $exception): void
{
// Handle failure - log, notify, etc.
}
}use App\Jobs\SendWelcomeEmailJob;
// Get the dispatcher
$dispatcher = $app->resolve('queue.dispatcher');
// Create and dispatch the job
$job = new SendWelcomeEmailJob($app, [
'email' => 'user@example.com',
'name' => 'John Doe'
]);
$jobId = $dispatcher->dispatch($job);# Start a worker for the emails queue
php artisan queue:work --queue=emailsThat's it! Your job will now be processed by the worker.
Jobs are the fundamental unit of work in the queue system. Each job is a PHP class that extends BaseJob and implements a handle() method.
Job Lifecycle:
Created → Queued → Reserved → Processing → Completed
↓
Failed → Retry → ...
↓
Permanently Failed
Queues are named channels that hold jobs waiting to be processed. You can have multiple queues for different job types:
-
default- General purpose jobs -
emails- Email sending jobs -
images- Image processing jobs -
reports- Report generation jobs -
high- High-priority jobs
Workers are processes that pull jobs from queues and execute them. Workers can run:
- Single-run mode: Process available jobs then exit
- Daemon mode: Run continuously until stopped
- With limits: Stop after N jobs or memory usage
Drivers determine how jobs are stored and retrieved:
| Driver | Use Case | Pros | Cons |
|---|---|---|---|
database |
General purpose | Persistent, queryable | Slower than Redis |
redis |
High performance | Very fast, feature-rich | Requires Redis |
file |
Simple setups | No dependencies | Not for high volume |
memory |
Testing | Instant | Not persistent |
sync |
Development | Immediate execution | Blocks request |
The database driver stores jobs in SQL tables, providing persistence and queryability.
Setup:
php artisan queue:initThis creates three tables:
-
jobs- Pending jobs -
failed_jobs- Failed jobs -
processed_jobs- Completed jobs
Configuration:
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'failed_table' => 'failed_jobs',
'processed_table' => 'processed_jobs',
],Features:
- Job reservation with timeout
- Automatic retry of stale jobs
- Full job history
- Transaction support
The Redis driver provides high-performance job storage using Redis lists and sorted sets.
Setup:
# Install Redis extension
pecl install redis
# Or use Predis
composer require predis/predisConfiguration:
'redis' => [
'driver' => 'redis',
'host' => env('REDIS_HOST', '127.0.0.1'),
'port' => env('REDIS_PORT', 6379),
'password' => env('REDIS_PASSWORD', null),
'database' => env('REDIS_DATABASE', 0),
'prefix' => 'queue:',
'retry_after' => 90,
'timeout' => 2.5,
],Features:
- Atomic operations
- Delayed jobs with sorted sets
- Reserved job tracking
- Pub/sub notifications (optional)
The file driver stores jobs as JSON files, suitable for simple deployments.
Configuration:
'file' => [
'driver' => 'file',
'path' => __DIR__ . '/../storage/queue',
'retry_after' => 90,
],Directory Structure:
storage/queue/
├── pending/ # Jobs waiting to be processed
├── processing/ # Jobs currently being processed
├── failed/ # Failed jobs
└── processed/ # Completed jobs
The memory driver stores jobs in PHP arrays, useful for testing.
Configuration:
'memory' => [
'driver' => 'memory',
'retry_after' => 90,
],Note: Jobs are lost when the process ends. Use only for testing.
The sync driver executes jobs immediately, blocking the request.
Configuration:
'sync' => [
'driver' => 'sync',
],Use Cases:
- Development environments
- Testing
- Jobs that must complete before response
php artisan queue:make-job ProcessOrderCreates app/Jobs/ProcessOrderJob.php:
<?php
namespace App\Jobs;
use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseJob;
class ProcessOrderJob extends BaseJob
{
public function handle(): void
{
$payload = $this->getPayload();
$orderId = $payload['order_id'] ?? null;
// Process the order
$order = $this->app->resolve('db')->table('orders')->find($orderId);
$order->status = 'processing';
$order->save();
// ... more processing
$this->addMetadata('processed_at', date('Y-m-d H:i:s'));
}
}When creating a job, you can specify various options:
$job = new ProcessOrderJob($app, $payload, [
'queue' => 'orders', // Queue name
'maxAttempts' => 5, // Maximum retry attempts
'timeout' => 120, // Timeout in seconds
'delay' => 60, // Initial delay in seconds
'retryDelay' => 30, // Base retry delay
'compressPayload' => true, // Compress large payloads
'metadata' => [ // Custom metadata
'user_id' => 123,
'source' => 'api',
],
]);The generator supports different job templates:
# Standard job
php artisan queue:make-job ProcessData
# Email job
php artisan queue:make-job SendNewsletter --type=email --queue=emails
# Model job with database
php artisan queue:make-job SyncUser --type=model --database<?php
namespace App\Jobs;
use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseJob;
use Mlangeni\Machinjiri\Core\Transport\Mail\MailMessage;
use Mlangeni\Machinjiri\Core\Transport\Mail\MailManager;
class SendNewsletterJob extends BaseJob
{
public function handle(): void
{
$data = $this->getPayload();
$messageData = $data['message'] ?? [];
$message = MailMessage::fromArray($messageData);
$mailManager = $this->app->make(MailManager::class);
$response = $mailManager->send($message);
$this->addMetadata('message_id', $response->getMessageId());
}
}$dispatcher = $app->resolve('queue.dispatcher');
$job = new ProcessOrderJob($app, ['order_id' => 123]);
$jobId = $dispatcher->dispatch($job);$jobId = $dispatcher->dispatchToQueue($job, 'high-priority');// Delay 60 seconds
$jobId = $dispatcher->dispatchWithDelay($job, 60);$result = $dispatcher->dispatchNow($job);$jobs = [
new SendEmailJob($app, ['to' => 'user1@example.com']),
new SendEmailJob($app, ['to' => 'user2@example.com']),
new SendEmailJob($app, ['to' => 'user3@example.com']),
];
$jobIds = $dispatcher->dispatchBulk($jobs);// Via dispatcher
$queue = $app->resolve('queue');
$queue->pushWithPriority($job, 'default', 10); // Priority 10
// Via job metadata
$job->addMetadata('priority', 10);php artisan queue:work| Option | Short | Description | Default |
|---|---|---|---|
--queue |
Queue name | default |
|
--sleep |
-s |
Seconds to sleep when empty | 3 |
--memory |
-m |
Memory limit in MB | 128 |
--timeout |
-t |
Job timeout in seconds | 60 |
--max-jobs |
Max jobs before exit | unlimited | |
--stop-on-empty |
Stop when queue is empty | false |
|
--tries |
Max attempts per job | 3 |
|
--daemon |
Run continuously | false |
|
--force |
-f |
Run in maintenance mode | false |
# Process emails queue
php artisan queue:work --queue=emails
# High-throughput worker
php artisan queue:work --queue=high --sleep=1 --memory=256
# Process specific number of jobs
php artisan queue:work --max-jobs=100
# One-time processing (stop when empty)
php artisan queue:work --stop-on-empty
# Daemon mode with PID file
php artisan queue:work --daemon --pid-file=/var/run/worker.pidWorkers respond to Unix signals:
-
SIGTERM/SIGINT: Graceful shutdown (finish current job) -
SIGKILL: Force kill (may leave job in reserved state)
# Start single worker
php artisan queue:worker-start
# Start multiple workers
php artisan queue:worker-start --concurrency=4
# For specific queue
php artisan queue:worker-start --queue=emails --concurrency=2php artisan queue:worker-status --queue=defaultOutput:
+----------+-------+---------+---------+-------------+---------------------+
| Instance | PID | Running | Healthy | Memory (MB) | Last Heartbeat |
+----------+-------+---------+---------+-------------+---------------------+
| 1 | 12345 | Yes | Yes | 45.2 | 2024-01-15 10:30:00 |
| 2 | 12346 | Yes | Yes | 42.8 | 2024-01-15 10:30:00 |
+----------+-------+---------+---------+-------------+---------------------+
# Stop all workers for queue
php artisan queue:worker-stop --queue=default
# Stop specific instance
php artisan queue:worker-stop --queue=default --instance=1php artisan queue:worker-restart --queue=default --concurrency=4php artisan queue:worker-cleanupThe supervisor monitors workers and restarts them if they fail:
# Run supervisor in daemon mode
php artisan queue:supervisor --queue=default --concurrency=4 --daemonWhat the Supervisor Does:
- Checks worker status every N seconds
- Restarts dead workers
- Restarts unhealthy workers (stale heartbeat)
- Restarts workers exceeding memory limit
Create /etc/systemd/system/queue-worker.service:
[Unit]
Description=Machinjiri Queue Worker
After=network.target
[Service]
Type=simple
User=www-data
WorkingDirectory=/var/www/app
ExecStart=/usr/bin/php artisan queue:supervisor --queue=default --concurrency=4 --daemon
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.targetEnable and start:
sudo systemctl enable queue-worker
sudo systemctl start queue-workerCreate /etc/supervisor/conf.d/queue-worker.conf:
[program:queue-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work --daemon
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/worker.log
stopwaitsecs=3600# Check queue health
php artisan queue:health
# Check database connection
php artisan queue:db-infophp artisan queue:statusOutput:
+---------+----------+---------+-------+---------+
| Pending | Reserved | Delayed | Total | Health |
+---------+----------+---------+-------+---------+
| 45 | 4 | 12 | 61 | Healthy |
+---------+----------+---------+-------+---------+
php artisan queue:failedOutput:
+----+--------+---------+---------------------+-----------------+
| No | Job ID | Queue | Failed At | Error |
+----+--------+---------+---------------------+-----------------+
| 1 | job_01 | emails | 2024-01-15 10:30:00 | Connection ti...|
| 2 | job_02 | default | 2024-01-15 10:25:00 | Invalid argum...|
+----+--------+---------+---------------------+-----------------+
# Retry specific job
php artisan queue:retry job_01
# Retry all failed jobs
php artisan queue:retry all
# Retry jobs from specific queue
php artisan queue:retry all --queue=emails# Remove specific job
php artisan queue:forget job_01
# Flush all failed jobs
php artisan queue:flushJobs are retried automatically with exponential backoff:
Attempt 1: Fail → Wait 60s
Attempt 2: Fail → Wait 120s (60 * 2^1)
Attempt 3: Fail → Wait 240s (60 * 2^2)
Attempt 4: Fail → Permanently failed
Configure retry delay:
$job = new MyJob($app, $payload, [
'maxAttempts' => 5,
'retryDelay' => 30, // Base delay in seconds
]);config/queue.php:
<?php
return [
'default' => env('QUEUE_DRIVER', 'database'),
'drivers' => [
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'failed_table' => 'failed_jobs',
'processed_table' => 'processed_jobs',
],
'redis' => [
'driver' => 'redis',
'host' => env('REDIS_HOST', '127.0.0.1'),
'port' => env('REDIS_PORT', 6379),
'password' => env('REDIS_PASSWORD', null),
'database' => env('REDIS_DATABASE', 0),
'queue' => 'default',
'retry_after' => 90,
'block_for' => null,
],
'file' => [
'driver' => 'file',
'path' => __DIR__ . '/../storage/queue',
'retry_after' => 90,
],
'memory' => [
'driver' => 'memory',
'retry_after' => 90,
],
'sync' => [
'driver' => 'sync',
],
],
'failed' => [
'driver' => 'database',
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];config/jobs.php:
<?php
return [
'defaults' => [
'queue' => 'default',
'max_attempts' => 3,
'timeout' => 60,
'retry_delay' => 60,
],
'jobs' => [
'send_email' => [
'queue' => 'emails',
'max_attempts' => 5,
'timeout' => 120,
'class' => App\Jobs\SendEmailJob::class,
],
'process_image' => [
'queue' => 'images',
'max_attempts' => 3,
'timeout' => 180,
'class' => App\Jobs\ProcessImageJob::class,
],
],
'queue_priority' => [
'high',
'default',
'low',
'emails',
'notifications',
],
];# Driver
QUEUE_DRIVER=database
# Worker Settings
QUEUE_WORKER_MAX_MEMORY=128
QUEUE_WORKER_MAX_JOBS=500
QUEUE_WORKER_CHECK_INTERVAL=5
QUEUE_WORKER_HEARTBEAT_TTL=15
QUEUE_WORKER_HEARTBEAT_INTERVAL=60
QUEUE_WORKER_GRACE_PERIOD=10
QUEUE_WORKER_STOP_ON_EMPTY=false
# Redis (if using Redis driver)
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DATABASE=0Create a custom driver by extending BaseQueue:
<?php
namespace App\Queue\Drivers;
use Mlangeni\Machinjiri\Core\Artisans\Contracts\BaseQueue;
use Mlangeni\Machinjiri\Core\Artisans\Contracts\JobInterface;
class CustomQueue extends BaseQueue
{
protected array $config = [];
public function __construct($app, string $name, array $config = [])
{
parent::__construct($app, $name, $config);
$this->config = array_merge([
// Default configuration
], $config);
}
public function push(JobInterface $job, string $queue = 'default', int $delay = 0): string
{
// Implement job storage
$jobId = $job->getId();
$serialized = json_encode($job->serialize());
// Store in your backend
return $jobId;
}
public function pop(string $queue = 'default'): ?JobInterface
{
// Implement job retrieval
return null;
}
// Implement other required methods...
}Add middleware to jobs for cross-cutting concerns:
class MyJob extends BaseJob
{
public function handle(): void
{
// Pre-processing
$this->beforeHandle();
try {
// Main logic
$this->process();
} finally {
// Post-processing
$this->afterHandle();
}
}
protected function beforeHandle(): void
{
$this->addMetadata('started_at', microtime(true));
}
protected function afterHandle(): void
{
$started = $this->getMetadata()['started_at'] ?? 0;
$duration = microtime(true) - $started;
$this->addMetadata('duration', $duration);
}
}Listen to job lifecycle events:
use Mlangeni\Machinjiri\Core\Artisans\Events\EventListener;
$events = new EventListener();
$events->listen('job.completed', function($data) {
// Log completed job
error_log("Job {$data['job_id']} completed");
});
$events->listen('job.failed', function($data) {
// Alert on failure
error_log("Job {$data['job_id']} failed: {$data['exception']}");
});For jobs with large payloads:
$job = new ProcessLargeDataJob($app, $largeData, [
'compressPayload' => true,
]);Process jobs in batches for efficiency:
$worker = $app->resolve('queue.worker');
$processed = $worker->processBatch('default', 10);For long-running workers, implement memory checks:
class MyJob extends BaseJob
{
public function handle(): void
{
// Check memory before heavy operation
if (memory_get_usage() > 100 * 1024 * 1024) {
throw new JobException('Memory limit approaching, retrying later');
}
// Process...
}
}| Command | Description |
|---|---|
queue:init |
Initialize queue system |
queue:make |
Create queue driver |
queue:make-job |
Create job class |
queue:status |
Show queue status |
queue:health |
Check queue health |
queue:db-info |
Show database info |
| Command | Description |
|---|---|
queue:work |
Start worker |
queue:worker-start |
Start managed workers |
queue:worker-stop |
Stop managed workers |
queue:worker-restart |
Restart workers |
queue:worker-status |
Show worker status |
queue:worker-cleanup |
Clean orphaned files |
queue:supervisor |
Run supervisor |
| Command | Description |
|---|---|
queue:failed |
List failed jobs |
queue:retry |
Retry failed job |
queue:forget |
Remove failed job |
queue:flush |
Clear failed jobs |
| Command | Description |
|---|---|
queue:clear |
Clear queue |
queue:job |
Process specific job type |
-
Check worker is running:
php artisan queue:worker-status
-
Check queue has jobs:
php artisan queue:status
-
Check worker logs:
tail -f storage/logs/queue-*.log -
Verify queue name matches:
// Job queue $job = new MyJob($app, $payload, ['queue' => 'emails']); // Worker queue php artisan queue:work --queue=emails
php artisan queue:db-infoIf you see "No such file or directory":
- Change
DB_HOST=localhosttoDB_HOST=127.0.0.1 - Or provide the correct socket path
Workers restart automatically when exceeding memory limit. To increase:
php artisan queue:work --memory=256Or set in .env:
QUEUE_WORKER_MAX_MEMORY=256If jobs are stuck in "reserved" state:
- Check if worker is still running
- Wait for
retry_aftertimeout - Or manually clear and re-queue
// Check reserved jobs
$stats = $queue->getStats('default');
echo "Reserved: " . $stats['reserved'];
// Wait for retry_after or restart workerEnsure storage directories are writable:
chmod -R 755 storage/queue
chown -R www-data:www-data storage/queueEnable verbose output:
php artisan queue:work -vQueue logs are stored in storage/logs/:
-
queue-supervisor-*.log- Supervisor logs -
queue-worker-*.log- Worker logs -
queue-processor-*.log- Processor logs -
worker-{driver}-{queue}.log- Individual worker output
abstract class BaseJob implements JobInterface
{
// Identification
public function getId(): string;
public function getName(): string;
// Payload
public function getPayload(): array;
// Attempts
public function getAttempts(): int;
public function incrementAttempts(): void;
public function getMaxAttempts(): int;
public function getNextRetryDelay(): int;
// Queue
public function getQueue(): string;
public function getDelay(): int;
public function getTimeout(): int;
public function getRetryDelay(): int;
// Metadata
public function getMetadata(): array;
public function setMetadata(array $metadata): void;
public function addMetadata(string $key, $value): void;
// Lifecycle
abstract public function handle(): void;
public function failed(JobException $exception): void;
// Serialization
public function serialize(): array;
public static function unserialize(array $data, ?Container $app = null): self;
}abstract class BaseQueue implements QueueInterface
{
// Push
public function push(JobInterface $job, string $queue = 'default', int $delay = 0): string;
public function bulk(array $jobs, string $queue = 'default', int $delay = 0): array;
public function pushWithPriority(JobInterface $job, string $queue = 'default', int $priority = 0, int $delay = 0): string;
// Pop
public function pop(string $queue = 'default'): ?JobInterface;
public function popBatch(string $queue = 'default', int $batchSize = 10): array;
// Release/Delete
public function release(JobInterface $job, string $queue = 'default', int $delay = 0): bool;
public function delete(JobInterface $job, string $queue = 'default'): bool;
// Status
public function size(string $queue = 'default'): int;
public function getQueues(): array;
public function getStats(string $queue = 'default'): array;
public function isHealthy(): bool;
public function getHealthMetrics(string $queue = 'default'): array;
// Failed Jobs
public function getFailed(string $queue = 'default', int $limit = 50, int $offset = 0): array;
public function retryFailed(string $jobId, string $queue = 'default'): bool;
public function forgetFailed(string $jobId, string $queue = 'default'): bool;
public function flushFailed(string $queue = 'default'): int;
// Clear
public function clear(string $queue = 'default'): int;
}class BaseWorker implements WorkerInterface
{
// Lifecycle
public function start(string $queue, array $options = []): void;
public function stop(): void;
public function pause(): void;
public function resume(): void;
public function isRunning(): bool;
// Status
public function getStatus(): array;
// Processing
public function processNextJob(string $queue): bool;
public function run(string $queue, ?int $maxJobs = null): int;
public function processBatch(string $queue = 'default', int $batchSize = 10): int;
// Heartbeat
public function setHeartbeatCallback(callable $callback, int $intervalSeconds = 60): void;
}class BaseJobDispatcher implements JobDispatcherInterface
{
// Dispatch
public function dispatch(JobInterface $job): string;
public function dispatchToQueue(JobInterface $job, string $queue): string;
public function dispatchWithDelay(JobInterface $job, int $delay): string;
public function dispatchNow(JobInterface $job): mixed;
public function dispatchBulk(array $jobs): array;
// Configuration
public function getDefaultQueue(): string;
public function setDefaultQueue(string $queue): void;
}class BackgroundWorkerManager
{
// Configuration
public function reloadConfig(): void;
// Worker Management
public function startWorker(string $queue, string $driver = 'default', int $concurrency = 1): int;
public function stopWorker(string $queue, string $driver = 'default', ?int $instance = null);
public function restartWorker(string $queue, string $driver = 'default', ?int $instance = null, int $concurrency = 1): int;
public function workerStatus(string $queue, string $driver = 'default', ?int $instance = null): array;
// Monitoring
public function monitorWorker(string $queue, string $driver = 'default', int $concurrency = 1, ?callable $output = null): void;
public function updateHeartbeat(string $queue, string $driver = 'default', ?int $instance = null): void;
// Cleanup
public function cleanupOrphanedPids(): void;
}// Create the job
class SendEmailJob extends BaseJob
{
public function handle(): void
{
$data = $this->getPayload();
$message = new MailMessage();
$message->to($data['to'])
->subject($data['subject'])
->body($data['body']);
$mailer = $this->app->make(MailManager::class);
$mailer->send($message);
}
}
// Dispatch the job
$dispatcher = $app->resolve('queue.dispatcher');
$job = new SendEmailJob($app, [
'to' => 'user@example.com',
'subject' => 'Welcome!',
'body' => 'Thanks for signing up.',
], ['queue' => 'emails']);
$dispatcher->dispatch($job);class ProcessImageJob extends BaseJob
{
public function handle(): void
{
$data = $this->getPayload();
$imagePath = $data['path'];
// Load image
$image = imagecreatefromjpeg($imagePath);
// Resize
$resized = imagescale($image, 800, 600);
// Save
imagejpeg($resized, $data['output_path'], 85);
// Cleanup
imagedestroy($image);
imagedestroy($resized);
$this->addMetadata('processed_at', date('Y-m-d H:i:s'));
}
}class GenerateReportJob extends BaseJob
{
public function handle(): void
{
$data = $this->getPayload();
$reportType = $data['type'];
$dateRange = $data['date_range'];
// Generate report
$report = $this->generateReport($reportType, $dateRange);
// Save to file
$path = storage_path("reports/{$reportType}_" . date('Ymd') . ".pdf");
file_put_contents($path, $report);
// Notify user
$this->addMetadata('report_path', $path);
$this->addMetadata('completed_at', date('Y-m-d H:i:s'));
}
protected function generateReport(string $type, array $range): string
{
// Report generation logic
return $pdfContent;
}
}// Dispatch a chain of jobs
$dispatcher = $app->resolve('queue.dispatcher');
$job1 = new DownloadFileJob($app, ['url' => $url]);
$job2 = new ProcessFileJob($app, ['depends_on' => $job1->getId()]);
$job3 = new CleanupJob($app, ['depends_on' => $job2->getId()]);
$dispatcher->dispatch($job1);
$dispatcher->dispatchWithDelay($job2, 5); // Wait 5s for job1
$dispatcher->dispatchWithDelay($job3, 10); // Wait 10s for job2// Good: Single responsibility
class SendWelcomeEmailJob extends BaseJob { /* ... */ }
class SendVerificationEmailJob extends BaseJob { /* ... */ }
// Bad: Multiple responsibilities
class SendAllEmailsJob extends BaseJob { /* ... */ }public function handle(): void
{
$orderId = $this->getPayload()['order_id'];
// Check if already processed
if ($this->isAlreadyProcessed($orderId)) {
return;
}
// Process...
}public function failed(JobException $exception): void
{
// Log with context
$this->getLogger()->error('Job failed', [
'job_id' => $this->getId(),
'payload' => $this->getPayload(),
'exception' => $exception->getMessage(),
]);
// Notify relevant parties
// Update database status
// Move to dead letter queue
}// Quick job - short timeout
$job = new SendEmailJob($app, $payload, ['timeout' => 30]);
// Long job - longer timeout
$job = new GenerateReportJob($app, $payload, ['timeout' => 300]);// High priority - process immediately
$dispatcher->dispatchToQueue($urgentJob, 'high');
// Normal priority
$dispatcher->dispatchToQueue($normalJob, 'default');
// Low priority - batch processing
$dispatcher->dispatchToQueue($bulkJob, 'low');# Set up monitoring
* * * * * php artisan queue:status >> /var/log/queue-status.log
* * * * * php artisan queue:health || echo "Queue unhealthy!" | mail -s "Alert" admin@example.compublic function handle(): void
{
$data = $this->getPayload();
// Validate required fields
if (!isset($data['user_id']) || !is_int($data['user_id'])) {
throw new JobException('Invalid user_id in payload');
}
// Sanitize inputs
$email = filter_var($data['email'], FILTER_VALIDATE_EMAIL);
if (!$email) {
throw new JobException('Invalid email address');
}
}// Bad: Password in payload
$job = new CreateUserJob($app, [
'email' => $email,
'password' => $password, // Don't do this!
]);
// Good: Reference only
$job = new CreateUserJob($app, [
'user_id' => $user->id, // Fetch password from secure storage
]);# Run workers as non-root user
sudo -u www-data php artisan queue:work
# Set appropriate file permissions
chmod 600 storage/queue/*.pid
chmod 700 storage/queue$job = new ProcessLargeDataJob($app, $largeData, [
'compressPayload' => true,
]);-- Add indexes for faster queries
CREATE INDEX idx_jobs_queue_available ON jobs(queue, available_at);
CREATE INDEX idx_jobs_reserved ON jobs(reserved_at) WHERE reserved_at > 0;# High-throughput configuration
php artisan queue:work \
--queue=high \
--sleep=1 \
--memory=256 \
--timeout=120Redis is significantly faster than database for high-throughput scenarios:
QUEUE_DRIVER=redis// Process jobs in batches
$worker->processBatch('default', 25);For database driver, use connection pooling:
$pool = new ConnectionPool();
$connection = $pool->getConnection('database', $config);- Identify slow operations:
// Before: Blocks request
$mailer->send($message);
// After: Async
$dispatcher->dispatch(new SendEmailJob($app, ['message' => $message->toArray()]));-
Update code to use dispatcher
-
Configure queue driver
-
Start workers
-
Monitor for failures
If migrating from Laravel Queue or similar:
| Laravel | Machinjiri |
|---|---|
dispatch(new Job()) |
$dispatcher->dispatch($job) |
dispatch()->onQueue('emails') |
$dispatcher->dispatchToQueue($job, 'emails') |
dispatch()->delay(60) |
$dispatcher->dispatchWithDelay($job, 60) |
$this->job |
$this (BaseJob) |
$this->attempts() |
$this->getAttempts() |
We welcome contributions! Please see our contributing guidelines for details.
# Clone the repository
git clone https://github.com/mlangeni/machinjiri.git
cd machinjiri
# Install dependencies
composer install
# Run tests
composer test
# Run queue tests specifically
composer test -- --group=queue# Unit tests
./vendor/bin/phpunit tests/Unit/Queue
# Integration tests
./vendor/bin/phpunit tests/Integration/Queue
# All tests
./vendor/bin/phpunitMachinjiri Docs
- Home
- Learn
- Build
-
Tutorials
- Routing
- Controllers and Middleware
- Views and Templates
- Database ORM
- Eloquent Database ORM
- Errors and Exceptions
- Forms and Validation
- Http
- Events and Logging
- Services and DI
- Security
- Authentication
- Caching
- File System
- Date and Time
- Debugging
- Queue and Jobs
- Task Scheduling
- Transport (Mail, SMS)
- LDAP
- Notifications
- Webhooks
- UUID Generations
- Terminal
- Helper Function
- Reference