Pragmatic approach to asyncron jobs for PHP8.x
This PHP project is for all the developers who want a queue system without installing huge dependencies. This project is heavily inspired by the lecture Daemons with PHP by Arne Blankerts from thePHP.cc
If you come to a point in your project that needs asynchronous jobs, you often tend to use big solutions like RabbitMQ or other systems. After realizing that such services have to be maintained and updated, you may search for simpler solutions like the queue package from the Laravel framework. A quick PHPloc will give you something like this:
./tools/phploc vendor/illuminate/
Directories 96
Files 653
Size
Lines of Code (LOC) 108002
Thats right ... 650+ Files with over 100k Lines of code.
Another Example from the "enqueue" package after require enqueue via composer:
Installing doctrine/cache
Installing doctrine/collections
Installing doctrine/event
Installing symfony/polyfill
Installing ramsey/collection
Installing brick/math
Installing ramsey/uuid
Installing queue-interop
Installing psr/cache
Installing doctrine/persistence
Installing doctrine/dbal
Installing enqueue/dbal
These are huge dependency's and have to be maintained. Here comes this package that aims to be as small as possible but deliver the important features:
- Asynchronous jobs.
- No accidentally multiple executions of one job.
- parallel jobs with max number of workers
- event driven
All this features could be archived by using the most basic storage aka files system with some leverage of systemd.
Systemd uses unit files to configure daemons, and path units monitor files and directories for events. When a specified event occurs, a service unit with the same name is executed. I'll demonstrate this with an example. So the concept is to create files in a specific directory and let systemd handle the starting event condition and keep that one unit running to avoid accidentally multiple executions of one job.
- Linux operating system with systemd
- The minimum required PHP version is PHP 8.0
To use this library install with:
composer require tillmannschiffler/simplequeueTo get started with this system, you can begin by making changes to the example files located at /src/Example. These files can be modified to suit your specific needs. For instance, you can customize the Sample Processor according to your desired functionality.
Setup systemd config file like this example
File: /etc/systemd/system/SimpleQueue.path
[Unit]
Description="Monitor for changes in Queue Inbox Path"
[Path]
DirectoryNotEmpty=<THE_DIR_TO_INBOX>
Unit=SimpleQueue.service
[Install]
WantedBy=multi-user.target
File: /etc/systemd/system/SimpleQueue.service
[Unit]
Description="Worker Starter for the Queue/Job handling
[Service]
Type=simple
ExecStart=/usr/bin/php <PATH_TO_YOUR_WORKER.php>
[Install]
WantedBy=multi-user.target
Remeber to reload the daemon and enable these 2 unit to get it to start on boot:
systemctl daemon-reload
systemctl enable SimpleQueue.path SimpleQueue.service
And of course start it:
systemctl start SimpleQueue.path SimpleQueue.service
In production, you will probably install the queue service in the global systemd configuration. For development, you can install it in the user's systemd configuration instead, which also doesn't require root privileges. The following details the steps required:
-
Create
systemdConfiguration Dirmkdir -p ~/.config/systemd/user -
Install Configuration Files
Install the below files, with
<GIT_ROOT>replaced by the absolute path to where the sources are checked out.File: ~/.config/systemd/user/SimpleQueue.path
[Unit] Description=Monitor for changes in Queue Inbox Path [Path] # NOTE: adjust path to the queue/inbox directory DirectoryNotEmpty=<GIT_ROOT>/queue/inbox Unit=SimpleQueue.service [Install] WantedBy=multi-user.targetFile: ~/.config/systemd/user/SimpleQueue.service
[Unit] Description=Worker Starter for the Queue/Job handling [Service] Type=simple # NOTE: adjust path to the worker here ExecStart=/usr/bin/php <GIT_ROOT>/src/Example/SampleWorker.php [Install] WantedBy=multi-user.target -
Start The Inbox Monitor
systemctl --user daemon-reload systemctl --user start SimpleQueue.path
You can check
journalctl --usernow, it should contain a message that the inbox monitor was started.
You can follow the systemd logs to watch the queue in operation.
Just keep journalctl --follow --user --unit SimpleQueue.service running in a terminal.
In order to test the queue, use php src/Example/CreateSomeJobs.php 1, which creates a single job.
In the systemd logs, you should receive four event entries, StartedJob, StartedExecution, JobOutput, and FinishedJob.
To create a job, you have two options: you can either create the job manually and move it to /queue/inbox, or you can make use of the helper file located at /src/Example/CreateSomeJobs.php. This file can help you generate jobs quickly and easily.
Once you have created the job, and you can proceed to test the system by running /src/Example/SingleWorker.php. This will help you ensure that the setup is working as intended. If there are any issues, you can debug the system accordingly. With this simple setup process, you can get started with using this system in no time at all.
Please feel free to mail me with ideas or bugs you found tillmann.schiffler@gmail.com
Since the whole idea to use the file system as backend storage for our jobs. The monitoring is as easy as it sounds.
Get a count of jobs just count the files in the corresponding directorys like:
ls | wc -l
in the inbox directory and so on.
Since this project is using systemd as leverage we can use journal log to optain statistics per time range:
journalctl -u SimpleQueue.path --since "1 minute ago" | grep "Started SimpleQueue.path" | wc -l
To get a throughput of jobs there is also a simple and pragmatic method - use the event subscriber on the "emitFinishedJob" event and write that event to journal of systemd. After that you can use the command above to determine the jobs done per time range.