Async Queue Handler for Craft 3
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
src
.gitignore removed composer.lock from the package Apr 5, 2018
CHANGELOG.md better Windows support Jun 15, 2018
LICENSE.md initial Aug 22, 2017
README.md sponsor notice Jun 5, 2018
composer.json better Windows support Jun 15, 2018
phpstan.neon tuned phpstan & fixed possible logging bug Apr 6, 2018

README.md

Async (Background) Queue

Latest Stable Version Total Downloads Monthly Downloads

With Craft's job queue you can run heavy tasks in the background. Unfortunately, this is not entirely true, when runQueueAutomatically => true (default), the job queue is handled by a ajax (FPM) call. With many jobs in the queue and limited PHP-FPM processes this can break your site.

This plugin replaces Craft's default queue handler and moves queue execution to a non-blocking background process. The command craft queue/run gets executed right after you push a Job to the queue.

Here you can find the initial discussion I started at craftcms/cms.

Sponsor

Development happens in my free time, but also during working hours. Thanks fortrabbit.com!

Requirements

  • Craft 3
  • Permissions to execute a php binary
  • proc_open()

Installation

  1. Install with Composer via composer require ostark/craft-async-queue from your project directory
  2. Install plugin with this command php ./craft install/plugin async-queue or in the Craft Control Panel under Settings > Plugins

Configuration (optional)

The plugin uses symfony/process to execute the php binary. Usually the binary is located in /usr/bin/, but other common locations are auto detected as well. With the ENV var PHP_BINARY you can explicitly set the path, e.g. in your .env file like this:

PHP_BINARY="/usr/local/Cellar/php71/7.1.0_11/bin/php"

By default 2 background processes handle the queue. With the ASYNC_QUEUE_CONCURRENCY ENV var you can modify this behaviour.

# No concurrency
ASYNC_QUEUE_CONCURRENCY=1

# Or max 5 background processes
ASYNC_QUEUE_CONCURRENCY=5

Under the hood: Process list

Empty queue (only php-fpm master is running)

$ ps auxf | grep php

root      2953  0.0  0.0 399552 13520 ?        Ss   12:27   0:00 php-fpm: master process (/etc/php/fpm.conf)

New job pushed (php-fpm master + child + /usr/bin/php daemon started)

$ ps auxf | grep php

root      2953  0.0  0.0 399552 13520 ?        Ss   12:27   0:00 php-fpm: master process (/etc/php/fpm.conf)
app       3031  2.2  0.2 718520 45992 ?        S    12:31   0:00  \_ php-fpm: pool www
app       3033  1.2  0.2 280936 32808 ?        S    12:31   0:00 /usr/bin/php craft queue/run
app       3034  0.0  0.0   4460   784 ?        S    12:31   0:00  \_ sh -c /usr/bin/php craft queue/exec "1234" "0" "1"
app       3035  1.2  0.2 280928 32280 ?        S    12:31   0:00      \_ /usr/bin/php craft queue/exec 1234 0 1