Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queues and sleep option #13932

Closed
dragosperca opened this issue Jun 10, 2016 · 6 comments
Closed

Queues and sleep option #13932

dragosperca opened this issue Jun 10, 2016 · 6 comments

Comments

@dragosperca
Copy link

dragosperca commented Jun 10, 2016

By default, the sleep option is set to 3 seconds. In some scenarios, setting it to lower values can decrease the time it takes for workers to do pending jobs.

Currently, you can't set it lower than 1 second, since sleep() function is used, instead of usleep().

Say you're developing a search engine, and you spin up 10 workers that listen for incoming jobs. Supervisor will ensure jobs are up and running, and we set the sleep value to 1 second. Once 50 jobs have been created, workers will start to consume them. But if a worker has already started sleeping while the job was created, it has to wait for up to a second before it sees new jobs as available.

The statistical chances of all workers being on the same time horizon is small. So a worker can already work on those 50 jobs while some 9 workers are still asleep (that 1 second).

In average, this affects the search engine performance with up to a second. An extra second for a user waiting for results to show is a lot.

Can you allow sleep to be lower than 1? I understand that in doing so, the CPU load can increase. But a value of, say 0.2 is still fine.

@GrahamCampbell
Copy link
Member

We don't sleep in between jobs. We only sleep if nothing happened the last time.

@GrahamCampbell
Copy link
Member

If your jobs take less than 0.02 seconds to run, you probably don't need to worry about this.

@dragosperca
Copy link
Author

jobs usually take 0.2 - 0.3 seconds. Their number, could be in the range of, 50 - 100. This sleep parameter can affect the total duration with up to a second. Say now I press "Search", the system creates 50 jobs that need to be worked on. All workers are still sleeping (since no previous jobs were executed) and won't wake up for a second. This extra second is time we cannot optimise now.
Even if not all workers are sleeping, the number of the ones that are awake is too small to compensate.

@ph4r05
Copy link
Contributor

ph4r05 commented May 23, 2017

I had exactly same issue and ended up by implementing custom worker that can sleep in sub-second precisions:

<?php

namespace App;
use Illuminate\Queue\Worker;

class Ph4Worker extends Worker
{
    /**
     * Sleep the script for a given number of seconds.
     *
     * @param  int   $seconds
     * @return void
     */
    public function sleep($seconds)
    {
        usleep(floatval($seconds) * 1e6);
    }
}

Then just register it in custom QueueServiceProvider:

<?php

namespace App\Providers;

use App\Ph4Worker;
use Illuminate\Support\ServiceProvider;
use Illuminate\Contracts\Debug\ExceptionHandler;

class QueueServiceProvider extends ServiceProvider
{
    /**
     * Define your route model bindings, pattern filters, etc.
     *
     * @return void
     */
    public function boot()
    {
        $this->app->singleton('queue.worker', function () {
            return new Ph4Worker(
                $this->app['queue'], $this->app['events'], $this->app[ExceptionHandler::class]
            );
        });
    }
}

then invoke with desired sleep value

php artisan queue:work --sleep 0.05

@dragosperca
Copy link
Author

sounds great, will try this out

@timogoosen
Copy link

I'm bit late to the party,
but you can just open a database connection in your queue worker job and do:

select sleep(1);

It would look something like this:

// in your handle function
        $results = DB::select('select sleep(1)');

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants