Provide queue component

[samdark]

Useful for background jobs and all projects processing relatively large amounts of data.

Implementation can be designed the same way as cache i.e. base API + drivers such as:

• "immediate execution" (for local development)
• db
• http://aws.amazon.com/sqs/

Proposed base API:

abstract class Queue {
  // add job to the queue. $data will be passed as an argument to $callback and should be serializable
  public function push($callback, $data)
  {
  }

  // get a job and lock it queue for further processing
  public function get()
  {
  }
}

[bwoester]

Another implementation might be based on System V message queues if PHP was compiled with --enable-sysvmsg

[cebe]

the local thing may use http://de2.php.net/manual/en/class.splqueue.php

[petrabarus]

I really agree with this one.
The implementation on my project now is directly using AWS SDK for PHP v2. But if there is another layer of abstraction provided by Yii and an implementation for local development, I think this is really good.

[resurtm]

A bit detalized proposal and architecture: https://gist.github.com/resurtm/5723626

Queue FSM states and transitions/actions:

[yupe]

I've tried to implement it here https://github.com/yupe/yupe/tree/master/protected/modules/queue/components
Maybe you'd have found it useful

[nkostadinov]

That will be really great. Take a look at https://developers.google.com/appengine/docs/php/taskqueue/.

[petrabarus]

Here's for AWS SQS

http://docs.aws.amazon.com/aws-sdk-php-2/latest/class-Aws.Sqs.SqsClient.html
https://github.com/aws/aws-sdk-php/blob/master/src/Aws/Sqs/SqsClient.php

[nineinchnick]

I almost finished a queue interface with three implementations that I'd like to propose as a base for work on queue components for Yii2.

I got:

• a basic message queue,
• pubsub,
• message locking/reservation with timeout
• a SQL, Redis and SystemV implementations

Should I make a WIP labeled PR or just make a branch and reference it here?

The current code is in my Yii 1 nfy module repo. I am going to port the whole module to Yii 2. There still are some issues to discuss about the current implementation.

[samdark]

PR is better.

[danschmidt5189]

@samdark what is the queue component actually doing? (Besides adding/removing jobs from the queue.) Is it responsible for polling the queue and invoking callbacks as well?

Maybe I'm way off on the intended use-case, which to me is more like a glorified cache or db store:

public function uploadImage($uploadedImage)
{
    // Perform necessary stuff synchronously
    if (false === ($filename = $uploadedImage->save())) {
        return false;
    }

    // Perform time-consuming stuff asynchronously
    // Queue only stores the message. Listeners are responsible for processing them.
    Yii::$app->queue->add(new CreateThumbnailsTask($filename));
    return true;
}

If we're just defining an interface, why stop with push/pull? There are a lot of methods we might want to implement.

[nineinchnick]

@danschmidt5189 I'm working on a general interface for a message queue that can be used to provide a job queue. I'm going to post a PR in very short time for review.

Such interface will come with few implementations to assist communication between concurrent requests. While working on it I considered creating an implementation that uses cache interface provided by the framework but found it very limiting. That validates the need for a message queue component in the framework or as a core extension.

[sergebezborodov]

http://kr.github.io/beanstalkd/ is good solution for small-medium size projects. Has full control on tubes and tasks, really fast.
Here is my old implementation https://github.com/sergebezborodov/beanstalk-yii2

[lubosdz]

+1 if this can support concurrent/parallel jobs

Just a hint on possible implementation details:

• Cooperative multitasking via generators by Nikita Popov
• PHP multithreading via sockets

[samdark]

It's now managed in https://github.com/yiisoft/yii2-queue

[nineinchnick]

@samdark I guess I should start splitting and moving my #2378 PR there too

[samdark]

@nineinchnick check what's there first.