New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide queue component #492

Closed
samdark opened this Issue Jun 4, 2013 · 16 comments

Comments

Projects
None yet
@samdark
Member

samdark commented Jun 4, 2013

Useful for background jobs and all projects processing relatively large amounts of data.

Implementation can be designed the same way as cache i.e. base API + drivers such as:

Proposed base API:

abstract class Queue {
  // add job to the queue. $data will be passed as an argument to $callback and should be serializable
  public function push($callback, $data)
  {
  }

  // get a job and lock it queue for further processing
  public function get()
  {
  }
}
@bwoester

This comment has been minimized.

Show comment
Hide comment
@bwoester

bwoester Jun 4, 2013

Contributor

Another implementation might be based on System V message queues if PHP was compiled with --enable-sysvmsg

Contributor

bwoester commented Jun 4, 2013

Another implementation might be based on System V message queues if PHP was compiled with --enable-sysvmsg

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe
Member

cebe commented Jun 5, 2013

@petrabarus

This comment has been minimized.

Show comment
Hide comment
@petrabarus

petrabarus Jun 6, 2013

Contributor

I really agree with this one.
The implementation on my project now is directly using AWS SDK for PHP v2. But if there is another layer of abstraction provided by Yii and an implementation for local development, I think this is really good.

Contributor

petrabarus commented Jun 6, 2013

I really agree with this one.
The implementation on my project now is directly using AWS SDK for PHP v2. But if there is another layer of abstraction provided by Yii and an implementation for local development, I think this is really good.

@resurtm

This comment has been minimized.

Show comment
Hide comment
@resurtm

resurtm Jun 6, 2013

Contributor

A bit detalized proposal and architecture: https://gist.github.com/resurtm/5723626

Queue FSM states and transitions/actions:
queue-fsm

Contributor

resurtm commented Jun 6, 2013

A bit detalized proposal and architecture: https://gist.github.com/resurtm/5723626

Queue FSM states and transitions/actions:
queue-fsm

@yupe

This comment has been minimized.

Show comment
Hide comment
@yupe

yupe Jun 7, 2013

Contributor

I've tried to implement it here https://github.com/yupe/yupe/tree/master/protected/modules/queue/components
Maybe you'd have found it useful

Contributor

yupe commented Jun 7, 2013

I've tried to implement it here https://github.com/yupe/yupe/tree/master/protected/modules/queue/components
Maybe you'd have found it useful

@nkostadinov

This comment has been minimized.

Show comment
Hide comment
@nkostadinov

nkostadinov commented Jun 7, 2013

That will be really great. Take a look at https://developers.google.com/appengine/docs/php/taskqueue/.

@nineinchnick nineinchnick referenced this issue Dec 29, 2013

Closed

More implementations #6

2 of 2 tasks complete
@nineinchnick

This comment has been minimized.

Show comment
Hide comment
@nineinchnick

nineinchnick Jan 12, 2014

Contributor

I almost finished a queue interface with three implementations that I'd like to propose as a base for work on queue components for Yii2.

I got:

  • a basic message queue,
  • pubsub,
  • message locking/reservation with timeout
  • a SQL, Redis and SystemV implementations

Should I make a WIP labeled PR or just make a branch and reference it here?

The current code is in my Yii 1 nfy module repo. I am going to port the whole module to Yii 2. There still are some issues to discuss about the current implementation.

Contributor

nineinchnick commented Jan 12, 2014

I almost finished a queue interface with three implementations that I'd like to propose as a base for work on queue components for Yii2.

I got:

  • a basic message queue,
  • pubsub,
  • message locking/reservation with timeout
  • a SQL, Redis and SystemV implementations

Should I make a WIP labeled PR or just make a branch and reference it here?

The current code is in my Yii 1 nfy module repo. I am going to port the whole module to Yii 2. There still are some issues to discuss about the current implementation.

@samdark

This comment has been minimized.

Show comment
Hide comment
@samdark

samdark Jan 12, 2014

Member

PR is better.

Member

samdark commented Jan 12, 2014

PR is better.

@danschmidt5189

This comment has been minimized.

Show comment
Hide comment
@danschmidt5189

danschmidt5189 Jan 18, 2014

Contributor

@samdark what is the queue component actually doing? (Besides adding/removing jobs from the queue.) Is it responsible for polling the queue and invoking callbacks as well?

Maybe I'm way off on the intended use-case, which to me is more like a glorified cache or db store:

public function uploadImage($uploadedImage)
{
    // Perform necessary stuff synchronously
    if (false === ($filename = $uploadedImage->save())) {
        return false;
    }

    // Perform time-consuming stuff asynchronously
    // Queue only stores the message. Listeners are responsible for processing them.
    Yii::$app->queue->add(new CreateThumbnailsTask($filename));
    return true;
}

If we're just defining an interface, why stop with push/pull? There are a lot of methods we might want to implement.

Contributor

danschmidt5189 commented Jan 18, 2014

@samdark what is the queue component actually doing? (Besides adding/removing jobs from the queue.) Is it responsible for polling the queue and invoking callbacks as well?

Maybe I'm way off on the intended use-case, which to me is more like a glorified cache or db store:

public function uploadImage($uploadedImage)
{
    // Perform necessary stuff synchronously
    if (false === ($filename = $uploadedImage->save())) {
        return false;
    }

    // Perform time-consuming stuff asynchronously
    // Queue only stores the message. Listeners are responsible for processing them.
    Yii::$app->queue->add(new CreateThumbnailsTask($filename));
    return true;
}

If we're just defining an interface, why stop with push/pull? There are a lot of methods we might want to implement.

@nineinchnick

This comment has been minimized.

Show comment
Hide comment
@nineinchnick

nineinchnick Jan 18, 2014

Contributor

@danschmidt5189 I'm working on a general interface for a message queue that can be used to provide a job queue. I'm going to post a PR in very short time for review.

Such interface will come with few implementations to assist communication between concurrent requests. While working on it I considered creating an implementation that uses cache interface provided by the framework but found it very limiting. That validates the need for a message queue component in the framework or as a core extension.

Contributor

nineinchnick commented Jan 18, 2014

@danschmidt5189 I'm working on a general interface for a message queue that can be used to provide a job queue. I'm going to post a PR in very short time for review.

Such interface will come with few implementations to assist communication between concurrent requests. While working on it I considered creating an implementation that uses cache interface provided by the framework but found it very limiting. That validates the need for a message queue component in the framework or as a core extension.

@sergebezborodov

This comment has been minimized.

Show comment
Hide comment
@sergebezborodov

sergebezborodov Jun 24, 2015

Contributor

http://kr.github.io/beanstalkd/ is good solution for small-medium size projects. Has full control on tubes and tasks, really fast.
Here is my old implementation https://github.com/sergebezborodov/beanstalk-yii2

Contributor

sergebezborodov commented Jun 24, 2015

http://kr.github.io/beanstalkd/ is good solution for small-medium size projects. Has full control on tubes and tasks, really fast.
Here is my old implementation https://github.com/sergebezborodov/beanstalk-yii2

@lubosdz

This comment has been minimized.

Show comment
Hide comment
@lubosdz

lubosdz Jun 24, 2015

Contributor

+1 if this can support concurrent/parallel jobs

Just a hint on possible implementation details:

Contributor

lubosdz commented Jun 24, 2015

+1 if this can support concurrent/parallel jobs

Just a hint on possible implementation details:

@samdark

This comment has been minimized.

Show comment
Hide comment
@samdark
Member

samdark commented Sep 28, 2015

@samdark samdark closed this Sep 28, 2015

@nineinchnick

This comment has been minimized.

Show comment
Hide comment
@nineinchnick

nineinchnick Sep 28, 2015

Contributor

@samdark I guess I should start splitting and moving my #2378 PR there too

Contributor

nineinchnick commented Sep 28, 2015

@samdark I guess I should start splitting and moving my #2378 PR there too

@samdark

This comment has been minimized.

Show comment
Hide comment
@samdark

samdark Sep 29, 2015

Member

@nineinchnick check what's there first.

Member

samdark commented Sep 29, 2015

@nineinchnick check what's there first.

@cebe cebe removed this from the 2.1.x milestone Sep 29, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment