New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HOLD for 4.1] Queue class prototype #347

Open
wants to merge 6 commits into
base: develop
from

Conversation

Projects
None yet
3 participants
@noldor
Contributor

noldor commented Dec 30, 2016

#117
I try to make a queue prototype.
If this fits what you want, I will make the datails.

@noldor

This comment has been minimized.

Show comment
Hide comment
@noldor

noldor Dec 30, 2016

Contributor

usage:

<?php namespace App\Controllers;

class QueueTest extends \CodeIgniter\Controller
{
    public function enqueue()
    {
        $queue = service('queue');
        $queue->send('hoge');
    }

    public function fetch()
    {
        $queue = service('queue');
        $queue->fetch(
            function($data)
            {
                var_dump($data);
            }
        );
    }

    public function recieve()
    {
        $queue = service('queue');
        $queue->recieve(
            function($data)
            {
                var_dump($data);
            }
        );
    }
}
Contributor

noldor commented Dec 30, 2016

usage:

<?php namespace App\Controllers;

class QueueTest extends \CodeIgniter\Controller
{
    public function enqueue()
    {
        $queue = service('queue');
        $queue->send('hoge');
    }

    public function fetch()
    {
        $queue = service('queue');
        $queue->fetch(
            function($data)
            {
                var_dump($data);
            }
        );
    }

    public function recieve()
    {
        $queue = service('queue');
        $queue->recieve(
            function($data)
            {
                var_dump($data);
            }
        );
    }
}
@lonnieezell

From what I can see so far, yes, I like what you're doing. And thanks for jumping in on this!

Before going much farther, though, could you sketch out some rough docs explaining it's use? It can be fleshed out more later, but I'm trying to get an idea of how you would recommend setting up listeners, etc.

Looking forward to this!

Show outdated Hide outdated system/Queue/Handlers/DatabaseHandler.php
status int(11) NOT NULL,
weight int(11) NOT NULL,
retry_count int(11) NOT NULL DEFAULT 0,
exec_after datetime NOT NULL,

This comment has been minimized.

@lonnieezell

lonnieezell Jan 2, 2017

Collaborator

I see you have a exec_after command. This means we'd need a way to set that, like a delay() method or similar?

@lonnieezell

lonnieezell Jan 2, 2017

Collaborator

I see you have a exec_after command. This means we'd need a way to set that, like a delay() method or similar?

Show outdated Hide outdated system/Queue/Handlers/DatabaseHandler.php
public function __construct($group_config, \Codeigniter\Config\Queue $config)
{
$this->group_config = $group_config;
$this->config = clone $config;

This comment has been minimized.

@lonnieezell

lonnieezell Jan 2, 2017

Collaborator

I'm probably missing something here, but why clone the config? Do they get modified somewhere?

@lonnieezell

lonnieezell Jan 2, 2017

Collaborator

I'm probably missing something here, but why clone the config? Do they get modified somewhere?

Show outdated Hide outdated system/Queue/Handlers/DatabaseHandler.php
* @param string $queueName
* @return boolean whether callback is done or not.
*/
public function recieve(callable $callback, string $queueName = '') : bool

This comment has been minimized.

@lonnieezell

lonnieezell Jan 2, 2017

Collaborator

recieve is spelled wrong. The i/e should be swapped: receive()

@lonnieezell

lonnieezell Jan 2, 2017

Collaborator

recieve is spelled wrong. The i/e should be swapped: receive()

This comment has been minimized.

@noldor

noldor Jan 6, 2017

Contributor

oops... thank you for pointing it out.

@noldor

noldor Jan 6, 2017

Contributor

oops... thank you for pointing it out.

@noldor

This comment has been minimized.

Show comment
Hide comment
@noldor

noldor Jan 2, 2017

Contributor

Thank you very much.
I will try write docs. But I'm not good at English (it's harder than coding for me), so please give me some time.

Contributor

noldor commented Jan 2, 2017

Thank you very much.
I will try write docs. But I'm not good at English (it's harder than coding for me), so please give me some time.

@lonnieezell

This comment has been minimized.

Show comment
Hide comment
@lonnieezell

lonnieezell Jan 2, 2017

Collaborator
Collaborator

lonnieezell commented Jan 2, 2017

@noldor

This comment has been minimized.

Show comment
Hide comment
@noldor

noldor Jan 6, 2017

Contributor

I wrote functions I think the queue interface should have. Could you check it?


Queue

overview

This queue interface will be aware of RabbitMQ. There are 4 components.

  • Provider
  • Exchanger
  • Queue
  • Consumer

There are multiple Consumers listening to same Queue, however, each message is processed by just one Consumer. It will not process twice.

If error raise, message will get back to same queue and will be retried. Retry count is limited. If retry count is over, the message will be deleted.

retry delay

Sometimes, Consumer job fails repeatedly (e.g. when the mail server is temporary unavailable, etc...).
So we should retry with delay.
Delay time can be modified in the config file.

retry count limit

When a job that always fails retry forever, the queue is full of such jobs.
So a job should be restricted the number of retrying.
Retry limit can be modified in the config file.

Retry count is passed to the callback function.

rejecting a job by Consumer

When a error arises, job will be retried automatically. Also we can retry a job manually.
If a specific exception is thrown, then this queue system will retry as failure.

endless loop on Consumer's job

When a job fall into a endless loop, it seems that the job is processing forever from the point of view of the Queue. The job will be neither success nor failure.
This queue system doesn't take care of it. A programmer should set max_execution_time or set_time_limit() properly.

On the Database Handler nobody can decide whether a job is processing or not, so the handler regards a job taking more than 300 sec as failure and retry it.

return without waiting after fetching a message

Normally, a request via a browser should response ASAP. So fetch() returns right away whether there are messages or not.
fetch() work with a single message.

waiting when there are no messages

If a job works as background job, the job should wait for a messages when there are no messages.
Because starting PHP process require CPU power.

receive() is the same as fetch(), except it will wait if there are no messages.
receive() work with a single messages and we should not call it repeatly with endless loop like while(1). PHP process will kill itself by max_execution_time. We should use supervisor, daemontools, systemd or the like. At worst, crond (on cheap hosting services).


Thank you for taking time.

Contributor

noldor commented Jan 6, 2017

I wrote functions I think the queue interface should have. Could you check it?


Queue

overview

This queue interface will be aware of RabbitMQ. There are 4 components.

  • Provider
  • Exchanger
  • Queue
  • Consumer

There are multiple Consumers listening to same Queue, however, each message is processed by just one Consumer. It will not process twice.

If error raise, message will get back to same queue and will be retried. Retry count is limited. If retry count is over, the message will be deleted.

retry delay

Sometimes, Consumer job fails repeatedly (e.g. when the mail server is temporary unavailable, etc...).
So we should retry with delay.
Delay time can be modified in the config file.

retry count limit

When a job that always fails retry forever, the queue is full of such jobs.
So a job should be restricted the number of retrying.
Retry limit can be modified in the config file.

Retry count is passed to the callback function.

rejecting a job by Consumer

When a error arises, job will be retried automatically. Also we can retry a job manually.
If a specific exception is thrown, then this queue system will retry as failure.

endless loop on Consumer's job

When a job fall into a endless loop, it seems that the job is processing forever from the point of view of the Queue. The job will be neither success nor failure.
This queue system doesn't take care of it. A programmer should set max_execution_time or set_time_limit() properly.

On the Database Handler nobody can decide whether a job is processing or not, so the handler regards a job taking more than 300 sec as failure and retry it.

return without waiting after fetching a message

Normally, a request via a browser should response ASAP. So fetch() returns right away whether there are messages or not.
fetch() work with a single message.

waiting when there are no messages

If a job works as background job, the job should wait for a messages when there are no messages.
Because starting PHP process require CPU power.

receive() is the same as fetch(), except it will wait if there are no messages.
receive() work with a single messages and we should not call it repeatly with endless loop like while(1). PHP process will kill itself by max_execution_time. We should use supervisor, daemontools, systemd or the like. At worst, crond (on cheap hosting services).


Thank you for taking time.

@noldor

This comment has been minimized.

Show comment
Hide comment
@noldor

noldor Jan 6, 2017

Contributor

Some functions are not implemented. I will make it from now on.

Contributor

noldor commented Jan 6, 2017

Some functions are not implemented. I will make it from now on.

@lonnieezell

This comment has been minimized.

Show comment
Hide comment
@lonnieezell

lonnieezell Jan 10, 2017

Collaborator

@noldor Those features look like great additions.

Collaborator

lonnieezell commented Jan 10, 2017

@noldor Those features look like great additions.

noldor added some commits Jan 28, 2017

@lonnieezell

This comment has been minimized.

Show comment
Hide comment
@lonnieezell

lonnieezell Mar 30, 2017

Collaborator

@noldor are you still working on this? Looks like there were a couple of tests on the PostgreSQL front that didn't pass the last round of changes.

Collaborator

lonnieezell commented Mar 30, 2017

@noldor are you still working on this? Looks like there were a couple of tests on the PostgreSQL front that didn't pass the last round of changes.

@noldor

This comment has been minimized.

Show comment
Hide comment
@noldor

noldor Apr 5, 2017

Contributor

@lonnieezell
I'm sorry for not contacting you for a long time.
I was busy in March, and bit busy in April too. I will just remove the error this weekend.
The first week of May is a large holiday in Japan, so I'd like to work with File Handler at that time. I'm sorry my work is late.

Contributor

noldor commented Apr 5, 2017

@lonnieezell
I'm sorry for not contacting you for a long time.
I was busy in March, and bit busy in April too. I will just remove the error this weekend.
The first week of May is a large holiday in Japan, so I'd like to work with File Handler at that time. I'm sorry my work is late.

@lonnieezell

This comment has been minimized.

Show comment
Hide comment
@lonnieezell

lonnieezell Apr 5, 2017

Collaborator

@noldor No worries. Just checking to see if you were still interested in working on this, and it sounds like you are. Good to hear! And I completely understand being busy. I've had a bit of that myself a month ago where I didn't get to touch it.

Thanks for responding, and looking forward to getting to play with this a bit !

Collaborator

lonnieezell commented Apr 5, 2017

@noldor No worries. Just checking to see if you were still interested in working on this, and it sounds like you are. Good to hear! And I completely understand being busy. I've had a bit of that myself a month ago where I didn't get to touch it.

Thanks for responding, and looking forward to getting to play with this a bit !

noldor added some commits May 6, 2017

@lonnieezell

This comment has been minimized.

Show comment
Hide comment
@lonnieezell

lonnieezell May 19, 2017

Collaborator

@noldor Any luck on this?

Collaborator

lonnieezell commented May 19, 2017

@noldor Any luck on this?

@jim-parry

This comment has been minimized.

Show comment
Hide comment
@jim-parry

jim-parry Jul 13, 2017

Collaborator

not to be too picky, but we expect commits to be GPG-signed :)

Collaborator

jim-parry commented Jul 13, 2017

not to be too picky, but we expect commits to be GPG-signed :)

@lonnieezell lonnieezell changed the title from Queue class prototype to [HOLD for 4.1] Queue class prototype Feb 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment