Add cron worker for dispatched jobs through the database

[vzool]

Hi,

This is a cron job worker using the database as a transfer layer between the dispatcher and the consumer which is a cron job in our situation. and it does resolve the feature request #560

• First: initialize the worker by:

> bin/cron_worker init # this will create a new table called `enqueue`

• Second: call worker directly and it will consume any dispatched jobs

> /home/app/directus/bin/cron_worker

Any Task should implements TaskInterface like this:

namespace Directus\Customs\Tasks;

use Interop\Queue\PsrMessage;
use Interop\Queue\PsrConsumer;
use Directus\Task\TaskInterface;

class HelloWorld implements TaskInterface{

 	public function handle(PsrMessage $message, PsrConsumer $consumer){

                // $message->getProperties() 
		// $message->getHeaders()
		
		// process a message
		
		$consumer->acknowledge($message);
		// $consumer->reject($message);
	}	
}

Testing Progress:

• Local: direct interaction test at vagrant box(Ubuntu 18.04.1 LTS). [Done]
• Local: cron test at vagrant box(Ubuntu 18.04.1 LTS). [Done]
• Production: direct interaction test at Debian 9. [Done]
• Production: cron test at Debian 9. [Done]

Notes:

• If there is a failed or bad job in the queue, it will stop that queue and the worker will ignore the queue until you resolve the issue.
• Worker support multiple queues, so if job failed in one that will not affect the other.
• If a job failed the worker will requeue it again, exits the current queue and move to other queue looking for jobs in it.
• The worker runs in Exit-On-First-Error mode by default which make sense for cron jobs environment if you called every minute, but you can change that behaviour by the following line:

> /home/app/directus/bin/cron_worker ignore-errors 

• If you enabled Ignore-Errors mode and there is an error the worker will hit that error every single second until you fix it. This is so annoying if you have an automatic bug reporter by email or SMS.

Any idea is welcome here, Thanks 😘

[benhaynes]

Thank you @vzool !!! I wish I knew enough to review this myself! ;)

@wellingguzman will check it out Monday and we'll hopefully get this in the next release! Thanks again, we really appreciate the help!

[wellingguzman]

Thanks @vzool for this contribution. We are reviewing this PR.

[vzool]

😊

…

On Mon, Nov 5, 2018 at 6:57 PM Welling Guzmán ***@***.***> wrote: ***@***.**** commented on this pull request. Thanks @vzool <https://github.com/vzool> for this contribution. We are reviewing this PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#567 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuSoD6yWp68W5RnIM0FcTZWk0zQxjyRks5usF_egaJpZM4YM4OZ> .

[vzool]

@wellingguzman Finally, I guess I did all the tests now 😆

[vzool]

@wellingguzman Still reviewing this PR 🙃

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[benhaynes]

@wellingguzman — can you post some thoughts on this here?

[philleepflorence]

Is it possible for this to be bundled as an extension, also can the installer specify the name of the collection that would hold the queue?

[benhaynes]

Hey @vzool — Welling left the organization, but we'll have @BJGajjar take a look.

@philleepflorence — I would say that this should be in in the Core (not an extension)... as long as we find a modular way to allow for different server schedulers (Apache, nginx, etc). I think we could have a single system collection for managing the job queue since that is not server dependent. Something like directus_tasks:

• id
• title
• frequency (either in raw minutes or with a cron-like syntax )
• script (maybe a relational link to a file? If Directus Files we'd need a way to make these private)

Then we just need a modular way to run the code every minute. My thinking is that we have a cron job (or whatever your server supports) that runs every minute, checks the directus_tasks for anything that needs to run, and runs the script file for whatever has a matching frequency.

I'm not sure how this PR works... but that's my (super basic) concept. :)

[philleepflorence]

@benhaynes - agreed, makes sense.
The script could be a custom or core endpoint.
My use case on my custom cronjob build is syncing data between social media and directus DB via an endpoint.

[benhaynes]

Interesting, I like the idea of allowing the task to hit an endpoint. My only concern is that there's no way to setup custom endpoints through the App, which means you need server access. The benefit of the "file" approach is that we could allow for uploading and setting up everything through the app. So perhaps we look into supporting both?

[benhaynes]

@rijkvanzanten and @BJGajjar — could you guys give me your thoughts on this PR? I like the idea of having a way to schedule tasks, but I don't know enough to audit this code/approach.

[rijkvanzanten]

I like the idea of being able to do stuff at a given time, but it's unfortunate that you have to go "around" PHP (eg do stuff on the CLI) to set it up. It makes sense, as PHP doesn't have a constant running process, but that does make it a bit hard to use.

[philleepflorence]

One way to do it is to have a master Cron Process per installation that runs every minute and then manage project tasks via directus, having PHP call the scripts or URLs of the individual tasks.

On cronjobs collection save in the DB, update a file in the filesystem that the cronjob looks at - minimizes DB overhead.

This is a set up currently in use on some Directus installations. So there is no need for system admins after launch.

[benhaynes]

I like the idea of having this be "decoupled from PHP" as much as possible. I'd love to store the tasks in the database so that if we move to node.js API in the future the tasks themselves don't need to change.

And having a CRON job that runs every minute (like @philleepflorence mentioned) and then check/trigger the tasks as needed sounds perfect. I like it!