Run as daemon with workers

[steverobbins]

This might be a bit unicorny (ctrl+f Unicorny) of a feature but I would like to discuss it.

Is it possible to run cron schedules through a daemon rather than a cron job? For instance, the daemon is started and runs with 10 or so workers. The first 10 cron jobs to be processed this minute are selected and ran as 10 separate threads. Once one (or more) completes, the daemon takes the next 10 - running schedules and adds them to the queue.

The advantage of this is that several jobs can run asynchronously instead one after the other. A long running job can hold back others in the queue. The is something currently handled by Aoe_Scheduler with cron groups, but it requires setup. A daemon would require less setup.

There is the risk of not knowing the daemon is running. For instance, after a server restart cron starts up again without action. It would be easy to forget to restart the daemon, though it could be configured to start on boot.

I once attempted to build such a solution, but soon realized it would be a tougher task than I originally thought. I wouldn't be able to use the default and always event dispatchers. Logic would be needed to select schedules for this minute (or older), locate their model/name::method(), and run them independently. This is a very different approach from the default logic.

It looks like the event dispatches are being used here too, but perhaps it's something that could be worked in? It would be similar the scheduleNowAction().

[fbrnc]

Ok, so this is very interesting idea. Here are my thoughts:

Basically we have 4 different options:

• run all cron task in the same process
• run every cron task in a separate thread
• run every cron task in a separate fork
• run every cron task in a separate process (that is not a fork)

We do have a nice PHP library that comes with a OOP interface and some nice feature on top of the pcntl_* function which unfortunately is called "Threadi" (but basically it does forking, not threading): https://github.com/AOEpeople/Threadi - Wrapped in a Magento module: https://github.com/AOEpeople/Aoe_Threadi

While this works and we've been using this for indexers and importer this is not an ideal/generic solution:

• it requires pcntl to be enabled
• there are problems with database connections and other resources after forking. Those can be resolved, but it's very unlikely that we can get every cron task ever to take this into account.

With threading we'd most likely run into similar issues (never tried threading in PHP).

Running all tasks in a single process will most likely run into memory issues very quickly.

That leaves us with running every job in a separate process, and since the overhead of starting a new process is acceptable I think this is the way to go.

So the first step could be to introduce an action in the Aoe_Scheduler script that will run an individual task by id. That shouldn't be a big deal and could be a handy action for manual debugging.

Looking at the daemon portion of this idea we'd face following challenges:

• the daemon process could die or not be restarted after a server process. There are ways to assure a daemon process is automatically restarted (and of course started on server reboot), but both wouldn't be a good fit for most setups (and especially not for managed hosting.
• When deploying a new build (into a separate release folder), you'll need to take care of stopping the existing daemon (maybe letting the current jobs drain) and then starting a new one one the new build is live. At the very least the daemon should monitor the maintenance flag.

Since the actual goal that we're trying to accomplish here is running tasks in parallel and reducing the wait time for overdue tasks (e.g. being used as queue - which is the case for the EE indexers) my suggestion is to simply add a "daemon" mode to the existing scheduler script which would result in the process not to stop, but to keep looking for new jobs instead in a loop (making sure to have a minimum time span between database polls).
Then in order to be able to run this pseudo-daemon process forever, no job should be processed in this process, but the daemon only takes care of exec'ing new processes and monitoring their status. I'm confident this is something we'd be able to do without any memory leak allowing this process to run forever.
Then the scheduler script could continue to run over cron. cron,sh and scheduler_cron.sh both will take care of not starting a new process if another one is already running with a basic locking mechanism. This would result in cron being a basic watchdog mechanism that would restart the daemon once it died and simply check and move on most of the time.
This would make this whole concept backwards compatible and the only requirement continues to be cron.

So I think this could be a solution (and some documentation to help people understand what's going on and set it up):

Running an individual (existing) schedule
cd shell && php scheduler.php --action runSchedule --id <scheduleId>

Run schedules in individual processes
./scheduler_cron.sh --individualProcess (we should come up with something better :))

Run in daemon mode (don't stop, but go back and poll schedules)
./scheduler_cron.sh --daemon --workerCount <10>

And since --daemon without --individualProcess wouldn't work well for a very long time we should probably not even allow that and implicitly run schedules in individual processes.

But this should still be a separate option since people might prefer not run it in daemon mode but still in separate processes.

Also this minimalistic approach would allow allow you to continue using the cron_group concept to have separate pools of workers for different cron tasks (you might want to configure your setup to run them on different server instances).

So the daemon mode basically would be an endless loop polling the database, launching new processes (php scheduler.php --action runSchedule --id <scheduleId>) monitoring their status and launching new ones if there are available workers and things to do left in the schedules table.

Using the word "workers" here might be confusing since it implies that it's the same worker process that will be assigned a new schedule. But really the worker will simply stop and the daemon process will start a new one.

Since starting a new working might be overkill for some of the tiny cron tasks you could configure this behavior only for some jobs (by whitelisting or blacklisting the codes) using the cron-group mechanism.

[fbrnc]

Btw, Laravel has a very similar concept when it comes to processing queues (which basically is what we're doing here as well):
http://laravel.com/docs/4.2/queues#daemon-queue-worker

[steverobbins]

One issue I see with php scheduler.php --action runSchedule --id <scheduleId> is it would be possible for two schedules of the same kind to run at the same time.

It might be better to php scheduler.php --action runSchedule --code <foo_bar> --id <scheduleId>. The code is there so that when looking at running processes, the daemon will know not to start a duplicate one. The id is still needed so that when it finishes that specific record can be updated in the DB. At that point it should also UPDATE all pending schedules of the same code that are older than NOW() as skipped/missed (depending on the missed configuration).

---

I also wanted to mention that the daemon can launch/replace the watch dog. I say replace because the daemon will always be "watching" after all.

On the note about a daemon not being a good fit for everyone, I agree. The current cron behavior should still work. One can choose to use the daemon option instead. Documentation should be added to make that clear.

[beejhuff]

Thanks for sharing that link, Fabrizo! Have you used the laravel system
yourself? I dig through the docs and it appears that for more advanced
usage (the kind Steve's describing, I think) it relies on iron.io and the
IronMQ message queuing system.

We run Magento on Zend Server and I've been playing around with an adapter
for AOE_Scheduler that would allow me to use the Zend Job Queue and all of
the available cluster nodes to partition out the jobs. Seems like that
might be one way at least to mitigate the memory issue mentioned earlier.

I was endearing if anyone has worked on adapting the Scheduler to a Zend
Queue before? Most of the options discussed seem relevant to that type of
implementation also...

• BJ Hoffpauir

On Thursday, July 9, 2015, Fabrizio Branca notifications@github.com wrote:

Btw, Laravel has a very similar concept when it comes to processing queues
(which basically is what we're doing here as well):
http://laravel.com/docs/4.2/queues#daemon-queue-worker

—
Reply to this email directly or view it on GitHub
#134 (comment)
.

Sent from Gmail Mobile