Jobs as child processes?

[taijinlee]

It doesn't look like jobs are spun out as their own child processes. Was mainly wondering if this was a feature, or maybe I'm misreading the code somewhere. Thanks for the work!

[rschmukler]

Currently jobs are not spawned as child processes. Instead we use process.setImmediate to give up execution and schedule ourselves back into the node event loop. In this sense we avoid hogging the event loop and give up execution from time to time.

Hope this clarifies.

[taijinlee]

hrm, doesn't this mean that if there's lots of jobs that are very computationally expensive, it would process only single threaded? I think this makes sense as long as the jobs aren't computationally expensive?

[rschmukler]

Well, that would depend if the operations that are computationally intensive are good about using process.setImmedate / process.nextTick. If the underlying libs are good about that, then, it'd have less overhead than forking. If they aren't, then yes, it'd block.

I am open to adding an option on define, something like fork: true. By default though, I don't think it's worth the memory/overhead of forking.

[taijinlee]

I don't think I've personally seen much process.setImmediate / process.nextTick in most libraries. I think anything that's still computationally expensive would take a long time in a single thread.

I really like the idea passing in an option on define. I might take a stab at it when I get blocked on some other stuff

[rschmukler]

I am definitely open to a PR that'd implement this. Otherwise if I get a little time I might implement it myself.

[LiorCohen]

@rschmukler if you can tell me what such a push request would entail (config?), I might be willing to implement that change.

[bars3s]

I don't think we need to create a special setting for forking.
May be just add this code to doc?

var cluster = require('cluster'),
    cpuCount = require('os').cpus().length;

if (cluster.isMaster) {
    // Create a worker for each CPU
    for (var i = 0; i < cpuCount; i += 1) {
        cluster.fork();
    }

    cluster.on('exit', function (worker, code, signal) {
       console.log('job worker ' + worker.process.pid + ' died. Trying to respawn...');
       cluster.fork();
    });
} else {
    console.log('start job server: ' + cluster.worker.id);
    require('./app/agenda-worker');//initialize the agenda here
}

[rschmukler]

@taijinlee @LiorCohen I am using (in production, where I use agenda) code similar to what @bars3s posted. In addition to working across clustering, you can also have multiple servers pointing at the same job database and it will work.

I am wondering if @bars3s's proposal works for you for your use-case, or if you'd rather have the ability to specify per job. If so, we can think about what that should look like.

[LiorCohen]

@rschmukler, @bars3s: do you also have a server (socket, http, etc) running in the same cluster?

It does seem to me that there's no need to spawn child processes if you're already running inside a cluster, as each worker (server + agenda) would not block the entire cluster for long running jobs.

It should also be interesting to be able to manage how many worker processes will be allowed to handle jobs, but this seems to me like something outside of the scope of agenda.

Your thoughts, gentlemen?

[bars3s]

I agree that it's outside agenda's scope. It's scope of special library with load-balancing per process or per server and etc.

I usually use something like simple schema presented below

var cluster = require('cluster'),
    cpuCount = require('os').cpus().length,
    jobWorkers = [],
    webWorkers = [];

if (cluster.isMaster) {

    // Create a worker for each CPU
    for (var i = 0; i < cpuCount; i += 1) {
        addJobWorker();
        addWebWorker();
    }

    cluster.on('exit', function (worker, code, signal) {

        if (jobWorkers.indexOf(worker.id) != -1) {
            console.log('job worker ' + worker.process.pid + ' died. Trying to respawn...');
            removeJobWorker(worker.id);
            addJobWorker();
        }

        if (webWorkers.indexOf(worker.id) != -1) {
            console.log('http worker ' + worker.process.pid + ' died. Trying to respawn...');
            removeWebWorker(worker.id);
            addWebWorker();
        }
    });

} else {
    if (process.env.web) {
        console.log('start http server: ' + cluster.worker.id);
        require('./app/web-http');//initialize the http server here
    }

    if (process.env.job) {
        console.log('start job server: ' + cluster.worker.id);
        require('./app/job-worker');//initialize the agenda here 
    }
}

function addWebWorker() {
    webWorkers.push(cluster.fork({web: 1}).id);
}

function addJobWorker() {
    webWorkers.push(cluster.fork({job: 1}).id);
}

function removeWebWorker(id) {
    webWorkers.splice(webWorkers.indexOf(id), 1);
}

function removeJobWorker(id) {
    jobWorkers.splice(jobWorkers.indexOf(id), 1);
}

[rschmukler]

@bars3s fantastic example. We do something very similar in our environment.

I am also of the opinion that the management of the processes does fall out of agenda's scope.

If you guys are alright with it, I think we can close this issue? Otherwise I am more than happy to re-open if there is more discussion to be had.