Multi-server lock on jobs

[sansmischevia]

Having agenda be aware of other nodes in the system that connect to the jobs database will be helpful in the scenario you don't want a dedicated batch machine in your environment. This will prevent the same jobs being run twice in the same environment on different machines. (some sort of distributed lock mechanism with mongo?).

[rschmukler]

Currently, the way I am handling this is with cluster.isMaster();

basically:

if(cluster.isMaster()) {
   require('./lib/jobs');
}

I agree that ultimately we want to make sure that multiple nodes can process from the same database. This is a temporary workaround until we figure out something better.

Any ideas on how we should implement this @sansmischevia ?

[bars3s]

Hi!
I think this can be done through the lockedAt db field, findAndModify method and agent.define's lockLifetime argument or job.setLockLifetime method. What do you think about it?

[rschmukler]

I agree for the most part - specifically find and modify.

I don't know if we need lockedAt vs just a boolean locked. I suppose the advantage is if we crash we can have a timeout so we don't corrupt the database.

I think, then, that perhaps defining the lock timeout on the agenda instead of per job would be better. Otherwise we would have to query for a bunch of different locked times, which woulds result in n queries to the database, where n is the number of different lockLifetimes on the job. By doing it per agenda we can do it in 1 query.

[bars3s]

Ok. I will try to implement it via agenta.define('job name', {lockLifetime: 1000/*ms*/ }, cb), lockedAt db field and findAndModify method.

[rschmukler]

Implemented in 0.5.0

[sansmischevia]

Awesome, thanks @bars3s and @rschmukler! Now to re-write a ton of code with this new addition... agenda had a showstopper due to multi-server issues, but now it's a real lib that rivals quartznet.sourceforge.net