Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Creating 'slave' jobs from within a master job - job level events not emitting? #169

Closed
ragulka opened this Issue · 3 comments

3 participants

@ragulka

I have a huge job that creates multiple smaller jobs. I want to complete the master only after all the slave jobs have either completed or failed.

My problem is that when I create a slave job and listen to it's events inside the master job, sometimes the callbacks are firing, but more often they are not.

jobs.process('masterJob', function(job, done) {
  documents.find(function(err, docs) {
    var slaves = [];
    docs.forEach( function(doc) {
      slaves.push( function(cb) {
        var slaveJob = jobs.create('slaveJob', { ... }).save();
        slaveJob.on('complete', function() {
          cb();
        }).on('failed', function() {
          cb();
        })
      });
    });
    async.parallel(slaves, function (err) {
      return done( err );
    });
  });
});

It's weird because I have also a queue-level logger that happily sees that the slave jobs are completed. However, my master job is stuck, it will never complete because the 'complete' neither 'failed' events are not fired, ever. It happens intermittently.

@ragulka

Hmm, it seems that when I change

var slaveJob = jobs.create('slaveJob', { ... }).save();
slaveJob.on('complete', function() {
  cb();
})

to

jobs.create('slaveJob', { ... }).on('complete', function() {
  cb();
}).save();

The events seem to be emitting okay - unless I am using cluster. With cluster enabled, I have all the same issues as above.

Any idea why this is happening? Perhaps I am missing something obvious?

@aventuralabs

Most likely the job is being processed before your listener has been added. As a result, all events have already "fired", but you weren't listening.

By saving after the establish the listener, you circumvent this issue.

@ragulka

@brandoncarl - that makes sense, but it still doesn't explain why it won't work with cluster enabled.

@behrad behrad closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.