Duplicate Job Processing #29

Closed
aonic opened this Issue Jul 7, 2011 · 6 comments

Projects

None yet

2 participants

@aonic
aonic commented Jul 7, 2011

I have a simple job worker that takes a JSON object and uses it to talk to various APIs, but it seems that everytime the worker processes a job, the subsequent jobs get triggered twice for each incoming job.

Worker code:

jobs.process('api.search', processJob);

function processJob(job, done) {
    // job process code
}

Output:

First job

[-] processing: { q: 'fj cruiser',
  type: 'post',
  limit: 100,
  since: 1310014308 }
[-] finished processing 0 records for fj cruiser
[-]  (b) no more data for fj cruiser

Second job

[-] processing: { q: 'ford mustang',
  type: 'post',
  limit: 100,
  since: 1310014308 }
[-] processing: { q: 'ford mustang',
  type: 'post',
  limit: 100,
  since: 1310014308 }
[-] finished processing 2 records for ford mustang
[-]  (b) no more data for ford mustang
[-] finished processing 2 records for ford mustang
[-]  (b) no more data for ford mustang

As you can see, 'ford mustang' job triggered the job.process() twice for reasons unknown to me. The Worker code from above is only executed once

@aonic
aonic commented Jul 7, 2011

Hm, I wasn't able to duplicate the issue with a brand new job type/worker. Let me try to make a reproducible test

@tj tj closed this Jul 7, 2011
@tj
Contributor
tj commented Jul 7, 2011

interesting, let me know if you find anything, I'll do a few quick tests tomorrow morning to see if I can produce this as well

@aonic
aonic commented Jul 7, 2011

Will do. If you want I can email you the code of the job worker where it happens consistently as well. Thanks for all your help today!

@tj
Contributor
tj commented Jul 7, 2011

sure, some code would definitely help

@aonic
aonic commented Jul 7, 2011

The dangers of late night coding.. the condition around no more data for ... was calling done() -- even though I had already called done() after finished processing ...

Sorry for the false alarm :)

Am I correct in my usage where I only have to call job.process() once per node.js worker? I saw that there was a setTimeout in the library, but will it keep receiving more jobs as they get queued?

@tj
Contributor
tj commented Jul 7, 2011

yeah i had some timeouts just to fake creating more jobs and processing time etc. job.process() can be called a few times to run different types of jobs but yeah it's callback is then invoked per job

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment