Skip to content
This repository

Duplicate Job Processing #29

Closed
aonic opened this Issue · 6 comments

2 participants

Raja Kapur TJ Holowaychuk
Raja Kapur
aonic commented

I have a simple job worker that takes a JSON object and uses it to talk to various APIs, but it seems that everytime the worker processes a job, the subsequent jobs get triggered twice for each incoming job.

Worker code:

jobs.process('api.search', processJob);

function processJob(job, done) {
    // job process code
}

Output:

First job

[-] processing: { q: 'fj cruiser',
  type: 'post',
  limit: 100,
  since: 1310014308 }
[-] finished processing 0 records for fj cruiser
[-]  (b) no more data for fj cruiser

Second job

[-] processing: { q: 'ford mustang',
  type: 'post',
  limit: 100,
  since: 1310014308 }
[-] processing: { q: 'ford mustang',
  type: 'post',
  limit: 100,
  since: 1310014308 }
[-] finished processing 2 records for ford mustang
[-]  (b) no more data for ford mustang
[-] finished processing 2 records for ford mustang
[-]  (b) no more data for ford mustang

As you can see, 'ford mustang' job triggered the job.process() twice for reasons unknown to me. The Worker code from above is only executed once

Raja Kapur
aonic commented

Hm, I wasn't able to duplicate the issue with a brand new job type/worker. Let me try to make a reproducible test

TJ Holowaychuk

interesting, let me know if you find anything, I'll do a few quick tests tomorrow morning to see if I can produce this as well

Raja Kapur
aonic commented

Will do. If you want I can email you the code of the job worker where it happens consistently as well. Thanks for all your help today!

TJ Holowaychuk

sure, some code would definitely help

Raja Kapur
aonic commented

The dangers of late night coding.. the condition around no more data for ... was calling done() -- even though I had already called done() after finished processing ...

Sorry for the false alarm :)

Am I correct in my usage where I only have to call job.process() once per node.js worker? I saw that there was a setTimeout in the library, but will it keep receiving more jobs as they get queued?

TJ Holowaychuk

yeah i had some timeouts just to fake creating more jobs and processing time etc. job.process() can be called a few times to run different types of jobs but yeah it's callback is then invoked per job

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.