Removing completed jobs from workers

[dbousamra]

I ran into issues when processing large numbers of jobs (1 million +) where if I didn't remove completed jobs, Redis' memory would balloon out.

So now in each worker, I have some code like so:

// Remove completed jobs
queue.on('job complete', function(id, result) {
  kue.Job.get(id, function(err, job){
    if (err) return;
    job.remove(function(err){
      if (err) throw err;
      console.log('Removed completed job #%d', job.id);
    });
  });
})

This runs on every worker, and they ALL try to remove the job. Ideally I'd like the worker who completed the job, to remove it from completed. Is this possible?

[behrad]

1. queue's job complete event is a globally published event which has nothing to do with workers.
2. This is not a good idea to bind job lifecycle management into distributed workers in this manner

So, you can write the same code in your master process, and remove all jobs on completion, or even you can bring up a simple maintenance node process to do so....

[slyder]

Hello, i have same issue with completed jobs.
Every day i have more than 5 million jobs in kue and i have to run additional process to remove completed jobs. Why not to add some kue option to disable creation of completed jobs?

[behrad]

Why not to add some kue option to disable creation of completed jobs?

@slyder Nothing is created. your job status is just changed to complete :)
This is the client application duty to maintain their created jobs, you can simply remove them on completion or within housekeeping intervals.
What Kue can add, can be very simple implemented conventions - functions to be called by you instead of boilerplate above codes.

[behrad]

@slyder if you want to distribute you job removal load also you can bind to job level complete event in your producers code, and remove them there.

[slyder]

@behrad thanks! yes, i saw such code in example.
For now i have function on setInterval which selects all completed jobs and remove them. And all works good here.
But i have idea (for performance optimization) make kue fork, so after job completed - remove it automaticaly and do not mark it as completed.
Because then you change job status you should change relative arrays: q:jobs:active, q:jobs:completed, plus you should have timeout to make cleanup, or queue.on('job complete') handler.
I think it is not optimal solution. But probably it does not take a much of processor time and could be ignored =)

[behrad]

We may be able to add an autoRemove flag on job creation so that Kue removes it on completion.

[slyder]

Yeah, It is what i suggesting! I think it will be usefull feature, because there are a lot of cases which does not require moving jobs to completed array.

[behrad]

Would you please test this in real with the latest master branch and lemme know the results:

jobs.create( ... ).attempts().removeOnComplete(true).save();

[slyder]

@behrad thanks for quick fix and it works now
but you have bug in your code
here should be
job._removeOnComplete = hash.removeOnComplete;
instead of
job._removeOnComplete = hash._removeOnComplete;
please change it in repository

by the way i see a lot of places with 0 == arguments.length, why not to change it to
0 === arguments.length ?
not big but optimization too =)))

[behrad]

Oh, my bad! a typo when merging branches....
will fix it, thank you.