Removing completed jobs based on age not working as expected

[benja8151]

According to documentation, removeOnComplete job option allows us to configure maximum age for a job to be kept after completion, in seconds. However, the check for expired completed jobs appears to only be done after another job is processed.

Expected Behavior

Setting age property on removeOnComplete should limit how many seconds a job is kept after being completed. Job should be automatically removed after specified time.

Current Behavior

Job is removed based on age only after another job in the same queue is processed, which could lead to cases where jobs could be kept for indefinite time, if no new jobs are added to the queue. This behaviour is different if you set removeOnComplete: true or removeOnComplete: {count: 0} - here, job is immediately and correctly deleted at completion.

Steps to Reproduce

1. Create a Queue and QueueScheduler with defaultJobOptions:

const queue = new Queue("queue", {
  defaultJobOptions: {
    removeOnComplete: {age: 5},
  }
);
const queueScheduler = new QueueScheduler("queue");

2. Create a Worker:

const worker = new Worker("queue",
  async (job) => {console.log(`Job ${job.name}#${job.id} is processed.`);}
);
worker.on("completed", (job) => console.log(`Completed job ${job.name}#${job.id} with data ${job.data}`));

3. Add job to queue:

queue.add("job", { random: "data" });

4. Checking situation in redis shows us that job is not automatically removed.

[manast]

The feature is working as designed.
This may not be the desirable behaviour in some situations, but for most cases, when you have a healthy queue processing jobs regularly it should not be a big issue.