Feature Request: Remove completed/failed job after time interval

[n1ru4l]

Real World Use-Case

You use bulljs as a worker for sending emails.
You want to make sure a certain email is not sent a second time until a certain time period has elapsed.

The current state

It is possible to ensure a job is not enqueued twice by using the jobId option.

If you attempt to add a job with an id that already exists, it (the job) will not be added.(https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd)

It is possible to ensure that a job is deleted after it has been completed successfully with the removeOnComplete option

If true, removes the job when it successfully completes. A number specified the amount of jobs to keep. Default behavior is to keep the job in the completed set.
(https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd)

What is missing

There is currently no option for specifying that a job should be automatically deleted after it has completed successfully AND a certain time period has elapsed.

Workarounds

Create a separate Queue with "remove job operations"

Instead of having one queue you have two. One which does the actual job (e.g. sending emails) and one that includes the

/* worker.js */
const emailQueue = new Queue("email-queue");
const alreadySentEmails = new Queue("already-sent-email-queue");

emailQueue.process(async () => {
  // the email send logic
});

emailQueue.on("completed", job => {
  alreadySentEmails.add("the job", {
    jobId: job.id,
    // delete job after one week
    delay: 1000 * 60 * 60 * 24 * 7,
    removeOnComplete: true
  });
});

alreadySentEmails.process(async job => {
  const emailJob = await emailQueue.getJob(job.id);
  if (!emailJob) {
    return;
  }
  emailJob.remove();
});

/* somewhere-else.js */
const emailQueue = new Queue();
emailQueue.add(
  {
    // job data
  },
  { removeOnComplete: false /* defaultValue */ }
);

Possible Solution

Adding a new option that will make this possible.

removeOnCompleteAfter which default value is undefined. If removeOnCompleteAfter is set to a number it will override the behavior of removeOnComplete.

removeOnFailAfter which default value is undefined. If removeOnFailAfter is set to a number it will override the behavior of removeOnFail.

Change removeOnComplete behavior with backward compatibility in mind

The changed API could look similar to this:

interface RemoveConfigurationImmediate {
  mode: "immediate";
}

interface RemoveConfigurationKeepLatest {
  mode: "keepLatest";
  amount: number;
}

interface RemoveConfigurationAfterDelay {
  mode: "afterDelay";
  delay: number;
}

type RemoveConfiguration =
  | RemoveConfigurationImmediate
  | RemoveConfigurationKeepLatest
  | RemoveConfigurationAfterDelay;

type RemoveOnCompleteOption =
  | undefined
  | number
  | RemoveConfiguration;

type RemoveOnFailOption =
  | undefined
  | number
  | RemoveConfiguration

Similar Issues

I think that the workaround mentioned above ☝️might also be a valid workaround for issue #854 and #1387 (at least for cleaning successful jobs).

[stansv]

I wouldn't say the way you want to use Bull to achieve this is very straightforward. Why not just to take ioredis library which Bull also uses and put email-to-timestamp mappings into a hash set (where timestamp points to moment of time when you sent an email), then if now() - timestamp is less than one week you will know email should not be sent, otherwise update timestamp for your email and send it if CAS check succeeded.

(I don't think this feature is totally useless; and it's indeed very easy to implement manually using delayed jobs. You just don't need to depend on Bull to implement this feature)

[n1ru4l]

I am totally fine with implementing this in user land, I just wanted to make sure whether this would be something that might be contributed back to the core or not (since there are quite some similar issues around).

Thank you for your quick response!

[rohitkhatri]

Is this option already added?

or it should be done on the user side?

[shivajikewat]

once a job is getting completed in queue.process

later on I am trying to take a instance of queue by job id

now I am trying to move the job to failed (moveToFailed) but the same jobid is reflecting in both completed and failed (in arena)

can you suggest me what should i do so that jobid will removed from completed and will only reflect in failed?

[JasonMan34]

For those that got here from a google search, this is supported through removeOnComplete which you can use along with defaultJobOptions to apply for all messages in the queue (unless explicitly overwritten)

https://docs.bullmq.io/guide/queues/auto-removal-of-jobs#keep-jobs-based-on-their-age