Adding a job with a previously used id will silently drop the job

[creativefctr]

Description

We use the database primary key as job_id, after a successful run of the queue, when we want to run it again for the next update, queue.add works but the jobs are not added back. It fails silently! Clearing redis with FLUSHALL will solve the problem.

Minimal, Working Test code to reproduce the issue.

1. create the queue:

const QUEUE = new Queue<Data>(
  NAME.ACCOUNT.METRICS,
  {
    redis: {
      host: process.env.REDIS_HOSTNAME,
    },
    defaultJobOptions: {
      attempts: 3,
      removeOnComplete: true,
      removeOnFail: false,
      timeout: 2 * 60 * 1000,
    },
  },
);

2. add jobs with a set id:

// id = some random number id from database
queue(NAME.ACCOUNT.METRICS)
                .add(
                    {
                        id,
                        username,
                        picture_id,
                    },
                    {
                        jobId: id,
                    },
                )
                .catch(error => Sentry.captureException(error));

3. Let it complete

4. Repeat stage 2. This time 98% of the jobs are not added in the waiting list.

5. Go to redis-cli and run FLUSHALL

6. It works

Bull version

3.18.0

Additional information

• Bull-board was used to observe the queue

[manast]

This is actually not a bug but works as design. This behaviour is necessary in order to implement idempotent jobs:
https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd

[creativefctr]

I get it but it means that all these jobs are stored in Redis? What if we add 500k per week? In a few months millions of jobs will end up in Redis. Is there an expiry setting for this?

[creativefctr]

also, is there any way to manually clear this without dropping the whole redis database?

[manast]

You normally configure the queue to just keep a safe number of completed or failed jobs:
https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd

[creativefctr]

Sorry, but I am still a bit confused, I have set removeOnComplete to true as you saw, but still the jobs were not fully removed and the job id was considered duplicate. Does this mean that in order to get them fully removed, I should set a number instead of true?i?

[manast]

with that setting they should be deleted when the job completes, can you provide a test example where this does not hold true?

[creativefctr]

That was the original issue. The test case is exactly the code that I have in the main issue message. As I said, they appear to be deleted (in bull-board I see zero jobs) but when I tried adding them back (more than 6 times), it did not work until I did FLUSHALL in Redis.
I should also note that I switched to many different stuff during the test, upgraded Bull from 3.12 to 18, added and removed rate limiter and set removeOnFail to false once... could it be due to this? I will test this one more time tomorrow to make myself 100% sure

[manast]

yes but we have code in the tests that demonstrates that the issue does not exist, so you need to provide me one test case that reproduces what you claim.

[manast]

to be clear, a piece of code that I can run directly.

[coder-pm]

We ran into the same issue, adding job with the same jobId is not starting the job. removeOnComplete is set to true. The job is not visible in the bull board but after getting the job from the queue (queue.getJob(jobId)) it's returning it in a stuck state. We got exactly the same configuration as in the initial description.

[manast]

@coder-pm
stuck state? I wonder what that is. In any case, if you can provide a code snippet that shows what you mean we will look into it.

[coder-pm]

@manast, sorry for the delay in responding. We have been quite busy lately. I can confirm that the issue I mentioned exists in @nestjs/bull. However, we have since switched to @taskforcesh/nestjs-bullmq-pro and that issue no longer exists.

[vytautas-pranskunas-]

Is this issue resolved?

[eldimious]

Hello, i have faced some issue, as mentioned above. When try to create jobs with same jobId, in most cases are working fine, but sometimes i do see in redis a bull job eg with name bull:price:testid. This job is not removed from redis even if i have removeOnComplete + removeOnFail equal to true, and new jobs with bull:price:testid are not executed.
Not sure why it remains in redis this job. @manast any idea?