Bull is not removing idle keys from Redis for completed/failed jobs

[jcalvento]

I read this issue #225 and added the options the remove jobs on complete and fail by default.
Ended up with something like this:

const defaultJobOptions = { removeOnComplete: true, removeOnFail: true };
const channelQueue = new Queue("Channel", {
  connection: redis,
  defaultJobOptions: defaultJobOptions,
});

Then every time I need to add a new job, I'm just using that queue with the job options if necessary, i.e.

await channelQueue.add("syncAll", {}, { repeat: { cron: "15 * * * *" } });

If I look at the keys stored in Redis, I can find old/idle keys for completed jobs that are not being deleted

{
    name: 'syncAll',
    data: '{}',
    opts: '{"attempts":0,"delay":3599963,"repeat":{"cron":"15 * * * *","count":2438},"jobId":"repeat:syncAll:cb9618d676fe6cb7fc182b28022e33bd:1619752500000","timestamp":1619748900037,"prevMillis":1619752500000}',
    timestamp: '1619748900037',
    delay: '0',
    priority: '0',
    processedOn: '1619752500200',
    returnvalue: 'null',
    finishedOn: '1619752500225'
  }

I have a QueueScheduler too but doesn't seem to have an option to add removeOn* behaviour

const channelScheduler = new QueueScheduler("Channel", { connection: redis });
await channelScheduler.waitUntilReady();

Is there anything missing on my config or it's not working as expected?

I'm using bullmq "1.14.0"

[MatayoshiMariano]

Having the same issue, any work around this?

[AdventureBeard]

I believe if you use delayed jobs with fixed IDs, this can have some nasty results; i.e. many silent failures.

[jcalvento]

@AdventureBeard I'm not using fixed IDs, what you see in Redis is managed by Bull internally when creating/updating jobs

[manast]

@jcalvento we have some unit tests that cover this scenario: https://github.com/taskforcesh/bullmq/blob/master/src/test/test_worker.ts#L68

Maybe you can find a way to write such a test that shows a bug? I can look into it ASAP if that is the case.

[rbalaine]

Hello,

We have the same issue (using bullmq 1.30.2). We set { removeOnComplete: true } in all our queues defaultJobOptions. It's working fine for all jobs except those with repeat option.
We define one QueueScheduler instance per queue with a dedicated Redis connection.

[roggervalf]

@jcalvento I tried to reproduce it using some tests without success :(, maybe you can change something in these tests to reproduce the issue:

it.only('should remove job after completed if removeOnComplete', async () => {
      const queueScheduler = new QueueScheduler(queueName);
      await queueScheduler.waitUntilReady();
  
      const worker = new Worker(queueName, async (job, token) => {
        expect(token).to.be.string;
        expect(job.data.foo).to.be.equal('bar');
      });
      await worker.waitUntilReady();

      const job = await queue.add(
        'test',
        { foo: 'bar' },
       { repeat: { cron: '*/2 * * * * *' },
        removeOnComplete: true },
      );
      expect(job.id).to.be.ok;
      expect(job.data.foo).to.be.eql('bar');

      return new Promise((resolve, reject) => {
        worker.on('completed', async (job: Job) => {
          try {
            const gotJob = await queue.getJob(job.id);
            expect(gotJob).to.be.equal(undefined);
            const counts = await queue.getJobCounts('completed');
            expect(counts.completed).to.be.equal(0);
            await worker.close();
            resolve();
          } catch (err) {
            reject(err);
          }
        });
      });
    });

    it.only('should remove a job after completed if the default job options specify removeOnComplete', async () => {
      const newQueue = new Queue(queueName+2, {
        defaultJobOptions: {
          removeOnComplete: true,
        },
      });
      const queueScheduler = new QueueScheduler(queueName+2);
      await queueScheduler.waitUntilReady();

      const worker = new Worker(queueName+2, async job => {
        expect(job.data.foo).to.be.equal('bar');
      });
      await worker.waitUntilReady();

      const job = await newQueue.add('test', { foo: 'bar' }, { repeat: { cron: '*/2 * * * * *' } });
      expect(job.id).to.be.ok;
      expect(job.data.foo).to.be.eql('bar');

      return new Promise((resolve, reject) => {
        worker.on('completed', async job => {
          try {
            const gotJob = await newQueue.getJob(job.id);
            expect(gotJob).to.be.equal(undefined);
            const counts = await newQueue.getJobCounts('completed');
            expect(counts.completed).to.be.equal(0);
            await worker.close();
            await newQueue.close();
            resolve();
          } catch (err) {
            reject(err);
          }
        });
      });
    });

[adamreisnz]

I believe if you use delayed jobs with fixed IDs, this can have some nasty results; i.e. many silent failures.

@AdventureBeard Can you elaborate on this some more please? We use repeating jobs with a unique ID to avoid duplication, and would not want to have those run into silent failures.