Missing lock for job error

[jstayton]

I've looked through the various other issues that specifically mention the "Missing lock for job" error, but all of them seem to use flows, which I'm not.

I've seen the error for single jobs:

Error: Missing lock for job 45689. failed
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/scripts.js", line 208, col 24, in Scripts.finishedErrors
    return new Error(`Missing lock for job ${jobId}. ${command}`);
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/job.js", line 378, col 32, in Job.moveToFailed
    throw this.scripts.finishedErrors(code, this.id, command, 'active');
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/scripts.js", in runMicrotasks
  File "node:internal/process/task_queues", line 96, col 5, in processTicksAndRejections
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/worker.js", line 322, col 21, in handleFailed
    await job.moveToFailed(err, token);
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/worker.js", line 458, col 24, in Worker.retryIfFailed
    return await fn();
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/worker.js", line 140, col 46, in Worker.run
    const completedIdx = await Promise.race(promises.map((p, idx) => p.then(() => idx)));

As well as repeat jobs:

Error: Missing lock for job repeat:6fef8602fee8e8a10cb79bcf89578c22:1666166400000. failed
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/scripts.js", line 208, col 24, in Scripts.finishedErrors
    return new Error(`Missing lock for job ${jobId}. ${command}`);
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/job.js", line 378, col 32, in Job.moveToFailed
    throw this.scripts.finishedErrors(code, this.id, command, 'active');
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/scripts.js", in runMicrotasks
  File "node:internal/process/task_queues", line 96, col 5, in processTicksAndRejections
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/worker.js", line 322, col 21, in handleFailed
    await job.moveToFailed(err, token);
  File "/srv/build/1665511762/node_modules/bullmq/dist/cjs/classes/worker.js", line 458, col 24, in Worker.retryIfFailed
    return await fn();

Any idea what's going on? This is using BullMQ v2.1.1 with ElastiCache Redis, if that makes a difference.

Thanks for your help!

[manast]

This usually happens if for example a job has been stalled (the worker did not manage to update the lock on the job in time, which is 30 seconds by default), so some other worker took the job. Eventually the worker that lost the lock try to complete the job and since it does not own the lock anymore it will give the above error.

[jstayton]

Thanks, that's helpful!

What is your recommended course of action? Ensure that the worker can update the lock in time? Increase the lock time (if that's possible?) to give jobs more time to complete?

[manast]

If this happens often then it could be that the job is very CPU intensive so the lock cannot be renewed in time. In that case, the best would be to either break the job into smaller steps or try using sandboxed processors on a multi-core CPU, so that the process that renews the lock is running in its own core and the job in the others.

[jstayton]

Thanks!