Missing lock for job

[ashkank83]

Description

Would you kindly let me know what can be causing the following error?

Error: Missing lock for job 01001520190312141017.CD finished\n    
at Object.finishedErrors 
(\\BullPoC\\Bull\\node_modules\\bull\\lib\\scripts.js:179:16)\n   
at job.queue.client.moveToFinished.then.result 
(\\BullPoC\\Bull\\node_modules\\bull\\lib\\scripts.js:166:23)\n    
at process._tickCallback 
(internal/process/next_tick.js:68:7)

I have around 1500 jobs registered in a queue and everytime I restart the server to retest my scenario, one or two jobs randomly get this error message. (FlushAll Redis before application restart so it's all cleared)

The job doesn't fail because of this error, however Bull tries to rerun the job and I get duplicate index issues which causes the job to fail.

Is it possible to guide me towards what can be causing this issue please? and also is there a way to avoid it/fix it?

I also have to add my processors are in a sandbox.

Bull version

3.7.0
Rdis: 4.0.2.3

Additional information

I searched through all the similar issues and read all of them, nobody seems to be having a stack trace like my issue.
Doing further investigation the error is coming from line 40 of moveToFinished-6 lua script file (Below), but how is it possible that the job has started and the lock is not available?

local rcall = redis.call

if rcall("EXISTS", KEYS[3]) == 1 then -- // Make sure job exists
  if ARGV[5] ~= "0" then
    local lockKey = KEYS[3] .. ':lock'
    if rcall("GET", lockKey) == ARGV[5] then
      rcall("DEL", lockKey)
    else
      return -2
    end
  end

[ashkank83]

I'm going to try and see if I can fix the issue with modifying

interface AdvancedSettings {
  lockDuration: number = 30000; // Key expiration time for job locks.
  stalledInterval: number = 30000; // How often check for stalled jobs (use 0 for never checking).
  maxStalledCount: number = 1; // Max amount of times a stalled job will be re-processed.
  guardInterval: number = 5000; // Poll interval for delayed jobs and added jobs.
  retryProcessDelay: number = 5000; // delay before processing next job in case of internal error.
  backoffStrategies: {}; // A set of custom backoff strategies keyed by name.
  drainDelay: number = 5; // A timeout for when the queue is in drained state (empty waiting for jobs).
}

Sorry wasn't aware of these options before creating the issue.

[Gappa88]

@ashkank83 did you solve this issue?
I have the same problem with the same stacktrace.

I also have put my processor in a child_process in order to avoid stalled jobs and missing locks but those errors continue to appear.
I noticed that missing lock occurs in less than 30s. I checked my timers and It occurs in few seconds.
I did not change the AdvancedSettings,

@manast do you have any suggestion for this issue?

[manast]

@Gappa88 if you are able to provide a reproducible test case we can take a look into it, otherwise it is pretty difficult for us to understand what is going on...

[PrestonR]

I've actually seen this consistently every 6 weeks in our application, and I've been unable to identify what's happening.

I'd like to spend some time and produce some code to reproduce. Haven't had a moment yet.

Using Bull: ~3.5.2 and Redis: 3.2.7

Might this be fixed in the latest version of Bull?