How to prevent "Error: Missing lock for job ___", how to add an event listener (docs wrong)

[GeoffreyPlitt]

Node 14.21.3
BullMQ 3.10.2

Our company uses BullMQ and we noticed logs saying "Error: Missing lock for job ___". I understand in theory what this means and how to fix it, but I'm having trouble with the specifics.

The problem happens because the job takes longer than allowed (we aren't using updateProgress yet) so it gets retried, but then the original job finally succeeds and tries to update the status to successful, but can't because the system moved on and retries it elsewhere. I get that. Our job is idempotent so there isn't a big problem with the behavior, but I'd like to remove the noise in the logs and display this situation more effectively.

Question: Will subscribing to the "stalled" event prevent this error from printing? I'll assume yes for the rest of this post, but it isn't clear to me from the docs.

When I go to subscribe to the "stalled" event, I'm following the docs at https://docs.bullmq.io/guide/events; however I think the docs are out of date because typescript says the parameters are wrong when I try that:

Trying to understand how to fix this, I look at the source, which says on() takes a queue listener...

I can't find QueueListener in the source, except in a gitbook:
https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.queuelistener.md

Which does not explain the params for a successful event listener.

Are there examples of event listeners out there that work? Was I looking in the wrong places to find them?

[roggervalf]

hi @GeoffreyPlitt, from https://docs.bullmq.io/guide/events you can see some examples but it does mean that all events will handle the same parameters. I think that what you are looking for is WorkerListerner api reference https://api.docs.bullmq.io/interfaces/WorkerListener.html#stalled. What we can do is add Listeners references at the bottom in Read more section

[GeoffreyPlitt]

I'll take a look at WorkerListener, thanks.

But there's still an issue with the docs for Worker at that link.. the type of the event listener are not in agreement with the types in the source code. I tried adding an event listener directly to a worker with myWorker.on('completed', (job: Job) => { but that brings errors:

It looks like the handler is expected to take not a Job, but two strings (a jobId and a prev).

[GeoffreyPlitt]

Clarification: my screenshot doesn't show the construction of worker, so I'll include it here: const worker = new Worker(queueName, jobHandler, options)

[manast]

The problem happens because the job takes longer than allowed (we aren't using updateProgress yet) so it gets retried, but then the original job finally succeeds and tries to update the status to successful, but can't because the system moved on and retries it elsewhere.

I am confused by this explanation. The error you are getting is usually caused when a different worker is trying to complete (or fail) a job that has already been completed or failed OR a job that has stalled and moved back to wait. None of these situations are "normal", they can happen but if they happen often then something is not working correctly in your code. It could be that the jobs are locking the event loop, in that case, you should find strategies to avoid this locking or enable worker threads (with a concurrency lower than the number of CPUs).

https://docs.bullmq.io/guide/workers/sandboxed-processors#worker-threads

[manast]

Have you tried to listen to the error event on the worker? AFAIK the error you are seeing is emitted to that event but if you do not have any listeners it will just be printed to the console.

worker.on("error", (err) => { });

[manast]

@roggervalf do you have any further ideas of what can be going on here?

[GeoffreyPlitt]

@manast I'm doing exactly what you saying, did you see my screenshot? When I add the stalled event handler on the worker, there's a TypeScript type mismatch error. That's the whole reason for my post. There's a discrepancy between the docs, and your typescript types. That's what my post is about. The discrepancy between the docs and the actual types.

[manast]

@GeoffreyPlitt ok, sorry for the confusion, I was responding to another issue related with stalled jobs and I got things mixed up. However after re-reading this whole thread I can see that the answer @roggervalf gave you above was on point. There is no example in the documentation that you pointed out where it says that the stalled event returns a Job instance, indeed typescript is telling you exactly what it is expecting, a jobId and a prev (job status) string, exactly as it is documented here: https://api.docs.bullmq.io/interfaces/WorkerListener.html#stalled

[roggervalf]

hey @GeoffreyPlitt please review our comments above, also Read more section is providing Listener reference documentation https://docs.bullmq.io/guide/events#read-more where you can find the all our events. Hope this help on clarifying your case.