Lost listener event?

[njavilas2015]

Good evening, we are testing a couple of microservices in a development environment simulating requests of 100 per minute and we noticed that it is possible that the listener misses the job finished event. Is there any way to solve it?

import { Queue, QueueEvents, Job, JobsOptions } from "bullmq";

async function main(params: {}, connection: redis) {

    var queue = new Queue('task', { connection });

    var queueEvents = new QueueEvents('task', { connection });

    const job = await queue.add('task1', params);

    await job.waitUntilFinished(queueEvents);

    const result = await Job.fromId(queue, job.id);

    return await result.returnvalue;

}

simulating requests of 150 per minute, it stops responding in the range of 1-50 and since it is inside an await, the thread remains waiting. thus blocking the whole process

Node: 18
bullmq: 1.80.3 and testing lastest version 1.85.3

solution temporal add delay await setTimeout(150); in worker

NOT WORK Freeze task forever :(
await myQueue.add('house', { color: 'white' }, { delay: 5000 });

[manast]

Can you provide a complete test that includes those 150 per minute requests?

[manast]

And also please explain which part of the code is not working, it is not clear by your example.

[njavilas2015]

Excuse me, of course, I will send you all the information to solve this problem. I have created a class to work with microservices in node. In the same way, I will leave you a link to the repo in question so that you can clone it.

It is written in typescript is the simplest for and with instructions to copy and run. I appreciate your time
is public repo

https://gitlab.com/sarys.inc/bullmq-microservice-base.git

Maybe I am overlooking some configuration of the library, in the same way I am still continuing with the debug, I would also love to know what to do in a case like this. Since we are testing to adopt this library

[manast]

What I am seeing is that you are creating a new Queue instance via the Api class for every new job added to the queue: https://gitlab.com/sarys.inc/bullmq-microservice-base/-/blob/main/api/src/index.ts#L23
Every such new instance requires Redis connections, etc. You must re-use the queue instance if you want to achieve high throughput without strange side effects.
Finally, the use of "waitUntilFinished" is discouraged in production unless you have a heavy reason to do so, most likely this is a design error that can be implemented in much more robust ways. I recommend you to study the tutorials here for some design ideas.

[njavilas2015]

I understand, thank you very much for your prompt response. I will try it!

[njavilas2015]

How would I replace "waitUntilFinished" to wait for the response of the task? is there another way?
The code that I try to do is to wait for the task to finish and deliver the data obtained

await job.waitUntilFinished(this.queueEvents);

[manast]

Because you normally do not wait for the tasks to complete. The task itself does whatever it needs to do when it completes, for instance, it could create new tasks, etc. If you think about your problem from a different angle, say you cannot wait for the task to complete, how would you design your system to work in that way?