Instantiating a queue is blocking the node process. How to add a job to a queue from a web process?

[tolgap]

I am baffled by this, I must be missing something.

I have separate processes:

• A web process which a is a Koa application.
• A worker process which instantiates the BullMQ Workers and QueueSchedulers

In order to add a job to the queue from my web process, I need access to the Queue instance. But as soon as I instantiate a Queue, the process gets blocked as BullMQ creates a connection to the IORedis client. This causes my web process to never cleanly exit once the HTTP request is done.

The easiest way to explain this: simply instantiate a Queue:

// connection.ts
import { RedisOptions } from "ioredis";

const port = parseInt(process.env.REDIS_PORT, 10);
const host = process.env.REDIS_HOST || "127.0.0.1";
const db = parseInt(process.env.REDIS_DB, 10);
const username = process.env.REDIS_USERNAME;
const password = process.env.REDIS_PASSWORD;

export const connection: RedisOptions = {
  db,
  host,
  password,
  port,
  username,
};


// default.queue.ts
import { Queue } from "bullmq";
import { connection } from "../connection";

export const DEFAULT_QUEUE_NAME = "default_queue";
export const defaultQueue = new Queue(DEFAULT_QUEUE_NAME, {
  connection,
});

Simply executing the default.queue.ts file with npx ts-node default.queue.ts causes the process to never exit. So importing my queue in the handler of a web request will also cause it to never cleanly exit.

I never had any idea that this was the behavior of a Queue. I thought it was just a simple "build Job and insert into redis" kind of class.

How do I insert jobs into my queue, without blocking the node process/thread?

[manast]

In your example you are instantiating a queue but you are not closing it, you would get the same effect if you instantiated an ioredis instance (or any other database for that matter), not sure what is your expectation or what you want to achieve.

[tolgap]

@manast So every time I want to insert something into my Queue, I should instantiate a new Queue instance, and then close that instance? This part was unclear to me reading the docs, as I haven't been able to find an example explaining that you are supposed to call queue.close() in the docs.

I'll change my code to reflect this. Thanks.

[manast]

@manast So every time I want to insert something into my Queue, I should instantiate a new Queue instance, and then close that instance? This part was unclear to me reading the docs, as I haven't been able to find an example explaining that you are supposed to call queue.close() in the docs.

I'll change my code to reflect this. Thanks.

No, you should not do that, as you would not connect to a database every time you want to make a query to it, you just have one connection and just reuse that.

Maybe these tutorials can help you in understand how to build your application: https://blog.taskforce.sh/implementing-mail-microservice-with-bullmq/