Sandbox process stuck in active tab (it's not a duplicate question)

[matheuschimelli]

Description

I know that probably this is a old question, but I'm sure i checked every issue related about this kind of error, and i couldn't find any solution.

Packages, OS and versions:

• Windows 10 (also tested in Windows 10 WLS v2 w/Ubuntu 20.04.3 LTS)
• Node v16.8.0 (alto tested on Node v.15.4.)
• Bull.js v3.29.1 (also tested on 3.20.0)
• Typescript 4.3.5

The problem

When a job is added to queue to run on a separate process, it gets stuck in active tab and doesn't returns any error, and it doesn't return a stalled job error. The job just get stuck in queue.

Minimal, Working Test code to reproduce the issue.

(An easy to reproduce test case will dramatically decrease the resolution time.)

Command used to run with Typescript:

    "dev:queue": "ts-node-dev --respawn --ignore-watch node_modules --transpile-only ./src/queue.ts",

queue.ts

import Queue from './lib/Queue';

Queue.startProcess().catch((e) => { throw new Error(e) });

lib/Queue.ts

import Queue, { Job } from 'bull';
import signale from 'signale';
import redisConfig from '../config/redis';
import path from 'path'
const ext = path.extname(__filename)
const sandboxFile = path.resolve(__dirname, `sandbox${ext}`)


const workerServerQueue = new Queue("WorkerServer", redisConfig)
const jobProcessorQueue = new Queue("JobProcessor", redisConfig)
const AnotherNameQueue = new Queue("AnotherName", redisConfig)
const BackgroundJob = new Queue("BackgroundJob", redisConfig)
const mainServer = new Queue('MainServer')

async function handle(job: Job) {
    console.log("Handling job from Main Server")
    console.log('Received message', job.data.msg);
    await mainServer.add({ msg: 'Sending back to main server' })
    await BackgroundJob.add({ msg: 'job.data' })

    return Promise.resolve({ done: true });
}

async function startProcess() {
    signale.success('🐂 Bull running');
    console.log(sandboxFile)

    workerServerQueue.process(handle)
    jobProcessorQueue.process(handle)
    AnotherNameQueue.process(handle)
    BackgroundJob.process(sandboxFile)

}
export default {
    startProcess,
    queues: [workerServerQueue, jobProcessorQueue, AnotherNameQueue, BackgroundJob]
}

sandbox.ts

import { Job } from 'bull'

export default async function (job: Job) {
    console.log('Running on sandbox')
    console.log(job.id)
    try {
        console.log(job.data)
        console.log("It is working?")
        return Promise.resolve({ data: 'done' })
    } catch (error) {
        console.log(error)
        return Promise.reject(new Error(error))
    }
}

config/redis.ts

import { QueueOptions } from 'bull';

export default {
  host: process.env.REDIS_HOST!,
  port: process.env.REDIS_PORT!,
} as QueueOptions;

I'm exporting queues as an array because I'm also using Bull Dashboard.

Basically i can run any job, except jobs which are processed on a separate proccess. When i call await BackgroundJob.add({ msg: 'job.data' }) i get this on bull dashboard:

It looks like the job process start, but it never completes, and return basically nothing.

Bull version

• Bull.js v3.29.1 (also tested on 3.20.0)

Additional information

Thanks in advance for any help! (I gonna try to provide a codesandbox to help the reproduction)

[manast]

Change to this code instead:

export default async function (job: Job) {
    console.log('Running on sandbox')
    console.log(job.id)
    console.log(job.data)
    console.log("It is working?")
    return { data: 'done' };
}

[matheuschimelli]

export default async function (job: Job) {
    console.log('Running on sandbox')
    console.log(job.id)
    console.log(job.data)
    console.log("It is working?")
    return { data: 'done' };
}

Thanks for the answer, but unfortunately the job still get stuck in the active tab.

I waited about 30 minutes but nothing.

[manast]

Can you simplify your test case and just use 1 queue, test it and if it does not work please post the updated code and I will check it myself.

[matheuschimelli]

I'm made a huge quantity of tests, and actually i don't know if it's a bug or other thing, but this is what i found:

• Using Redis server v=5.0.7 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=636cde3b5c7a3923

High CPU usage with the code below:

• Queue.ts

import Queue, { Job } from 'bull';
import signale from 'signale';
import redisConfig from '../config/redis';
//import * as jobs from '../jobs';
import path from 'path'
const ext = path.extname(__filename)
const sandboxFile = path.resolve(__dirname, `sandbox${ext}`)


const sandboxQueue = new Queue("SandboxTest", redisConfig)

function startProcess() {
    signale.success('🐂 Bull running');
    console.log(sandboxFile)

    sandboxQueue.process(sandboxFile)
}
sandboxQueue.add({ testing: "testing data" })

sandboxQueue.on('failed', (job, err) => {
    console.log('Job failed', job.data)
    console.log(err)
})

sandboxQueue.on('completed', (job, result) => {
    console.log('Job Completed')
    console.log(result)
})

sandboxQueue.on('global:completed', function (job, result) {
    console.log(`Job ${job} completed! Result: ${result}`)
})
export default {
    startProcess,
    queues: [sandboxQueue]
}

• sadbox.ts

import { Job } from 'bull'

export default function (job: Job) {
    console.log('Running on sandbox')
    console.log(job.id)
    console.log(job.data)
    console.log("It is working?")
    return { data: 'done' };
}

The most strange part, is that I'm using ts-node-dev what means i have a autoreload when i save my code. I just saved, without any change a couple of times, and suddenly the queue was proccessed and it returned an error:

[INFO] 18:13:17 Restarting: C:\Users\chimelli\programing\projects\brasileis\worker\src\lib\Queue.ts has been modified
√  success   🐂 Bull running
C:\Users\chimelli\programing\projects\brasileis\worker\src\lib\sandbox.ts
Running on sandbox
3
{ testing: 'testing data' }
It is working?
Job Completed
{ data: 'done' }
Running on sandbox
8
Job 3 completed! Result: {"data":"done"}
{ testing: 'testing data' }
It is working?
Job Completed
{ data: 'done' }
Job 8 completed! Result: {"data":"done"}
Running on sandbox
9
{ testing: 'testing data' }
It is working?
Job 9 completed! Result: {"data":"done"}
Job Completed
{ data: 'done' }
Job failed { testing: 'testing data' }
Error: job stalled more than allowable limit
    at C:\Users\chimelli\programing\projects\brasileis\worker\node_modules\bull\lib\queue.js:961:13
    at processTicksAndRejections (node:internal/process/task_queues:93:5)
    at async Promise.all (index 0)

In Bull Dashboard, it showed me that the first job took 4 minutes to be processed:

After this, i cleaned redis using the flushall command and i ran again the queue using yarn dev:queue, and as i expected the job got stuck again in the queue:

And again i had a high cpu usage:

just to compare, without running the queue my CPU stays around this:

And yes, i know every time i was saving my code I was adding a job to the queue (due the autoreload) but only just this way i got some job processed.

Anyways, i don't know, but it could be a bug. I absolutely have no idea why it's happening. I tried several different codes and nothing.
I've been using bull about 1 year in production, but due the fact our software offers a kind of crawler scrapping service, i got the necessity to separate the jobs from the main application to a different one, and yesterday when i started the refactoring on a empty repository i started getting this issues.

I really appreciate any help and the time you are spending to help me with this.

[matheuschimelli]

I was testing again, and i realized that the sandbox works when i calling a javascript file. So i tested with this:

• sandbox.js

module.exports = async (job) => {
    console.log(job)
    job.log("Working from separated proccess")
    return Promise.resolve({ done: true })
};

and in Queue.ts i changed this line

const sandboxFile = path.resolve(__dirname, `sandbox${ext}`)

to:

const sandboxFile = path.resolve(__dirname, `sandbox.js`)

and then the job started processing.

So, probably this is related with typescript. So, there is any way to use Typescript on a sandbox process? Or basically the only way to achieve the job to be processed is using Javascript?

And, i know this is not related with this issue, but i also have tried using BullMQ, and it worked only with javascript as well. So my question is, is BullMQ suitable for production and there is any difference besides the full typescript rewrite between bull.js and BullMQ?

[matheuschimelli]

So, probably this is related with typescript. So, there is any way to use Typescript on a sandbox process? Or basically the only way to achieve the job to be processed is using Javascript?

I found a solution. Instead of using ts-node-dev i switched to nodemon using this nodemon.json:

{
    "watch": ["src"],
    "ext": "ts,json",
    "ignore": ["src/**/*.spec.ts","node_modules","dist"],
    "exec": "ts-node ./src/queue.ts"     
}

And now the queue are been executed even with Typescript. I would like to suggest to add a typescript section on README, because other people probably also had or are been having the same issue.

[msyahidin]

Hi,

I'm have similar problem with queue stuck in active.

It moved to complete when interval reach 100

This usually happen on first queue added, the second one smoothly completed.

I am using nestjs https://docs.nestjs.com/techniques/queues

Sometimes it goes to completed, sometimes to failed with "Random Error"

[manast]

@msyahidin please open a new issue with reproducible code.
I am closing this since the issue seems to be resolved by the author.