How could i solve this? job stalled more than allowable limit

[ohseunghyeon]

I'm running some batch job at night and maybe a few of them takes a quite long time.

so.. I can see some jobs remained and the failedReason is job stalled more than allowable limit

I tried sandboxed processor but it actually didn't work.

I think job status changes to stalled when it takes much time right?

no?..

any idea please..

[manast]

You can configure how many times a job is allowed to stall using this: https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.queuescheduleroptions.md

When using sandboxed processors jobs should not stall unless the CPU that is running the queue is so stressed that it is not able to extend the lock of the job that is being processed.

[ohseunghyeon]

Thank you for the quick answer.
I will increase the maxStalledCount and see how it goes.

[Niek]

What's the best approach when using child_process.fork() in a job worker? It seems to stall a lot for long-running processes. Would starting the fork() inside a sandboxed process work?
I tried child_process.exec() which seemed to fix it, but I rely on fork() since I require IPC.

[manast]

@Niek not sure I understand your question. Sandboxed processors are already running i their own process separated from the main worker process.

[Niek]

@manast my use-case is a bit weird:

• worker picks up a job
• sets a bunch of env vars, prepares some external apps
• then it launches another node app through fork(), progress is reports back through IPC

I assume the fork() implementation differs from exec() in the sense that it's CPU-heavy, causing the stalls. Increasing maxStalledCount works, but it's not really the best solution I think.

[manast]

It should not matter, fork will create a new process. The question is if this new process is so CPU heavy that produces starvation on the other processes running in the same CPU.