Threaded sandboxed jobs, how does it work?

[fjeddy]

I'm having an issue where jobs running in a sandboxed child process, is all running at the same time, normally this wouldn't be a problem, but the jobs in question are using the same subset of files to perform a build, and they can therefor not run at the same time.

You can run blocking code without affecting the queue (jobs will not stall). What exactly does this mean? jobs will not stall. I interpret this as the concurrency limit is ignored, but how can that be? I've read in several issues that child processess will be spawned and kept for re-use, but not above the maximum concurrent limit.

So, when I have no concurrent limit (defaults to 1), how come all my child processess are running at the same time? Am I misunderstanding how something works?

[fjeddy]

In my code, I figured it out, it was an issue with blocking code and not handling failures correctly. So it works like it should :)

But I would very much like an explanation on what exactly is meant by "Jobs will not stall". If a job does not stall, then what exactly happens in a scenario where the job would / should have stalled?

[nsfilho]

Hi! You need use a child_process fork to create a "killable" process. Because of that, for use sandbox, you will need to use a queue.process("filename.js") or even a typescript file (.ts).

In the main page of Bull.js show a example of this.

[fjeddy]

Right, but how does that prevent the job from stalling? Will bull terminate the process if it's not responding after a certain amount of time?

[manast]

@fjeddy jobs stall if they are very CPU intensive and nodejs event loop is too busy to be able to renew the job lock. Since sandboxed processes run in a separate process they do not block nodejs master process event loop.

[fjeddy]

So what it does is prevent the master process from stalling (Which is quite obvious), it does not prevent the job itself from stalling, like the description claims.

[manast]

no, it prevents the job from stalling which in bull terms means that no other worker will pick up that same job at the same time.