[Bug]: Windows sandboxed processors broke in 4.15.2 and onward

[clewisln]

Version

v4.15.2

Platform

NodeJS - v18.18.0

What happened?

When creating a sandboxed processor pool in Windows, using an absolute path, an error will occur when the child is spawned. This will cause the job to fail immediately, and then the Child is put into a non-working state.

This does not seem to occur in a Linux environment.

I have tracked the regression to this commit:
d97a5e0

If this is no longer the correct way to specify the worker script, please suggest the correct approach and update the docs.

How to reproduce.

1. Install bullmq 4.15.2 or higher
2. Create a queue and sandboxed worker, pointing to a local file in the dist using an absolute path.
   Example:

const WORKER_SCRIPT = path.join(__dirname, 'worker.js');
const pool = new Worker(name, WORKER_SCRIPT, {...})

3. Add a new job to queue

Relevant log output

Error [ERR_UNSUPPORTED_ESM_URL_SCHEME]: Only URLs with a scheme in: file, data, and node are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at new NodeError (node:internal/errors:405:5)
    at throwIfUnsupportedURLScheme (node:internal/modules/esm/load:136:11)
    at defaultLoad (node:internal/modules/esm/load:87:3)
    at nextLoad (node:internal/modules/esm/loader:163:28)
    at ESMLoader.load (node:internal/modules/esm/loader:603:26)
    at ESMLoader.moduleProvider (node:internal/modules/esm/loader:457:22)
    at new ModuleJob (node:internal/modules/esm/module_job:64:26)
    at #createModuleJob (node:internal/modules/esm/loader:480:17)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:434:34)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Code of Conduct

• I agree to follow this project's Code of Conduct

[clewisln]

Additional Information:
nodejs/node#34765

When using node.js to resolve ESM imports, absolute paths are treated as a URL in all environments.

In order to fix without reverting the d97a5e0 commit, using a file:// url must be supported.
Example: new Worker(this.name, url.pathToFileURL(WORKER_SCRIPT).href, {...});

However, there is currently a file existence check in bullmq which blocks being able to do this:

if (!fs.existsSync(processorFile)) {
	throw new Error(`File ${processorFile} does not exist`);
}

[clewisln]

Note that my app is using CJS and not ESM.

For the CJS build, all of these issues go away if the processor uses require, instead of import

class ChildProcessor {
    async init(processorFile) {
        let processor;
        try {
            const { default: processorFn } = require(processorFile);
            processor = processorFn;
            if (processor.default) {
                // support es2015 module.
                processor = processor.default;
            }
            if (typeof processor !== 'function') {
                throw new Error('No function is exported in processor file');
            }
        }
		...
	}
}

[roggervalf]

hi @clewisln, you may want to try v4.17.0 that includes this change #2346

[clewisln]

@roggervalf, after further investigation, it appears newer versions of bull now supports passing a URL object as the processor.

When specifying the processor in this way, it works as expected on Windows. Specifying the file as a raw path string, per the docs, is no longer supported.

Now invalid: https://docs.bullmq.io/guide/workers/sandboxed-processors

Working example:

const workerURL = url.pathToFileURL(WORKER_SCRIPT);
this.pool = new Worker(this.name, workerURL, {
  connection: this.connection,
  concurrency: this.maxworkers
})

[clewisln]

With the documentation in PR #2373, this issue can be closed.