Support Web Workers

[sindresorhus]

What is the problem this feature will solve?

Creating cross-platform (Node.js + browsers) code has never been more important, but there are still some sharp edges. fetch support was recently added, but there's another important and popular API; Web Workers. Node.js does have worker_threads, but the API differs in many ways and it's really difficult to properly bridge them. There are attempts at bridging these APIs in user-land, but the most popular one is incomplete and not actively maintained.

What is the feature you are proposing to solve the problem?

I propose adding support for Web Workers in Node.js. The Web Workers API is essential to keep apps and servers responsive by moving CPU heavy work off the main thread. I strongly feel it should be part of Node.js.

What alternatives have you considered?

Continue using one of the available polyfills, but that means larger dependency trees, more bugs, and more workaround code.

[aduh95]

Is it something you'd be ready to implement this yourself?

/cc @nodejs/workers

[JounQin]

Similar to https://www.npmjs.com/package/web-worker?

[devsnek]

The linked module appears to mostly be adding things node workers already support (web events, data uris). Looking at MDN the only thing that really jumps out to me is type: classic/module and name.

[aduh95]

The linked module appears to mostly be adding things node workers already support (web events, data uris). Looking at MDN the only thing that really jumps out to me is type: classic/module and name.

Also the web version only supports URLs (strings) in the constructor, while the Node.js one supports paths and URL objects.

[Trott]

IIRC, when @addaleax implemented worker_threads, she modeled the API on Web Workers but it was not possible to support the entire API, so it diverged in places. I'm not sure if the specifics are documented anywhere or if anything has changed.

@sindresorhus Any chance you have specific API pain points you'd highlight? If we can't do everything but we can do some things, it would be good to know what is (at least in your view) the high-priority stuff.

[Trott]

@nodejs/workers

[addaleax]

but the most popular one is incomplete and not actively maintained.

That’s arguably a sign against inclusion in Node.js core, unless you have a reason to believe that development as part of Node.js would improve this situation (isn’t obvious to me why that would be).

she modeled the API on Web Workers but it was not possible to support the entire API, so it diverged in places

The goal was to implement an API that matches Node.js’s abilities and requirements. You can probably support all or almost all of the Web API if you try hard enough, and the Node.js API is certainly inspired by the Web API on the parent thread side. On the child thread side, you are just running code in a completely different environment to begin with.

[jasnell]

That’s arguably a sign against inclusion in Node.js core, unless you have a reason to believe that development as part of Node.js would improve this situation (isn’t obvious to me why that would be).

Definitely have to agree with this... It could also be a sign that worker_threads are "close enough" to discourage further effort there.

I'd certainly be open to PRs that move the current worker_threads implementation closer to the standard alignment. I don't think we really need to do a new implementation of anything, just incremental changes here and there.

tl;dr ... PRs welcome ;-)

[sindresorhus]

Is it something you'd be ready to implement this yourself?

No

[sindresorhus]

IIRC, when @addaleax implemented worker_threads, she modeled the API on Web Workers but it was not possible to support the entire API, so it diverged in places.

I'm curious why it was not possible to support the entire API? And what parts?

[sindresorhus]

That’s arguably a sign against inclusion in Node.js core, unless you have a reason to believe that development as part of Node.js would improve this situation (isn’t obvious to me why that would be).

The maintainer released something they needed and got busy. I don't think that's evidence for whether or not it would prosper as a part of Node.js. The repo does have a lot of pull requests, which suggests people would be willing to help improve it.

[sindresorhus]

It could also be a sign that worker_threads are "close enough" to discourage further effort there.

Why add fetch then? http.get is close enough. Why add Web Streams? Node streams are close enough. There are huge benefits for Node.js and browser sharing some APIs. Familiarity. Code sharing. Less dependencies.

Deno supports Web Workers.

I'd certainly be open to PRs that move the current worker_threads implementation closer to the standard alignment.

Moving worker_threads closer to the Web Workers API would be a welcome change, but Node.js should still have a spec compliant Web Worker API. worker_threads cannot ever be fully spec compliant.

[jasnell]

Why add fetch then? http.get is close enough. .

No, I meant that it could explain why the userland module wasn't advanced further, not offering any kind of reason why we wouldn't continue to make improvements in core. I'm all for that, just need someone to volunteer to do the work

[devsnek]

I don't think anyone is against improving this, we're more trying to figure out what specifically you are looking for in the worker api. Like what prompted you to open this issue? That information can help us triage and understand what scope of work is needed.

[fabiospampinato]

WebWorker is probably the single most important Web API that I miss not having in Node. The use case for me is just wanting to write code once that works everywhere, if possible.

worker_threads exists, and it works, but it just doesn't allow me to write code once that works both in the browser and in Node, not without more code to abstract the differences anyway.

Somewhat interestingly worker_threads is not usable in Electron's renderer thread with Node integration enabled, it's just disabled for whatever reason, so potentially one would have to support both WebWorker and worker_threads even when running code only in environments where Node is available, but I guess Electron's renderer threads with Node integration enabled are a bit niche.

The web-worker package is good for shimming out the differences between Node and the Browser but I think it doesn't really support bundling. I have written my own little shim that supports bundling (https://github.com/fabiospampinato/webworker-shim), but it only supports loading a base64-encoded module and really I only wanted to write my original module in an isomorphic way, in order to do that I had to write this thing and now I have to maintain it, ideally I should just delete this because Node would support WebWorker.

Whether WebWorker should be implemented or not seems like a no-brainer to me, whether this is the right time to implement it or whether there are more important problems to allocate resources to I don't know.

[sindresorhus]

My use-case: I try to make my packages work in both Node.js and browser whenever possible. Sometimes I would like to offload CPU heavy work to a worker (important to keep servers and apps responsive). However, because it's such a pain to deal with the differences of worker_threads and Web Worker APIs, I have generally just not done it, which is a shame. I did end up doing it in my latest package which prompted this issue.

I love Node.js, but the constant differences in essential APIs is a daily pain-point. Mostly Buffer, crypto (being fixed), and workers.

This is basically my Node.js wishlist and Node.js is getting closer every year: https://deno.land/manual/runtime/web_platform_apis#web-platform-apis

[sindresorhus]

I'm certain Node.js will have a spec compliant Web Worker API at some point (whether it's now or in 10 years). Same as I was certain Node.js would get fetch at some point. It just makes sense. My hope is for this issue to attract some interest from the core team and community so that it can happen sooner rather than later.

[fabiospampinato]

I love Node.js, but the constant differences in essential APIs is a daily pain-point. Mostly Buffer, crypto, and workers.

I think Node now has a WebCrypto implementation. Personally I just load those APIs via this trivial module and have code that works both in the browser and in Node trivially. I'd love to do the same, or better, for WebWorker and other Web APIs.

FWIW I think generally if an API is supported both by the browser and by Deno the question should be flipped: why shouldn't Node also implement it?

[addaleax]

Maybe start with a simple question: What does the global scope inside a Web Worker spawned by Node.js look like? Does it match WorkerGlobalScope? Does it match a regular Node.js global scope?

[antonmedv]

I used for node and browsers this package: https://www.npmjs.com/package/threads

Works pretty cool 👌🏻

[sindresorhus]

Maybe start with a simple question: What does the global scope inside a Web Worker spawned by Node.js look like? Does it match WorkerGlobalScope? Does it match a regular Node.js global scope?

I think it should match WorkerGlobalScope as close as possible.

If you need to access any Node.js-specific stuff, you can just use import.

[sindresorhus]

To make it easier to implement this, I would personally be fine with it only supporting module-type workers: new Worker('x.js', {type: 'module'});

[sindresorhus]

I used for node and browsers this package: https://www.npmjs.com/package/threads

This package would benefit from Node.js supporting Web Workers as it would not need separate code paths for Node.js and browsers.

[Jamesernator]

Definitely have to agree with this... It could also be a sign that worker_threads are "close enough" to discourage further effort there.

Speaking from my own experience, the fact that worker_threads is different enough from Worker often makes me decide just not to bother with threads at all in code (even though it could often benefit) that is polymorphic for Node/Browsers.

Although I will say, even if Node did provide a web-compatible Worker, the whole Worker API leaves a lot to be desired due to how much boilerplate the event/messagechannel API requires to do relatively simple things. It would make worker wrappers easier though if Worker was global and had a more compatible surface though (particularly a common way to actually listen to events).

[Trott]

@bnoordhuis @cjihrig As folks who have worked for Deno Land Inc. and therefore are presumably familiar with challenges and benefits of supporting Web Workers in a JavaScript runtime, do you have any recommendations, cautions, enthusiastic endorsements, etc.?

[cjihrig]

I wasn't around when Web Workers were implemented in Deno, nor have I maintained them at all, so I don't have anything to add in that regard.

[bnoordhuis]

• A plain Web Worker environment offers rather limited functionality. I think it's telling deno decided to expose the (obviously non-standard) Deno object to provide a better out-of-the-box experience.

• Deno goes through some unnatural contortions (by faking up a window.location) to make imports sorta-kinda work like in the browser. I don't know how node would tackle that.

• There's a long tail of lesser issues where node is subtly incompatible with the Web Worker standard. Stuff like timers, rejection handling, error reporting, etc. Nothing that can't be fixed but whether the effort and complexity pay off? 🤷

[voxpelli]

If the Web Worker specification is hard to adapt to non-browser JS, then maybe a simpler subset of the Web Worker API could be designed and proposed to browsers as a simpler building block? (Skipping the .location and similar)

There is also other work going on when it comes to workers, like @surma's JS Module Blocks proposal, which I raised this issue in to discuss ways it which it could maybe be usable in non-browser contexts: tc39/proposal-module-expressions#61

Having a common way of running such Module Blocks would of course be helpful then.

[sindresorhus]

A plain Web Worker environment offers rather limited functionality. I think it's telling deno decided to expose the (obviously non-standard) Deno object to provide a better out-of-the-box experience.

Which is why I suggested that Node.js core modules could be importable (but not globals).

Deno goes through some unnatural contortions (by faking up a window.location) to make imports sorta-kinda work like in the browser. I don't know how node would tackle that.

I don't see any reasons to support that. You can just do the following (as the Deno docs also point out):

new Worker(new URL("./worker.js", import.meta.url).href, { type: "module" });

There's a long tail of lesser issues where node is subtly incompatible with the Web Worker standard. Stuff like timers, rejection handling, error reporting, etc. Nothing that can't be fixed but whether the effort and complexity pay off?

Ideally, these things should be fixed regardless as it will benefit any kind of Node.js-browser shared code, not just Web Workers usage.

[bnoordhuis]

I don't see any reasons to support that.

We're talking about different things. My comment is on how to handle imports inside the script.

Think import "foo" - where is foo loaded from? Node and browsers (and deno) give different answers.

[sindresorhus]

Think import "foo" - where is foo loaded from? Node and browsers (and deno) give different answers.

Browsers do not support bare imports specifiers unless you use import maps. So the answer is the same as importing in a non-Web Worker context; Either bundling or Node.js supporting import maps.

[sindresorhus]

Bun also plans to support Web Workers: https://github.com/Jarred-Sumner/bun/issues/159

[lgarron]

I would be really glad to see web workers in node. I have spent literally hundreds of hours over the last decade trying to write portable web worker code, and my experience agrees with a lot of others in this thread:

• Web workers are the standard way to do non-blocking work in JS, which is critical for some applications and libraries.
• node is the main environment which uses a non-standard worker API, and it's significantly more work to support portable worker code if you want it to work in node. I'm often tempted to say "it's not worth it for this project". By contrast, other non-browser environments like Deno work out of the box.
  • Even libraries dedicated to web workers, like Comlink, have trouble working with node out of the box, and require careful setup. I had to fork and modify the code so that I could avoid .unref() issues in node.
  • I have yet to see reliable wrapper for worker_threads that implements an API sufficiently compatible with web workers. I'm glad to see people trying, but the resulting libraries are always missing compatibility details, or outright fail to work with some environments and/or bundlers. At this point, I think it's a fair assessment that worker_threads is not sufficiently compatible for the lay coder.

I think it would be sufficiently useful even if just module workers were supported, as ESM's import.meta URL is already needed to instantiate a web worker in the de facto portable relative syntax (once Firefox finishes implementing support):

new Worker(new URL("./worker.js", import.meta.url).href, { type: "module" });

(It may soon become feasible to use new Worker(import.meta.resolve("./worker.js"), { type: "module" }); instead, but the same point about ESM stands.)

For maximally portable code (i.e. a single implementation that works for CDN-hosted libraries, without environment sniffing), it would further be valuable if the node implementation was compatible with an object URL trampoline:

// The following four lines work in `node` these days!
const workerURL = new URL("./worker.js", import.meta.url);
const importSrc = `import ${JSON.stringify(workerURL)};`;
const blob = new Blob([importSrc], { type: "text/javascript" });
const objectURL = URL.createObjectURL(blob);
// This doesn't work yet:
new Worker(objectURL, { type: "module" });

But that's less critical, and blank workers or module blocks will hopefully make this workaround obsolete in the long term.

As a library author, I would find this especially valuable, because we have often have to use careful workarounds to prevent bundlers from making changes to our code that break different environments. If the same code path works in browsers and node, this concern would change from a fundamental risk into a (hopefully) rare edge case.

As an added benefit, it would be nice to avoid an import of worker_threads (or node:worker_threads). Many bundlers still detect such an import as a dependency and try to install, bundle, or error — even if it's a runtime dependency that is only imported when running in node. It's not a great experience if someone tries to use our library, only to find that that their bundler is suddenly complaining about this thing called worker_threads that they've never heard of — if it's not reliable and painless for someone to use a library that depends on workers, then the end user experience/performance can suffer. (For the time being, we try to obfuscate the import so that bundlers don't detect it as a static dependency, which in turn requires more workarounds to avoid issues with bundlers.)

Is it something you'd be ready to implement this yourself?

If it makes a difference, I'd be interested in pitching in, if it can be done without investing extensive time in learning the codebase.

[lgarron]

If it makes a difference, I'd be interested in pitching in, if it can be done without investing extensive time in learning the codebase.

Any interest from node maintainers?
The lack of web worker compatibility in node is a continual pain point for me.

[bnoordhuis]

It's like that Sinatra song: you can't have one without the other. Wanting to contribute a non-trivial feature without learning the code base isn't going to work.

[lgarron]

Wanting to contribute a non-trivial feature without learning the code base isn't going to work.

That's not what I said, though. I'm more than willing to spend a few dozen hours, I just can't spend a few hundred. But it wouldn't make sense to get started unless there is clear indication that this is a feature the core team would support/review/accept such contributions at this time.

[bnoordhuis]

Someone (multiple someones, really) will review it, you don't have to worry about that part. Whether it gets accepted is another matter, of course.

There are a bunch of unresolved questions, see discussion above. Those should have answers before you start on the code, otherwise you can expect a lot of back-and-forth on your pull request.

For the same reason, you'll want to post a plan of attack and have people review it before you start writing code.

[GeoffreyBooth]

Speaking only for myself, I think it would be very likely that support for web workers would be welcomed; in recent years we’ve been pushing for more interoperability with other runtimes, both browser and server-side, and this helps achieve that goal. (See also: fetch.) If you want a stamp of approval before beginning work, I can raise it with the TSC.

That said, I agree with @bnoordhuis, that you should try to resolve as many questions as possible before asking for that approval or before starting writing code. I suggest you start a new discussion thread with a proposal for how to implement adding web workers support, including outstanding questions that can be resolved in that thread; when that discussion runs its course and hopefully all issues are addressed, the TSC can give its blessing (if you want it) and then it’s just a matter of doing the work.

[jimmywarting]

I did now also just recently wrote a worker using worker_threads with MessageChannels. Had to look up a few things of how to use it properly.
One being { type: 'module' } which NodeJS lacks. the other was how to listen for messages and respond back.
it is based on EventEmitter instead of EventTarget, and the event do not have a event.data or event.ports so writing stuff cross env. friendly is a 👎

i too wish that it where more spec compatible and that only new sync stuff where only added onto worker threads.
for instances i did kind of wish that some part of node:fs sync specific things where only available on a worker... like sindresorhus said "its important to keep the main thread responsive"

So here is what i would like:

• A new globalThis.Worker that is more web spec'ed
• A new globalThis.MessageChannel that extends EventTarget instead of node's eventEmitter
• Listening / registering events on the worker should be based on EventTarget instead of node's eventEmitter
  • you should for instance be able to write
    globalThis.addEventListener('message', ({data, ports}) => {}), { signal }
• From now on, only new sync features should be only available inside of workers.

[aduh95]

• A new globalThis.MessageChannel that extends EventTarget instead of node's eventEmitter

new MessageChannel instanceof EventTarget returns false in all runtimes (I think), it would be weird for Node.js to make this change.

[jimmywarting]

The MessageChannel itself isn't a insteance EvenTarget, the ports are.

new MessageChannel().port1 instanceof EventTarget

just now realized that ☝️ is true even in NodeJS but it also have EventEmitter functionality

if we could get a new and spec'ed Worker, could we perhaps then also change what things are available on the globalThis?

• As per Locking down the "process" and "Buffer" globals modules#235 discussion and Restrict process and Buffer globals to CommonJS #26334 remove global Buffer and Process.
• Change globalThis to be an instance of EventTarget.

[tniessen]

Isn't it true that when you create a worker in NodeJS then you start a hole new NodeJS instance / Child process?

@jimmywarting No.

so I'm just wondering: why is worker built on spawning new processes

It's not.