Host API compatibility (e.g. Node.js)

[ttraenkler]

@guest271314 posted on #389 and suggested to move this here:

I'm looking at this https://github.com/kilian-ai/linuxontab and thinking the kernel can be slimmed down to something like a mini ISO - with node already included. Voila, node in WASM. Something like this https://github.com/cemalgnlts/now.

My take on Node.js compatibility is like every other host API is that we should consider these imports that anyone can satisfy in the way they see fit:

It could be a real Node.js host just passing through its APIs or an adapter shim mapping to Deno APIs, using Wasmer's Edge.js WASI implementation - or what you mention above.

So in a way this could be considered orthogonal to core js2wasm and even be compatible with other projects similar to js2wasm (.js or .ts based variants).

I think this is where a .wit world could be the natural fit for a specification of the exact Node.js APIs that Edge.js might already satisfy, but haven't looked into it yet.

While my focus right now is getting js2wasm off the ground, just saying how this would click together later is exactly a good example of the modularity WebAssembly brings to the table where you can just swap out implementations as long as they are semantically consistent - at least that's the dream.

It will be interesting to see if this will be realized, and this is the right place to discuss bigger ideas to shape, expand and share a bigger vision.

[guest271314]

I looked in to Wasmer's edge.js. Couldn't get it to build for a while wasmerio/edgejs#70. I don't think I was able to create a Native Messaging host using edge.

Node.js "compatibility" I think was part of your goal, correct? To import packages from NPM?

I'd go with Deno and Bun API's. Deno for WHATWG Streams and Web API's, Bun for Bun Shell and build.

But really, ECMA-262 needs to write out I/O, so as to not add yet more implementations, or not.

See, WASI has a concept of cli, io, stin, stdout, fd_read, fd_write. ECMA-262 has no such interfaces.

So, when I ask the StarlingMonkey bytecodealliance/StarlingMonkey#314, ComponentizeJS bytecodealliance/ComponentizeJS#342, ByteCode Alliance folks - how exactly are you reading STDIN in the Component Model based on JavaScript input, I could get an answer pointing to a formal, canonical specification.

How is that WIT actually written in JavaScript? You wrote it one way, componentize-qjs does it a different way andreiltd/componentize-qjs#2 that I couldn't get to work as a Native Messaging host.

That's all good. I use AssemblyScript, Javy, js2wasm, and qjs-wasi.wasm. With qjs-wasi.wasm, we get QuickJS NG, the engine.

[guest271314]

I mean, there's doens of JavaScript engines, runtimes, interpreters - a lot outside of the browser. But there's no reason the browser can't have globalThis.stdin for each process.

So, Node.js "compatibility" is alright. But really, Node.js is just one tool in my JavaScript toolbox. tjs is around 8.6 MiB. Can;t get WAMR to do some things, but that's 132 MiB less than node v27.0.0-nightly202606114241d0d968. If we are talking about AOT JavaScript, and not using QuickJS engine compiled to WASM with the embedded JavaScript, we have to explain why Node.js get an exception - being over 100 MiB more than tjs. For a WASM target?

[ttraenkler]

Regarding compatibility I would first focus on NPM compatibility in the sense that an NPM package can be compiled to .wasm and used as a drop in replacement in the original host like in a browser or Node.js host as a baseline to start with.

This makes the path shorter to get something working and also allows for hands on experience with the original APIs before swapping them against API compatible alternative implementations like for WASI hosts.

I have not yet tried out Edge.js myself and indeed I am curious how a portable definition of a host API would look like and if it should be in WIT or typescript like I currently do for the Web APIs.

[guest271314]

The phrase "NPM compatibility" needs some serious clarity. There are a whole lot of packages on NPM. Some still using CommonJS - because Node.js still supports CommonJS. If you do something like fetch the Node.js nightly tarball every day or so, and run the npm shellscript, the npm shellscript itself says it's not compatible with the version of node running.

I would just focus on JavaScript - rather than the bottomless pit of trying to be "Node.js compatible". I don;t think it's possible. Deno tried. Bun tried. I think they each have tempered their campaigns in that regard and focused more on what their own gear is doing. That's rational, instead of chasing node.

Yeah, I use node. I also use deno, bun, qjs, tjs, shermes, workerd, llrt, qjs-wasi.wasm, V8's d8, SpiderMonkey's js, LadyBirds's js, JavaScriptCore's jsc, and other JavaScript engines, runtimes, and interpreters https://gist.github.com/guest271314/bd292fc33e1b30dede0643a283fadc6a. It ain't 2015...

Are you really gonna write the dozens and dozens of MiB in NPM libraries into WASM memory? I have to religiously clear the cache of Bun and Deno - and of the stuff in .npm when I use npm, which is rare.

I have not really been able to get the Wasmer stuff working. I tried when Edge.js was named Winter.js...

QuickJS NG for the win in JavaScript world, as engines go. You are gonna have your work cut out for you trying to keep up with implementing stuff like FinalizationRegistry, using, dynamic import(), resizable ArrayBuffer, and so forth. But, hey, I get it; that's what you're on. I got my popcorn, and will test your gear until it breaks - as long as you can handle my methodology.

[guest271314]

I guess the first question is where is are the NPM packages gonna reside?

[ttraenkler]

The phrase "NPM compatibility" needs some serious clarity. There are a whole lot of packages on NPM. Some still using CommonJS - because Node.js still supports CommonJS.

The goal would be that any NPM package can be compiled to .wasm, starting with simpler cases like acorn. Note CommonJS is already partially implemented and on the roadmap.

I would just focus on JavaScript - rather than the bottomless pit of trying to be "Node.js compatible". I don;t think it's possible. Deno tried. Bun tried. I think they each have tempered their campaigns in that regard and focused more on what their own gear is doing. That's rational, instead of chasing node.

Yeah, I use node. I also use deno, bun, qjs, tjs, shermes, workerd, llrt, qjs-wasi.wasm, V8's d8, SpiderMonkey's js, LadyBirds's js, JavaScriptCore's jsc, and other JavaScript engines, runtimes, and interpreters

Exactly - focus is on ECMAScript compatibility. Web and Node.js APIs are just an example how to deal with host APIs in general – not implementing each of them in this repo, but showcasing a generic solution that can work for all of the above hosts as well.

If an NPM package depends on a host API (say fs.readFile), the compiled .wasm module should expect it as an import (say fs_readfile), not caring how it is satisfied. In the trivial case such a .wasm module would run in its native host, e.g. if depending on Node APIs inside Node.js, just as .wasm. The types for those interfaces can be generated from the original Typescript type definitions @types/node - just like I do for Web APIs already.

Are you really gonna write the dozens and dozens of MiB in NPM libraries into WASM memory? I have to religiously clear the cache of Bun and Deno - and of the stuff in .npm when I use npm, which is rare.

You probably meant to say binary - not memory? I imagine each npm package could be compiled to a standalone .wasm file and optionally "bundled" (merged) into a single .wasm file (like for JS bundlers).

QuickJS NG for the win in JavaScript world, as engines go. You are gonna have your work cut out for you trying to keep up with implementing stuff like FinalizationRegistry, using, dynamic import(), resizable ArrayBuffer, and so forth. But, hey, I get it; that's what you're on. I got my popcorn, and will test your gear until it breaks - as long as you can handle my methodology.

Support for weak references or FinalizationRegistry will indeed be hard, the rest I see as relatively manageable. 🍿

I guess the first question is where is are the NPM packages gonna reside?

Do you mean where they reside remotely or locally? Remotely they could stay on the NPM registry at least until something better is found, but .wasm binaries could potentially also land on an OCI registry or CDN. Locally I assume having a deduplicated cache for the user would be most efficient where modules are just symlinked instead of copied into the projects to avoid filling up the disk on filesystems that dont deduplicate automatically, but that's not really in focus right now.

[guest271314]

I truly marvel at the focus on Node.js and NPM in repositories that include the term "JavaScript" in their descriptions. bytecodealliance/jco#1629.

Have you compared the implementation of WebTransport between Node.js and Deno?

Have you tried to fetch() file: protocol using node?

Have you compared the Node.js SEA implementation to deno compile?

Bonus to Node.js for node:wasi. However, we can implement WASI P1 in plain ole JavaScript.