Publish emsdk on npm + integrate with npm

[martijnthe]

Right now it's kind of kludgy to create npm packages that are built from C sources using Emscripten. Normally, when doing a local 'npm install', the devDependencies for the package are pulled from npm. But if Emscripten is a development dependency, people need to "manually" install emsdk, activate the right version, etc. It would be great if Emscripten could also be pulled from npm to 1) make this less kludgy 2) enable automatically activating the Emscripten/node/llvm versions required for the project in question, for example based on additional info in the package.json file.

[juj]

Do you know what other similar projects do that need e.g. LLVM to be bundled with them? Does npm architecture generally ship prebuilt binaries for each platform?

[martijnthe]

Answering in reverse order:

Does npm architecture generally ship prebuilt binaries for each platform?

Bundling precompiled binaries happens as well:
https://cylonjs.com/blog/2014/11/19/creating-multiplatform-precompiled-binaries-for-node-modules/
https://github.com/mapbox/node-pre-gyp/

In many projects native bindings C++/C code is built using node-gyp (often it's there's not a ton of code to compile so it's acceptable). But personally I would not want to build the pieces of Emscripten SDK from source because it takes a very long time... I'd much rather use pre-built binaries (for speed).

Another example that just came to mind that actually bundles entire Chromium+V8 binaries is this: https://www.npmjs.com/package/electron-packager / https://github.com/electron-userland/electron-download.

Do you know what other similar projects do that need e.g. LLVM to be bundled with them?

The binaries don't necessarily need to be bundled with them. I can imagine you could re-use most of the existing emsdk install script and run it as an "install" package script to install emsdk under node_modules/emscripten and activate the version that matches with the package.json: https://docs.npmjs.com/misc/scripts

Initially, I was thinking the install script could look at the package.json of the depender to find the desired combo of LLVM/node/..., but thinking it through a bit more, this is probably a bit tricky. AFAIK there's no good way to find the package.json of the depender. So, it'd probably be better to avoid this and just make the npm releases+versions follow the precompiled SDK versions that emsdk offers today (so doing npm install emscripten@1.37.22 would install sdk-1.37.22-64bit).

[lastmjs]

Even if emscripten were to be an npm module, without binaries the building takes forever, at least when I was trying to get everything set up to work with WebAssembly. What do people think about compiling Emscripten to WebAssembly? Once Node.js has full support for WebAssembly, we could use WASM as portable bytecode and just ship that, then there would be no building for each client, and Emscripten should work the same across all machines with Node.js WebAssembly support.

[juj]

At the moment Emscripten does run an infra to do precompile builds of each tag, so users don't have to build from source, unless they want the latest development branches. Probably emsdk install latest followed by emsdk activate --embedded latest might be the kind of activity to do for npm integration. If someone has a capability to start developing and maintaining npm packages of Emscripten SDK, that would be helpful to kick this further.

[FreezePhoenix]

Hey, wanting to use Emscripten in a package of my own - however it takes a long time to install (and it doesn't seem to work even when it finishes) I was hoping there was an NPM package that would do the install for me, but no such luck.

[FreezePhoenix]

I know it's a large task - but it does not seem unreasonable that emscripten itself could be compiled to WebAssembly, even if it's used in a node module.

[sbc100]

You'd have to start by compiler both python and llvm to emscripten since those are the core components of emscripten. Even with those tasks completed you'd need to figure out how to make sub-processes work. Emscripten forks many many subprocess. So yes, a mammoth task indeed.

[bvibber]

I suspect it's simpler to ship the existing emsdk in a way that's npm-friendly (installs into node_modules, downloads the necessary native binaries on-demand). Ideally it shouldn't have to modify a global ~/.emscripten file though; each instance must be able to have its own tool versions and config state.

[sbc100]

Yes, that is my understanding of what this issue is about, and seems orders of magnitude easier.

[bvibber]

nod quick overview of global bits:

• ~/.emscripten - points to all the various tools with absolute paths. No way to set it on CLI currently?
• ~/.emscripten_cache/ & ~/.emscripten_cache.lock - settable via --cache
• ~/.emscripten_ports - holds various libraries compiled on demand. No way to set on CLI?
• ~/.emscripten_sanity_wasm - ?? (seems to have the LLVM version of the upstream backend?)

If CLI settings or environment variables can be provided to override all of these, then wrappers for emcc, em++, emar etc can set them without people needing to fiddle with build scripts too bad, just setting a PATH or using relative paths.

It may also be possible to run the various tool wrappers via npx, eg npx emcc -o foo.js foo.c without requiring PATH fudging, which might work nice.

[bvibber]

Aha, there is an --em-config param that I didn't see before. Should help. :D

[bvibber]

I'll see if I can cobble together a proof of concept this week wrapping emsdk into an npm module along those lines, see if anything obvious still needs adjusting. Would make it easier to break up my own projects into smaller modules since I'd have a common dependency to work with. :D

[sbc100]

If you want to make config etc local the install take a look at emsdk activate --embedded. I think it does what you want.

[bvibber]

(emsdk's --embedded mode seems to set all the things I need. Coming along well!)

[bvibber]

Proof of concept that seems to work for me on Linux: https://github.com/brion/emsdk-npm ... went ahead and published it to npm for easy testing, I can iterate it from there or move/rename it to a final package name.

This makes commands runnable within the node project via npx: eg

# Helper to git-checkout emsdk into the node module dir
npx emsdk-checkout

# Helper to run emsdk to download/activate emscripten itself
# This uses --embedded behind the scenes.
npx emsdk install latest
npx emsdk activate latest

# Helper to run the commands within the emsdk_env.sh environment:
npx emsdk-run emcc test.c -o test.html

In theory you should be able to do stuff like CC="npx emsdk-run emcc" to pass into more complex build systems, but I've only tried on individual tests. It's also a bit verbose, as every invocation through emsdk-run runs emsdk_env.sh anew and it's pretty verbose.

[bvibber]

Oh -- this doesn't work on Windows yet, it may require a batch file to tie into emsdk_env.bat there.

[bvibber]

Now works on Windows as well as Linux/Mac. Haven't tested the intermediate batch file with ... "extreme CLI args" yet but it should work. ;)

[FreezePhoenix]

Ok, looks like things took a positive turn here. So what do I do to get precompiled binaries?

Or how do I compile it faster?

Or do I use this 'emsdk-npm' package?

[FreezePhoenix]

runner@repl.it:~$ npx emsdk-run emcc hello.cc -o hello.html
/home/runner/node_modules/emsdk-npm/bin/emsdk-run: 4: [: xemcc: unexpected operator
/home/runner/node_modules/emsdk-npm/bin/emsdk-run: 26: /home/runner/node_modules/emsdk-npm/emsdk/emsdk_env.sh: ./emsdk: not found
/home/runner/node_modules/emsdk-npm/bin/emsdk-run: 27: .: Can't open ./emsdk_set_env.sh
runner@repl.it:~$ cat node_modules/emsdk-npm/bin/emsdk-run
BINDIR=`dirname "$0"`
DIR=`dirname "$BINDIR"`

if [ "x$1" == "x" ]; then
    echo "emsdk-run"
    echo "Usage: emsdk-run <command> [<arg>...]"
    echo "Runs a given command within the context of the emsdk environment"
    echo "in the current node project."

cc @Brion

[bvibber]

Hmm... what Linux distribution are you running, and does it use a non-bash shell?

[bvibber]

@FreezePhoenix I've updated the package with a one-line fix (changed == to =) which should help with some non-bash shells. Let me know if that fixed it for you or if it's erroring out differently now. :D

[FreezePhoenix]

it's bash, pretty sure xd

[FreezePhoenix]

Now it just does nothing. Doesn't compile the files, doesn't do anything.

Wait, hold up. More errors.

runner@repl.it:~$ npx emsdk-run emcc hello.cc -o hello.html
/home/runner/node_modules/emsdk-npm/bin/emsdk-run: 26: /home/runner/node_modules/emsdk-npm/emsdk/emsdk_env.sh: ./emsdk: not found
/home/runner/node_modules/emsdk-npm/bin/emsdk-run: 27: .: Can't open ./emsdk_set_env.sh

Seems like it doesn't like relative paths.

[bvibber]

@FreezePhoenix hmm, the ./emsdk - not found is suspicious. I wonder if it's failing to run the emsdk script because python is missing? (Sometimes it's not there on fresh machines of various Linux flavors, or only python3 is present.) It uses a #!/usr/bin/env python shebang so needs "python" available in the system PATH.

If that's it, installing python 2 or python 3 (whichever gives you a python command on your distro) may fix it? I can probably detect this and handle it more gracefully. :D

[FreezePhoenix]

Ahhh python needs to be installed.
I read your message, was thinking since it was REPL.it that it wouldn't have python installed

Turns out it does .-.

"python" is installed, but I couldn't find it in /usr/bin/env, I instead found it in /usr/bin/

[bvibber]

@FreezePhoenix ok, so it sounds like something's gone awry but it should be able to find python, so I'm not quite sure what's going wrong. Can you open an issue over on my repo and we'll debug it further over there?

[mazmazz]

If anyone's interested, I've polished up @Brion's emsdk-npm and then I wrote a build library around it named emscripten-build to integrate with Node.js projects. Here are a few example use cases of this toolset.

I appreciate any feedback as I'd like to publish these on NPM in the future.

Resolved notes

(resolved by emscripten-build-npm#2)

I faced a couple concerns with how to handle EMSDK installing into node_modules, because:

1. Node dependencies are installed every time they're brought on by a dependent package, which leads to wasted space as EMSDK consumes 1 GB per install.

2. Windows installs can exceed MAX_PATH of 250 chars, which fails EMSDK install. This is very easy to hit if installing as a global package.

I resolved that the user should be allowed to specify their own EMSDK install path via a command line flag. emsdk-npm nags the user to do this on first run, then exits; then it proceeds to install EMSDK to the custom path (or node_modules if none) on second run.

If the end user specifies a custom install path, then all emsdk-npm installations will use it. The package author is able to specify an SDK version, which is activated on each build. I do want to allow custom install paths per-project soon.

I appreciate any feedback on how to handle this scenario better and any other UX/technical feedback. I'd like to work towards publication on NPM.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.