As a node-binding consumer I can create distributable libraries that work on various platforms without forcing developers to compile the binding at install time.

[ernieturner]

Problem

When we compile the Neon node-binding source, the Node addon binary that it generates is not portable between architectures. This prevents us from being able to compile the source and publish it to NPM as we don't know the platform the user is installing from. Having users compile the addon at install time isn't very feasible since the chance that they have the various build dependencies (Rust, Neon, etc) is very minimal. Instead, we need to generate compiled binaries for various platforms and have those be retrieved at install time depending on the environment we're present in.

This is a known issue within Neon, but not something that appears to be actively developed or discussed even.

node-gyp

A rough parallel to what we're doing within Neon is node-gyp which is a tool for building Node addons from C++ code for Node. Where Neon helps us build Rust code into a native Node module, node-gyp does the same thing from C++ to native Node modules. As node-gyp is a more stable build tool, there's a larger ecosystem of tooling around dealing with this issue.

node-pre-gyp

What appears to be the most popular solution for handling the pre-compiled binary problem is the node-pre-gyp toolkit. This project aims to make it easy for distributing Node addons via pre-compiling binaries for various platforms. The quick and dirty workflow is

• Configure node-pre-gyp for the location of your binaries as well as where they are hosted
• Auto build and upload your binaries to an AWS bucket
• Add a postinstall script to your package.json so that when users run NPM install on it, once it's been installed it discovers the machines platform and requests the appropriate binary from the AWS bucket, extracts it and configures all paths.
• Optionally falls back to compiling the code via node-gyp if a binary cannot be found for the users machine.

This rough workflow is about the same as we'd ideally do with our node-binding. We likely won't have support for falling back to compiling locally given the lifting required there, but everything else should work roughly the same.

Resources

Other various resources/thoughts I've found so far after looking into this

• By default node-pre-gyp defaults to uploading/downloading to AWS buckets. There are tools such as node-pre-gyp-github that upload to GitHub releases. It seems like using GitHub releases for our binaries might be a good idea. Alternatively we would probably use a GCP bucket.

• Building the binaries for various platforms should be able to easily be done via Travis and there are a number of examples about how this is done for node-pre-gyp projects. Travis can build for both linux/osx as well as various NodeJS versions. There isn't Windows support or 32-bit support, so we'd have to decide how much we care about those platforms.

• This is a very detailed article which goes through the lengthy process that is involved to setup node-pre-gyp.

• I'm pretty skeptical that we'd be able to use node-pre-gyp with our stuff since we're in Rust. Maybe there's a way to use it for only the uploading/post-install downloading, but we might end up doing the same amount of work if we just wrote a tool ourselves. More investigation would be needed before we can decide on a route there.

• I've proven that the Neon module that is generated for these bindings works on both Node 8 and Node 10. However, there are no guarantees that future versions will work. So compiled versions would need to be build per Node version. This means that the cross-product of binaries would be [version of binding] x [version of node] x [platform (linux,osx,win)] x [supported architecture (x32/x64)]. Yeesh.

• A good project to look at for using node-pre-gyp is the Node Sqlite3 project.

Open Questions

• Even if we end up getting this all working above, there's still an issue that the whole process only works if consumers of the binding build their code on the same platform where it runs. If we have the above setup correctly but a developer is building on their local Mac before pushing the results to some location where it'll be run in Linux, the whole system breaks down. We might end up needing a way for developers to specify their platform via some environment variable so that I could build on one architecture but have the results be built for a different architecture.

• It should also be noted that Node 8/10 support WASM modules and I've proven that they work as expected. The WASM modules are also platform agnostic so they'd work to publish them to NPM. However the performance isn't great compared to these bindings so it depends on how much performance we need to wring out of this library.

[samsieber]

FWIW, with regards to travis, a lot of rust projects use a combination of travis and appveyor, using appveyor to produce the windows builds. There's actually a pretty good template for using travis + appveyor for rust packages: https://github.com/japaric/trust

[clintfred]

@samsieber Thanks. I did take a look at trust and we may well base our builds off that template in the future as we want to add more architectures. For now we are going to offer Linux and Mac builds for Node 8/10. I also noticed that Travis just started supporting Windows builds, so we may look at that if there's interest in Windows binaries.

https://blog.travis-ci.com/2018-10-11-windows-early-release