Support for pre-compiled binary modules in npm.

[tracker1]

Given that the Windows OS is supported for NodeJS and npm, I feel that the biggest hindrance at this point is lack of clear direction on supporting pre-compiled binary modules in npm, or at least a clear direction on how they should be supported.

With windows, there are essentially two versions needed, a 32-bit one, and a 64-bit one. Which is much less diverse than other platforms. I feel strongly that a blessed method for including binaries for windows is very important. Given that in production windows servers, the likelihood of having Python 2.7 and a build environment for C/C++ is extremely unlikely, and that the "global" install path for NPM modules is per user, it stands to reason that having a standard practice for binary modules is very important.

[Mithgol]

The following receipt is not an official path of supporting pre-compiled binary modules in npm, but it should work nevertheless.

1. Require (and use) node-bindings. Make yourself familiar with node-binding's documentation.

2. Compile the binary part of your module for all of your supported platforms (win32/ia32, win32/x64, darwin/x64, linux/ia32, linux/x64... sunos/ia32?), put the resulting binaries inside your npm package and let node-bindings pick one on user's side.

3. Put binding.gyp in .npmignore, otherwise npm will still attempt to build from the source. (Do not forget to add some contents of .gitignore to .npmignore when you create the latter, but only some: npm should not ignore the binaries, for example.)

4. (optional) Put binaries in .gitignore to prevent them from being hosted on GitHub. (This step is necessary only if you hate mixing sources with binaries. This step may also be completed already if you build in build/ and have already put that in .gitignore.)

After that npm install yourmodule would involve only downloading and unpacking, and thus would just work.

Node-gyp and its requirements (Python, make and gcc or Visual Studio Express and sometimes Windows 7 64bit SDK) would no longer be required on user's side.

This is also less ugly on the surface (because the npm install yourmodule command for your pre-compiled module becomes as simple to run as if it were some 100% JS module), but is slightly uglier inside (because the npm-served package contains binaries for each of the five or six supported platforms).

[AlexeyKupershtokh]

@Mithgol any chance to support pre-compiled modules just on some systems but compile from scratch on all other?

[Mithgol]

That would likely require some support from @isaacs (in npm's code).

Currently (unless I misunderstand something) npm is designed to build the module on all systems if binding.gyp is encountered.

Alternatively you may try to write your binding.gyp file so that building of your module is prevented on Windows and the existing file from the npm package is used instead. The language of binding.gyp seems complex enough to allow that. However, it also seems complex enough for me not to know if that solution is actually possible at all (and if it is, then what the exact receipt is).

[piscisaureus]

I think it's not a bad idea to distribute binaries. However I don't think making the module author responsible for all this is a scenario that works.

[Mithgol]

What do you suggest then?

[isaacs]

Here's a path for this which was explored a little bit last year, and needs some work to make real some day:

1. In the registry doc for each package version, there's a dist object with tarball and shasum fields. This is where the source is located. In addition to dist.tarball and dist.shasum add something like dist.win32_x64, so that the overall dist object looks like:

   { "dist": {
     "tarball":"http://registry.npmjs.org/blah/-/blah-1.2.3.tgz",
     "shasum":"deadbeef0000decaf0000000bbbabacadaba",
     "win32_x64": {
       "tarball":"http://registry.npmjs.org/blah/-/blah-1.2.3-win32_x64.tgz",
       "shasum":"deadbeef0000decaf0000000bbbabacadaba",
       "signature": "SD8T5h4hxRxQIJ1HS9FFCramtp7CuCW8ouHWTeLlusX10HlF"
     }
   } }

   I'm not sure what the "signature" is exactly, but presumably it'd be some way for you to verify that the binary came from a particular source that you trust. It should be a signature of the shasum using the private key of the party doing the builds.

2. When installing, if there's a dist entry for your platform/arch, then use that. First, test that the signature of the shasum valid using the pubkey of the source. If it is, then download the tarball, make sure it matches the shasum (like we do now) and proceed as normal, but don't run install scripts.

3. Microsoft, Joyent, and perhaps a few other interested and trustworthy parties get together and set up some build farms to pull down the packages that need a build step, try to build them, and update the registry with this extra information.

The same infrastructure could be used to run tests, nag authors for bad github repos, or do other fun stuff with packages.

[glennblock]

Yes it would be awesome if we can move this forward. We would love to help. We are having build farm discussions / exploration on this right now.

[glennblock]

@isaacs, for the proposal above would it always download, or have smart detection of if I have the prereqs to build? What if I want the modules to actually build as I am developing a native module?

[Mithgol]

Separate tarballs for win32 and win64 should be possible (for smaller downloads).

[TooTallNate]

Another thing to consider is that we would need separate tarballs for each released version of node. I'd imagine the npm repo would look something like:

  ...
  "win32_x64_0.8.15": {
    "tarball":"http://registry.npmjs.org/blah/-/blah-1.2.3-win32_x64_0.8.15.tgz",
  ...

In the ideal world, we would only need versions for minor version changes of node (0.6, 0.8, 0.10, etc.) but in reality the ABI has changed from time to time in between stable patch releases, and the ABI changes often during unstable releases.

Obviously we can't support precompiled binaries for every version of node in existence, so that leaves the question of which node versions should be supported and for how long (perhaps the last 2 stable minor versions?).

[piscisaureus]

In the ideal world, we would only need versions for minor version changes of node (0.6, 0.8, 0.10, etc.) but in reality the ABI has changed from time to time in between stable patch releases,

I think we have been pretty decent, atleast since 0.8.

[piscisaureus]

Microsoft, Joyent, and perhaps a few other interested and trustworthy parties get together and set up some build farms to pull down the packages that need a build step, try to build them, and update the registry with this extra information.

I would say we just allow anyone to upload things. To keep it safe, just tell the user that he/she is about to install binaries provided by Microsoft/Joyent/... and whether the user trusts them.

[glennblock]

Yep, that makes sense.

…

On Wednesday, December 12, 2012, Bert Belder wrote: Microsoft, Joyent, and perhaps a few other interested and trustworthy parties get together and set up some build farms to pull down the packages that need a build step, try to build them, and update the registry with this extra information. I would say we just allow anyone to upload things. To keep it safe, just tell the user that he/she is about to install binaries provided by Microsoft/Joyent/... and whether the user trusts them. — Reply to this email directly or view it on GitHub<#4398 (comment)>.

[trevnorris]

@isaacs any status on this?

[rvagg]

probably doesn't belong here and discussion should be moved to a duplicate issue here: npm/npm#1891

[isaacs]

Yes, thanks, this is an npm issue, and not one that we'll get to very soon. Closing this.