Discussion about performance

[o-alexandre-felipe]

This is about documentation not a bug.

It is claimed that a bundle with binaries for multiple platforms is faster than an install script for a single platform.
This is reasonable for small modules, but not for large modules, do you have any benchmark data that you could share, it would be interesting to point this in the README for future readers. If not I would be happy to help in this evaluation.

Thank you

[mafintosh]

No benchmarks.

The main reason we bundle all the builds at once is to make sure it always works, including when install scripts are disabled.

If you wanna test it yourself, sodium-native is the beefiest one I build (3.5mb total binaries). I'm sure you can find some using multiple roundtrips that fetches on-demand.

[mafintosh]

As a non-scientific example here is some results from my computer on my network (Copenhagen, Denmark), YMMV

# time how long it takes to fetch prebuild, our sister library that fetches ondemand (and other stuff)
npm cache clean --force
rm -rf node_modules && npm install prebuild
...
added 237 packages from 162 contributors in 11.367s
...

# time how long it takes to fetch node-pre-gyp, another popular prebuild module that fetches ondemand
npm cache clean --force
rm -rf node_modules && npm install node-pre-gyp
...
added 67 packages from 25 contributors in 2.744s
...

# time how long it takes to fetch sodium-native, the "fattest" prebuildify'ed module I maintain (bundles 6 platforms)
npm cache clean --force
rm -rf node_modules && npm install sodium-native
...
added 2 packages from 4 contributors in 0.737s
...

Note that after installing the "on-demand" libraries they still have to do another roundtrip to fetch the actual builds,
but sodium is ready to go at this stage. Also note that gzip is quite good! Sodium is ~5mb unpacked but ~1.7mb packed.

[vweevers]

If you wanna test a fatter one, try https://www.npmjs.com/package/rocksdb (42 MB unpacked)

[o-alexandre-felipe]

Thank you for the feedback

I tested your fat package, in my machine the run time is about the same, for that 42MB unpacked and for a 12MB in two stages.
With a faster internet (say running in a datacenter) it will probably win.

Now I come with two other questions:

• Is it too difficult to get the binaries that are not for the correct platform deleted after download, or even better, only extracting the binary for the target platform?
• This could be solved by getting the selection of the package done by NPM on the server side based on request platform information that could be provided by request parameters/headers?

[vweevers]

Is it too difficult to get the binaries that are not for the correct platform deleted after download, or even better, only extracting the binary for the target platform?

That would (with current npm abilities) require a postinstall script. I wouldn't want to add that requirement to node-gyp-build. You could of course write your own tool; which would delete any prebuild/<unwanted-platform>/ directory (and perhaps more specific files within them).

This could be solved by getting the selection of the package done by NPM on the server side based on request platform information that could be provided by request parameters/headers?

In an ideal world, maybe! That's a big task though, because the npm client would have to collect various data points (os, libuv version, ARM version, libc flavor, etc), allow them to be overridden in various scenario's (notably when installing packages on a machine that won't eventually run the code), and make it easy (enough) to add new data points.

There are downsides to this approach too. For example, when switching between platforms (in Docker containers and what not) with a shared node_modules folder between them, I don't want to npm install every time. Other folks are packaging up Electron apps (with native addons) for multiple platforms, and need all prebuilds to be available at that time.