use process.release, make aware of io.js & node v4 differences #711

rvagg · 2015-09-04T13:56:47Z

Looking for patient reviewers, ASAP please. I'd like to be able to ship v4 with this version patched over the top of npm and upstream it to npm as soon as they are happy with it as well.

introduce a lib/process-release.js module that encapsulates all of the logic around downloads, names, versions and directories
introduce a test for process-release.js that hits it with various configurations and asserts that it comes up with the right answers
make node-gyp aware of process.release where it exists (>=3.0.0) and use it to determine download locations and names
make node-gyp aware of "iojs" where process.release doesn't exist, thereby making it work for all versions of Node and io.js from 0.8 to current so an npm install npm@latest -g even on one of the older versions of io.js will just work (magic).
make node-gyp able to compile against header-only tarballs, see the one-line change in addon.gypi and the search in two locations for common.gypi, for >=3.0.0 we are shipping header-only tarballs which are super quick to download, but it also needs to be able to compile against full source as well
keep the --target stuff intact (although I'd love to know if anybody actually uses that)
make node-gyp aware of the change in location for node.lib (& iojs.lib) for >=1.0.0. In <1.0.0 you find 32-bit node.lib in the root of the version's dist directory (e.g. http://nodejs.org/dist/v0.12.7/node.lib) while the 64-bit node.lib is in an x64 subdirectory (e.g. http://nodejs.org/dist/v0.12.7/x64/node.lib), for io.js we took the opportunity to clean this up and they are now in win-x86 and win-x64 subdirectories (e.g. https://iojs.org/dist/v3.3.0/win-x86/iojs.lib & https://iojs.org/dist/v3.3.0/win-x64/iojs.lib) (and the .msi's have been renamed and are in the parent directory, but that's not relevant here). Note that it still downloads both 32-bit and 64-bit .lib files, I haven't changed that logic.
introduced a test/docker.sh script that runs it forreal, but only on Linux obviously:
1. set up a base image with compile tooling
2. set up a clones image on top of the base image with git clones of some native addons to test
3. set up an image on top of the clones image for each of the versions of node and io.js to test plus the latest version of npm over the top
4. start a new container for each version of node & io.js, copy pwd into npm's node-gyp dependency folder so it uses the current code for testing when you use npm install etc., then do an npm install of the projects to see how it behaves.
5. you can see it downloading and using the source tarballs for all but io.js v3.3.0 for which it grabs the headers file because it's only in v3 that we started shipping that and introduced process.release

What does this buy us?

The ability to ship actually useful RC's, nightlies and master/unstable/canary builds which don't have to have proper release version numbers and sit in the release downloads directories, node-gyp will just work
The additional ability for sophisticated users to create custom / corporate / internal builds of Node that point to header/source tarballs that are at a different location e.g. inside a protected & HA network, thereby removing one more point of failure & build/bundle time (add in npme or similar and you can be fully HA). Note that this requires a custom build of Node, so re security it's the same discussion as the --link-module feature we introduced.
- The ability to support existing versions of io.js with new versions of npm (no more "you have to use the npm that ships with that version of io.js). We also get the ability to re-use the io.js brand if we revive it and start shipping again for some purpose, no more floating patches.

What's not here?

We need to finally make node-gyp aware of locally installed headers: this has been on multiple people's radars for a very long time but has never got done but it should get done. When you install via a complete tarball, or a make install from the source, you get the header files installed for that version. node-gyp should be able to use those to avoid any downloading. However this is not a panacea because:

This hasn't been enabled at all for Windows installers, although that's not an impossible task
Some old versions don't unpack headers and some just have broken headers—from memory there's some broken stuff across 0.12 and a couple of versions of io.js shipped without openssl headers
Some users don't have complete installs, we offer a plain node.exe for instance
node-gyp still has the ability to "cross compile" addons which won't work here (--target)
version managers will have to get creative with how they ship versions and communicate with node-gyp about the locations of header files
Doesn't solve the node.lib problem on Windows, although we may be able to ship that in the msi as well
... more mess ..

So this is a TODO, someone needs to take a bite at this soon, but it's not a top priority for shipping core.

Possible reviewers: @nodejs/build, @nodejs/addon-api, @TooTallNate, @Fishrock123, @othiym23, @bnoordhuis, @piscisaureus

rvagg · 2015-09-04T14:05:45Z

addon.gypi

@@ -5,6 +5,7 @@
    'product_prefix': '',

    'include_dirs': [
+      '<(node_root_dir)/include/node',


headers tarball has a different structure to the source tarball, no deps or src subdirectories, but this addition makes existing addons work with a headers tarball, a source tarball or a --nodedir

Fishrock123 · 2015-09-04T16:37:04Z

but it also needs to be able to compile against full source as well

@rvagg only for older versions, right?

Fishrock123 · 2015-09-04T16:41:42Z

Also, Fixes: #701

rvagg · 2015-09-04T16:42:00Z

but it also needs to be able to compile against full source as well

@rvagg only for older versions, right?

No, also for when you're compiling from the source tarball you can --nodedir and same for when you're doing it directly from the repo—like a lot of us in core have to do; we're stuck with both

rvagg · 2015-09-04T16:48:33Z

comments added to process-release.js

othiym23 · 2015-09-04T17:30:57Z

I've given it a quick pass-through but I want to pull it down and run the tests, which is going to take an hour or two. So far: looks good, like the addition of tests, npm will probably just integrate this as-is once it's published.

If it weren't for Node 4, I'd probably only land this in npm@3, because there are probably going to be enough knock-on effects from the switch as to cause issues (or at least surprises) for some users upgrading across npm@2 versions. I kind of wish that either npm@3 were the version going into Node 4 or this had a little more time to bake in before Node 4 drops, because I'm not sure I can anticipate all the implications of some of the changes in here.

rvagg · 2015-09-07T10:59:30Z

Updated to address the concern @bnoordhuis pointed out, it now looks for /\/win-(x86|x64)\// in the URL to switch for the two arch's, so you have to have withe /win-x86/ or /win-x64/ in the URL, which is harder to overload.

Unless there are no objections, I might merge this and release a v3 eh?

bnoordhuis · 2015-09-07T11:32:34Z

@rvagg The regex still looks the same to me. Can it be that you didn't push the commit?

rvagg · 2015-09-07T11:47:23Z

sorry, it seems that I have process.release branches on both this repo and my own fork .. pushed to the one that's behind this PR now.

  , bitsre = /\/win-(x86|x64)\//
  , bitsreV3 = /\/win-(x86|ia32|x64)\// // io.js v3.x.x shipped with "ia32" but should
                                        // have been "x86"

bnoordhuis · 2015-09-07T12:04:48Z

No problem and LGTM.

ralphtheninja · 2015-09-07T12:42:23Z

Unless there are no objections, I might merge this and release a v3 eh?

Yes! 👯

semver.satisfies() doesn't play nicely with prerelease tags

rvagg · 2015-09-07T12:51:07Z

OK, two more changes. nodejs/node#2719 reminded me that we can't trust semver.satisfies() any more so I've switched to checking major version numbers where needed, see bfaf7df. Also I've removed node_modules from the tree in 44dae28, I hope there are no objections to this.

mgol · 2015-09-07T13:15:01Z

@rvagg Argh, sorry for this suggestion, I totally forgot most semver ranges ignore pre-releases now.

mgol · 2015-09-07T14:30:04Z

nodejs/node#2719 reminded me that we can't trust semver.satisfies() any more so I've switched to checking major version numbers where needed

Any chance for applying the same patch to pangyp?

Fishrock123 · 2015-09-07T22:29:15Z

remove node_modules from tree

If we could do that elsewhere that would be nice. I'm actually curious if there are any advantages to them being in-tree for us.

rvagg · 2015-09-07T23:20:54Z

@mzgol done and released a patch update, just note that as soon as node-gyp@3 goes out I'm using the deprecate hammer on all those versions

PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

semver.satisfies() doesn't play nicely with prerelease tags PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

rvagg · 2015-09-08T00:04:49Z

merged and published as v3.0.0, thanks for all the help with this one folks!

* support process.release * support all io.js versions * support node v4+ including new download locations * enable delay-load hook by default by default * download header-only tarballs instead of full source See nodejs/node-gyp#711 for full details PR-URL: #2700 Reviewed-By: Forrest L Norvell <forrest@npmjs.com>

This was referenced Sep 4, 2015

Add io.js support #564

Closed

Node.js v4 Release Timeline nodejs/node#2522

Closed

rvagg reviewed Sep 4, 2015
View reviewed changes

rvagg added 2 commits September 7, 2015 22:48

test version major directly, don't use semver.satisfies()

bfaf7df

semver.satisfies() doesn't play nicely with prerelease tags

remove node_modules from tree

44dae28

mgol mentioned this pull request Sep 7, 2015

Build node 4.0 binaries - NODE_MODULE_VERSION 46 sass/node-sass#1122

Closed

9 tasks

rvagg added a commit that referenced this pull request Sep 8, 2015

use process.release, make aware of io.js & node v4 differences

9e9df66

PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

rvagg added a commit that referenced this pull request Sep 8, 2015

refactor for clarity, fix dist-url, add env var dist-url functionality

938dd18

PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

rvagg added a commit that referenced this pull request Sep 8, 2015

test version major directly, don't use semver.satisfies()

6dcf220

semver.satisfies() doesn't play nicely with prerelease tags PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

rvagg added a commit that referenced this pull request Sep 8, 2015

remove node_modules from tree

9720d03

PR-URL: #711 Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>

rvagg closed this Sep 8, 2015

rvagg deleted the process.release branch September 8, 2015 00:05

xzyfer mentioned this pull request Sep 8, 2015

Move back to node-gyp sass/node-sass#1124

Closed

sanemat mentioned this pull request Sep 9, 2015

deps: bump node-gyp to 3.0.1 nodejs/nan#445

Merged

brianc mentioned this pull request Sep 11, 2015

Support node v4.0 brianc/node-pg-native#27

Closed

rainabba mentioned this pull request Sep 17, 2015

Node 4.0.0 - Require heapdump - Cannot find module '../build/Debug/addon' bnoordhuis/node-heapdump#66

Closed

TooTallNate mentioned this pull request Sep 28, 2015

fix install: add iojs support #669

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use process.release, make aware of io.js & node v4 differences #711

use process.release, make aware of io.js & node v4 differences #711

rvagg commented Sep 4, 2015

rvagg Sep 4, 2015

Fishrock123 commented Sep 4, 2015

Fishrock123 commented Sep 4, 2015

rvagg commented Sep 4, 2015

rvagg commented Sep 4, 2015

othiym23 commented Sep 4, 2015

rvagg commented Sep 7, 2015

bnoordhuis commented Sep 7, 2015

rvagg commented Sep 7, 2015

bnoordhuis commented Sep 7, 2015

ralphtheninja commented Sep 7, 2015

rvagg commented Sep 7, 2015

mgol commented Sep 7, 2015

mgol commented Sep 7, 2015

Fishrock123 commented Sep 7, 2015

rvagg commented Sep 7, 2015

rvagg commented Sep 8, 2015

use process.release, make aware of io.js & node v4 differences #711

use process.release, make aware of io.js & node v4 differences #711

Conversation

rvagg commented Sep 4, 2015

rvagg Sep 4, 2015

Choose a reason for hiding this comment

Fishrock123 commented Sep 4, 2015

Fishrock123 commented Sep 4, 2015

rvagg commented Sep 4, 2015

rvagg commented Sep 4, 2015

othiym23 commented Sep 4, 2015

rvagg commented Sep 7, 2015

bnoordhuis commented Sep 7, 2015

rvagg commented Sep 7, 2015

bnoordhuis commented Sep 7, 2015

ralphtheninja commented Sep 7, 2015

rvagg commented Sep 7, 2015

mgol commented Sep 7, 2015

mgol commented Sep 7, 2015

Fishrock123 commented Sep 7, 2015

rvagg commented Sep 7, 2015

rvagg commented Sep 8, 2015