release: bundling npm

[rvagg]

We didn't get to this discussion in the TC meeting today even though @piscisaureus put it on the agenda (I didn't see it and we ran out of time anyway). We need to resolve how we're going to handle bundling npm for releases, specifically the 1.0.0.

Some ideas:

• Release as it is now
• Upgrade to current 2.x.y as dictated by npm folks
• Remove npm completely from deps and add a build step that downloads an appropriate version so you can make install it like normal and also bundle it in a tarball for release

Whatever we go for, I'd like to see the npm client team be made responsible for decisions about what version to bundle (they can also take responsibility for breakages!). Perhaps we need to add @othiym23 as a collaborator and let him take on deps/npm, I doubt anybody currently on the team other than @isaacs really wants to take responsibility for it.

Suggestions so we can move forward with a decision for the release?

[isaacs]

I asked @othiym23 to submit a pull request adding the version of npm that he thinks is best. I'd also be in favor of an automated process that always pulls the latest npm into the source tree, perhaps as part of the CI process.

However it happens, my preference is always to pull in the latest npm with each release. Here's why:

1. There is almost never a downside to upgrading the npm client. In the rare cases that it presents a problem, it is for one of two cases.

   When there is a problem upgrading the npm client because npm introduces a bug, it is a short-lived problem, since the next release is only a week away, and an easily worked-around problem in the meantime with npm i -g npm. That is, the problem is fixed by continuing to upgrade, so this is at least as much an argument for shipping the latest npm as an argument against.

   Very rarely, there is a problem upgrading the npm client because a user requires an old version for some reason (eg, compatibility with a third-party private registry, integration into some odd custom build process, etc.) In that case, it's best for those users to set up their systems to pull a specific npm version, and nothing we ship is guaranteed to be good anyway.

2. Changes to the public npm registry frequently leave users of outdated npm versions behind. We do what we can to maintain API backwards compatibility, but sometimes this is not feasible. For example, the npm 1.x series made assumptions about the registry architecture that turned out to not be scalable. When we distributed the npm service to multiple databases behind a load balancer, some of those assumptions were no longer valid. Even though the API didn't change, logins and other server-state actions exposed nasty race conditions.

3. Because of the nature of npm as a "many to many" platform (almost as bad as web browsers and web pages), we are very limited in the breaking changes we can make to the client. It is virtually impossible to make sweeping npm client changes that leave old packages behind, so the idea that a change to the npm client will render run time behavior incompatible is wildly unlikely. (The last such change was the removal of modules and directories.lib from package.json in npm 0.3, which shipped with Node.js 0.4. And it nearly caused riots, even then!)

So, there is lots of hazards to not shipping the latest npm, virtually no down-side to keeping up to date, and many benefits.

[isaacs]

And -1 from me to all forms of "download stuff as part of make install", that user experience is just obnoxious. Optimizing for tarball size is 100% the wrong metric, adds complexity, and increases the likelihood of novel problems in the user's environment.

[othiym23]

I have opened a PR containing an update to npm@2.1.18 in #266.

Unsurprisingly, I am as of one mind with @isaacs as to my preferred policy for io.js on npm versions, both in principle and in the particulars he outlines above. I don't have a lot of extra time right now, but I can carve some out to help put together a way to automatically keep the io.js version of deps/npm up to date with npm@latest.

[dougwilson]

an easily worked-around problem in the meantime with npm i -g npm

crosses fingers that this will actually work on Windows with io.js

[othiym23]

We have some work to do on the installer, still, but I'm doing all of my development work with Windows now and was using io.js built from source yesterday with npm and nothing exploded. You can upgrade npm on Windows today, although it's kind of a pain in the ass.

[dougwilson]

I know there is currently a convoluted method, I just don't think it's sane :)

[jamonholmgren]

Using brew to install iojs and get this warning:

iojs currently requires a patched npm (i.e. not the npm installed by node).

Is there a recommended way to install that patched npm?

[jamonholmgren]

Maybe there should be an iopm. :)

[piscisaureus]

@isaacs @othiym23

Would it be an idea to turn the "node binary name" into a configuration option.
Currently npm doesn't work at all if the user doesn't (sym)link node to iojs.

[othiym23]

Maybe not a configuration option, but a build / release option. It wouldn't be that tough to patch bin/npm to use optionally use a $NPM_RUNTIME option (or something else with a name that's agnostic to the name of the executable).

That said, I'm not sure this would be a very satisfactory hackaround, due to how many lifecycle scripts are hardcoded to call node. It would be possible to do something clever to patch that up at runtime, but that would only work for scripts defined in package.json, and not scripts shipped with packages. It is far easier to come up with a solution to the packaging problem for Homebrew, especially if the end goal for io.js is for it and Node to merge back together.

[Fishrock123]

See also #895

Closing this issue, as it was for the initial release.