Change made in npm 3.8.8 breaks shrinkpacked projects

[gleseur]

We are not able to run npm install with recent npm versions

Expected behaviour

npm install with success on all (at least recent npm versions)

For now, we'll just keep on using a compatible npm version, but at some point, this is going to be an issue.
...

Actual behaviour

Did not try for all npm versions, but with 3.8.3 it is working fine, while with 3.8.9 or 3.9.2 (currently latest) npm install no longer finds the files.

From the error logs, the issue is pretty clear, the meaning of './' changed in a version between 3.8.3 and 3.8.9 and now, for dependencies, npm is building the path starting from the node_modules/package repertory instead of the working directory
...

Steps to reproduce

on a shrinkwrapped project
npm install -g npm@3.8.3
rm -rf node_modules
npm install
=> all good

npm install -g npm@3.9.2
rm -rf node_modules
npm install
=> all dependencies are failing
...

Software versions used

OS         :  Ubuntu 14.04
node.js    :  5.10.1
npm        : 3.8.3 => ok, 3.8.9 or 3.9.2 => not ok
shrinkpack : 0.12.3

Log output when running shrinkpack

$ rm -rf node_modules
$ npm install
Running node v5.10.1
npm ERR! addLocal Could not install /.../node_modules/dom-serializer/node_shrinkwrap/domelementtype-1.1.3.tgz
npm ERR! addLocal Could not install /.../node_modules/dom-serializer/node_shrinkwrap/entities-1.1.1.tgz
npm ERR! addLocal Could not install /.../node_modules/readline2/node_shrinkwrap/ansi-regex-1.1.1.tgz
npm ERR! addLocal Could not install /.../node_modules/readline2/node_shrinkwrap/mute-stream-0.0.4.tgz
npm ERR! addLocal Could not install /.../node_modules/readline2/node_shrinkwrap/strip-ansi-2.0.1.tgz
npm ERR! addLocal Could not install /.../node_modules/camelcase-keys/node_shrinkwrap/camelcase-2.1.0.tgz
npm ERR! addLocal Could not install /.../node_modules/indent-string/node_shrinkwrap/repeating-2.0.0.tgz
...

[zenyr]

npm 3.8.7 also seems to work, I guess?
(used npm i -g npm3 a few weeks ago, resulting npm3 -v == 3.8.7 & npm -v == 2.15.0)

A colleague of mine who had set her system up recently is also having this issue.
(not sure of the exact npm@3 version but updating npm was not working for her, while 100% identical shrinkwrap.json had worked on mine & another colleague just fine.. removing shrinkwrap.json and installing over HTTP fixedworkaround-ed the issue temporarily. )

[jacobbuck]

Can also confirm that the issue is replicable on npm 3.8.8 and newer.

[JamieMason]

Thanks @jacobbuck @zenyr @gleseur, I noticed this myself recently too. Another npm quirk to add to the ever-growing list to work around 😣

[JamieMason]

Summary of what's known so far;

1. It can be reproduced using the following steps

$ npm i -g shrinkpack
$ mkdir some-project
$ cd some-project
$ npm init -y
$ npm install --save ava
$ npm shrinkwrap
$ shrinkpack
$ rm -rf node_modules
$ npm i

2. The package.json and npm-shrinkwrap.json are attached to this issue.
3. The above works up to version v3.8.7 of npm, then fails in v3.8.8.
4. v3.8.8 and up gives the following error for some (but not all) of the entries in npm-shrinkwrap.json.

npm ERR! addLocal Could not install /Users/jamie/Sites/test/node_modules/meow/node_shrinkwrap/minimist-1.2.0.tgz

5. The correct path is /Users/jamie/Sites/test/meow/node_shrinkwrap/minimist-1.2.0.tgz.
6. node_modules is being inserted into some of the file paths read from npm-shrinkwrap.json in https://github.com/npm/npm/blob/cde367fbb6eebc5db68a44b12a5c7bea158d70db/lib/utils/child-path.js#L9.
7. These are the commits to npm between v3.8.7 and v3.8.8.

@iarna I'd be really grateful if you could offer some ideas as to what might have caused this difference in behaviour. Is this an intentional change with a recommended workaround? Or could it perhaps be some kind of regression.

Any input you could give would be hugely welcome.

Thanks all.

[JamieMason]

Among the changes were updates to these packages used by npm;

https://github.com/npm/npm-package-arg
– Parse the things that can be arguments to npm install http://npm.im/npm-package-arg

https://github.com/npm/realize-package-specifier
– Like npm-package-arg, but more so, producing full file paths and differentiating local tar and directory sources.

It's probable that it was a change in one of these two projects that has introduced this behaviour in npm.

[JamieMason]

cc-ing @zkat – would you be able to help us please?