npm-shrinkwrap.json handles metadata inconsistently

[shaneholder]

If npm resolves a file from a registry when shrinkwrap is run a "resolved" property is added to the npm-shrinkwrap.json file. If npm finds that the file is in the cache then this property is not written.

npm cache clear
npm init
npm install winston
npm shrinkwrap
check shrinkwrap file and see "resolved" properties
rm -rf node_modules npm-shrinkwrap.json
npm install winston
npm shrinkwrap
check shrinkwrap file and see "resolved" properties do not exist

It seems that npm shrinkwrap should generate the same output regardless of if the file came from the cache or if the file came from the registry. This becomes more important if one is mixing private and public registries as it seems that the resolved property is used to locate a dependency.

[gabrielf]

How does this problem affect you? For us it leads to lots of dependencies being downloaded over and over again.

Example:
A shrinkwrap file for an application that only depends on async can look in the following two ways depending on (as @shaneholder pointed out) whether async was installed form the cache or not.

If async was installed with an empty cache:

{
  "name": "myapp",
  "version": "0.0.0",
  "dependencies": {
    "async": {
      "version": "0.2.9",
      "from": "async@",
      "resolved": "https://registry.npmjs.org/async/-/async-0.2.9.tgz"
    }
  }
}

Or if async was installed from the cache:

{
  "name": "myapp",
  "version": "0.0.0",
  "dependencies": {
    "async": {
      "version": "0.2.9",
      "from": "async@",
    }
  }
}

If the resolved is set then that means the tgz will be downloaded every install regardless whether it's in the cache or not.

[gabrielf]

I suspect the reason is that the package.json for a downloaded dependency looks different when installed from the cache and when it's installed fresh from the central repo. I made two folders and installed async in both, first with an empty cache and then from the cache. A diff between the two folders yield:

diff -r application-no-cache/node_modules/async/package.json application-from-cache/node_modules/async/package.json
41,45c41
<   "dist": {
<     "shasum": "1d805cdf65c236e4b351b9265610066e4ae4185c"
<   },
<   "_from": "async@",
<   "_resolved": "https://registry.npmjs.org/async/-/async-0.2.9.tgz"
---
>   "_from": "async@"

[gabrielf]

I wrote a small, but ugly, shell script that creates a shrinkwrap file that does not download unnecessary modules again on subsequent npm install.

It basically removed all modules from node_modules that experience the problem, runs npm install to make sure they get put in the npm cache. It then deletes problematic modules again and runs node_modules to make sure all modules are installed from the cache. At the end a shrinkwrap file is created where all dependencies have been installed from the cache which mean they will not have to be re-downloaded.

function deleteModulesThatWasInstalledStraightFromCentralRepo() {
  for module in  node_modules/*
  do
    MODULE_PATH="$module"
    if grep -q '"_resolved": "https://registry.npmjs.org' "$MODULE_PATH/package.json" ; then
      echo "Deleting $MODULE_PATH"
      rm -rf $MODULE_PATH
    fi
  done
}

deleteModulesThatWasInstalledStraightFromCentralRepo
rm npm-shrinkwrap.json
npm install
deleteModulesThatWasInstalledStraightFromCentralRepo
npm install
npm shrinkwrap --dev

[aseemk]

Wow, after a lot of head-scratching and debugging, I finally arrived at this same thing. Two symptoms for me:

1. Subsequent npm installs re-install dependencies. This is particularly bad for large apps as they grow. I discovered this because Heroku is starting to cache node_modules, but even so this kicks in:

   heroku/heroku-buildpack-nodejs@83104b4#commitcomment-4413053

2. Re-installing (cleanly, i.e. deleting node_modules first) from shrinkwrap then re-shrinkwrapping causes random, undeterministic changes in the shrinkwrap, due to these resolved lines appearing and disappearing.

   You'd expect shrinkwrap to be idempotent!

[mgol]

@aseemk Completely agree, I've just been hit by those 2 issues you pointed out.

[vvo]

due to these resolved lines appearing and disappearing.

This is so annoying that coworkers suggested to modify shrinkwrap.json manually or removing shrinkwrap.

[aseemk]

I discovered a very simple workaround for now — just run this after every shrinkwrap:

var fs = require('fs');
var shrinkwrap = require('./npm-shrinkwrap.json');

function replacer(key, val) {
    if (key === 'resolved' && this.from && this.version) {
        return undefined;
    } else {
        return val;
    }
}

fs.writeFileSync('./npm-shrinkwrap.json', JSON.stringify(shrinkwrap, replacer, 2));

It takes advantage of JSON.stringify()'s replacer argument to recurse into every dependency and remove the resolved keys.

You can run that manually after every shrinkwrap, or wrap it in bash to shrinkwrap too:

npm shrinkwrap
echo '''
    // above JS goes here
''' | node

Hope this helps!

[mgol]

@aseemk Thx for the script!

[vvo]

I know we should not use npm while deploying BUT the current shrinkwrap problem might worsen the current npm instabilities issues.

If all projects using shrinkwrap are re-downloading packages I guess npm is under big load due to this bug (partly).

[vvo]

Funny enough the number of expressjs downloads doubled 5 monthes ago, when this issue was created

http://npm-stat.vorba.ch/charts.html?package=express

This does not seems to be a summer effect since last year did not see the same raise.

[isaacs]

It's safe to say that this bug is not significantly affecting downloads. We're simply still in the exponential portion of npm's growth curve. Shrinkwrap usage is a rounding error.

That being said, this is indeed a bug, and should be fixed. Those "resolved" fields should be added in the cache data, not just in the installed package.json.

[vvo]

I hope too it's not affecting downloads or not too much.

But given every company using npm install and shrinkwrap in deployments and given the hype for "continuous deployment" I thought that could be one of the tiny reason for the raise in npm downloads.

[moll]

This bug, as might've said above, also affects unpublished modules that were installed manually with npm install *.tgz. Those are still looked up in NPM and therefore cause npm install to fail.

[moll]

Seems that removing those resolved properties from the shrinkwrap isn't much of a workaround as it'll then not install modules with Git repository sources from the repositories, but still from the NPM repo.

[mgol]

@moll Ah, right, that's why my package was misbehaving...

I guess I'll just be making sure that one resolved property remains...

[Raynos]

👍 any suggestions on how to fix the cache to write the resolved fields ?

[hurrymaplelad]

Quick regex fix for me. Leaves resolved for git repos.

s/,\n^\s*"from": ".*registry.npmjs.org[^,\n]*(,?$)/$1/
s/,\n^\s*"resolved": ".*registry.npmjs.org[^,\n]*(,?$)/$1/

Were these from and resolved fields pointing at npm adding some value that I'm missing, or should shrinkwrap not generate them in the first place?

[Raynos]

Run npm cache clear before rm -rf ./node_modules && npm i && npm shrinkwrap

It will generate from and resolved fields properly.

shinkwrap just puts node_modules into a file. it's npm i that writes inconsistent package.json files into node_modules tree based on install from cache or install from full registry

[hurrymaplelad]

Heads up, wrote a quick tool, clingwrap to simplify updating shrinkwrap. Interested in a PR to get this behavior in core npm?

[shaobos]

if you attempt to remove 'from' and 'resolved' fields from shrinkwrap, be aware that ynpm install would fail if the version of dependent package is a git url instead of a regular semver, e.g.
https://github.com/jshint/jshint/blob/d5f6699a0269e2892629d22cf8e2941bd3374f74/package.json#L25

[hurrymaplelad]

@tinyfrog, exactly. We only want to prune from and resolved if they point to registry.npmjs.org.

[donaldpipowitch]

We stumbled over the same problems as the angular guys (again) in socketio/engine.io-client#370. These issues seem to be related and they have long discussions about this problem as you can see in their commits.

[othiym23]

This has been moved to the npm roadmap, which we're using instead of the confusing next-* labels now.

[glenjamin]

fwiw, we've been using https://www.npmjs.com/package/shonkwrap to handle this for a while now and not really run into any issues.

[othiym23]

This thread is long and covers a lot of ground, but I believe the original issue of the opener (that is, resolved fields come and go in npm-shrinkwrap.json) has been addressed within npm@2 and npm@3. There are a bunch of other possible feature requests and inconsistencies mentioned in this thread, and that makes it tough to act on any of them in a coherent way, so I'm going to close this issue, and if you think there's an outstanding issue with shrinkwrap that isn't covered by issues with the shrinkwrap label, please open new issues.

[vkrol]

The behavior from the issue description continues to occur in the npm@3.8.0. This is very annoying.

that is, resolved fields come and go in npm-shrinkwrap.json

@othiym23 please, can you reopen this issue?

[mockdeep]

I'm seeing the same issue. Shrinkwrapping on one computer differs from another:

"dependencies": {
  "babel-code-frame": {
    "version": "6.7.2",
      "from": "babel-code-frame@>=6.7.2 <7.0.0"
    }
  },
  ...
}

Becomes:

"dependencies": {
  "babel-code-frame": {
    "version": "6.7.2",
    "from": "babel-code-frame@6.7.2",
    "resolved": "https://registry.npmjs.org/babel-code-frame/-/babel-code-frame-6.7.2.tgz"
  },
  ...
}

The shorter version is definitely preferable, since otherwise it ends up being TMI, and verifying changelogs on updated packages is more of a headache, especially since Github will often not even show the lengthy diff.

[brianpchsu]

I also have the same issue with npm @3.10.3. Basically, I just updated my project number in package.json, but somehow the dependencies verion in shrinkwrap.json changed as well.

From:

"concat-stream": {
      "version": "1.4.10",
      "from": "concat-stream@>=1.4.7 <1.5.0",
      "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-1.4.10.tgz"
    },
    "connect-multiparty": {
      "version": "2.0.0",
      "from": "connect-multiparty@>=2.0.0 <3.0.0",
   ...
    },

To:

"concat-stream": {
      "version": "1.4.10",
      "from": "concat-stream@>=1.4.7 <1.5.0",
      "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-1.4.10.tgz"
    },
    "connect-multiparty": {
      "version": "2.0.0",
      "from": "connect-multiparty@latest",
    },