npm-shrinkwrap.json handles metadata inconsistently

[shaneholder]

If npm resolves a file from a registry when shrinkwrap is run a "resolved" property is added to the npm-shrinkwrap.json file. If npm finds that the file is in the cache then this property is not written.

npm cache clear
npm init
npm install winston
npm shrinkwrap
check shrinkwrap file and see "resolved" properties
rm -rf node_modules npm-shrinkwrap.json
npm install winston
npm shrinkwrap
check shrinkwrap file and see "resolved" properties do not exist

It seems that npm shrinkwrap should generate the same output regardless of if the file came from the cache or if the file came from the registry. This becomes more important if one is mixing private and public registries as it seems that the resolved property is used to locate a dependency.

[gabrielf]

How does this problem affect you? For us it leads to lots of dependencies being downloaded over and over again.

Example:
A shrinkwrap file for an application that only depends on async can look in the following two ways depending on (as @shaneholder pointed out) whether async was installed form the cache or not.

If async was installed with an empty cache:

{
  "name": "myapp",
  "version": "0.0.0",
  "dependencies": {
    "async": {
      "version": "0.2.9",
      "from": "async@",
      "resolved": "https://registry.npmjs.org/async/-/async-0.2.9.tgz"
    }
  }
}

Or if async was installed from the cache:

{
  "name": "myapp",
  "version": "0.0.0",
  "dependencies": {
    "async": {
      "version": "0.2.9",
      "from": "async@",
    }
  }
}

If the resolved is set then that means the tgz will be downloaded every install regardless whether it's in the cache or not.

[gabrielf]

I suspect the reason is that the package.json for a downloaded dependency looks different when installed from the cache and when it's installed fresh from the central repo. I made two folders and installed async in both, first with an empty cache and then from the cache. A diff between the two folders yield:

diff -r application-no-cache/node_modules/async/package.json application-from-cache/node_modules/async/package.json
41,45c41
<   "dist": {
<     "shasum": "1d805cdf65c236e4b351b9265610066e4ae4185c"
<   },
<   "_from": "async@",
<   "_resolved": "https://registry.npmjs.org/async/-/async-0.2.9.tgz"
---
>   "_from": "async@"

[gabrielf]

I wrote a small, but ugly, shell script that creates a shrinkwrap file that does not download unnecessary modules again on subsequent npm install.

It basically removed all modules from node_modules that experience the problem, runs npm install to make sure they get put in the npm cache. It then deletes problematic modules again and runs node_modules to make sure all modules are installed from the cache. At the end a shrinkwrap file is created where all dependencies have been installed from the cache which mean they will not have to be re-downloaded.

function deleteModulesThatWasInstalledStraightFromCentralRepo() {
  for module in  node_modules/*
  do
    MODULE_PATH="$module"
    if grep -q '"_resolved": "https://registry.npmjs.org' "$MODULE_PATH/package.json" ; then
      echo "Deleting $MODULE_PATH"
      rm -rf $MODULE_PATH
    fi
  done
}

deleteModulesThatWasInstalledStraightFromCentralRepo
rm npm-shrinkwrap.json
npm install
deleteModulesThatWasInstalledStraightFromCentralRepo
npm install
npm shrinkwrap --dev

[aseemk]

Wow, after a lot of head-scratching and debugging, I finally arrived at this same thing. Two symptoms for me:

1. Subsequent npm installs re-install dependencies. This is particularly bad for large apps as they grow. I discovered this because Heroku is starting to cache node_modules, but even so this kicks in:

   heroku/heroku-buildpack-nodejs@83104b4#commitcomment-4413053

2. Re-installing (cleanly, i.e. deleting node_modules first) from shrinkwrap then re-shrinkwrapping causes random, undeterministic changes in the shrinkwrap, due to these resolved lines appearing and disappearing.

   You'd expect shrinkwrap to be idempotent!

[mgol]

@aseemk Completely agree, I've just been hit by those 2 issues you pointed out.

[vvo]

due to these resolved lines appearing and disappearing.

This is so annoying that coworkers suggested to modify shrinkwrap.json manually or removing shrinkwrap.

[aseemk]

I discovered a very simple workaround for now — just run this after every shrinkwrap:

var fs = require('fs');
var shrinkwrap = require('./npm-shrinkwrap.json');

function replacer(key, val) {
    if (key === 'resolved' && this.from && this.version) {
        return undefined;
    } else {
        return val;
    }
}

fs.writeFileSync('./npm-shrinkwrap.json', JSON.stringify(shrinkwrap, replacer, 2));

It takes advantage of JSON.stringify()'s replacer argument to recurse into every dependency and remove the resolved keys.

You can run that manually after every shrinkwrap, or wrap it in bash to shrinkwrap too:

npm shrinkwrap
echo '''
    // above JS goes here
''' | node

Hope this helps!

[mgol]

@aseemk Thx for the script!

[vvo]

I know we should not use npm while deploying BUT the current shrinkwrap problem might worsen the current npm instabilities issues.

If all projects using shrinkwrap are re-downloading packages I guess npm is under big load due to this bug (partly).

[vvo]

Funny enough the number of expressjs downloads doubled 5 monthes ago, when this issue was created

http://npm-stat.vorba.ch/charts.html?package=express

This does not seems to be a summer effect since last year did not see the same raise.

[isaacs]

It's safe to say that this bug is not significantly affecting downloads. We're simply still in the exponential portion of npm's growth curve. Shrinkwrap usage is a rounding error.

That being said, this is indeed a bug, and should be fixed. Those "resolved" fields should be added in the cache data, not just in the installed package.json.

[vvo]

I hope too it's not affecting downloads or not too much.

But given every company using npm install and shrinkwrap in deployments and given the hype for "continuous deployment" I thought that could be one of the tiny reason for the raise in npm downloads.

[moll]

This bug, as might've said above, also affects unpublished modules that were installed manually with npm install *.tgz. Those are still looked up in NPM and therefore cause npm install to fail.