Skip to content
This repository has been archived by the owner. It is now read-only.

npm-shrinkwrap.json handles metadata inconsistently #3581

Closed
shaneholder opened this issue Jun 18, 2013 · 66 comments
Closed

npm-shrinkwrap.json handles metadata inconsistently #3581

shaneholder opened this issue Jun 18, 2013 · 66 comments

Comments

@shaneholder
Copy link

@shaneholder shaneholder commented Jun 18, 2013

If npm resolves a file from a registry when shrinkwrap is run a "resolved" property is added to the npm-shrinkwrap.json file. If npm finds that the file is in the cache then this property is not written.

npm cache clear
npm init
npm install winston
npm shrinkwrap
check shrinkwrap file and see "resolved" properties
rm -rf node_modules npm-shrinkwrap.json
npm install winston
npm shrinkwrap
check shrinkwrap file and see "resolved" properties do not exist

It seems that npm shrinkwrap should generate the same output regardless of if the file came from the cache or if the file came from the registry. This becomes more important if one is mixing private and public registries as it seems that the resolved property is used to locate a dependency.

@gabrielf
Copy link
Contributor

@gabrielf gabrielf commented Jul 4, 2013

How does this problem affect you? For us it leads to lots of dependencies being downloaded over and over again.

Example:
A shrinkwrap file for an application that only depends on async can look in the following two ways depending on (as @shaneholder pointed out) whether async was installed form the cache or not.

If async was installed with an empty cache:

{
  "name": "myapp",
  "version": "0.0.0",
  "dependencies": {
    "async": {
      "version": "0.2.9",
      "from": "async@",
      "resolved": "https://registry.npmjs.org/async/-/async-0.2.9.tgz"
    }
  }
}

Or if async was installed from the cache:

{
  "name": "myapp",
  "version": "0.0.0",
  "dependencies": {
    "async": {
      "version": "0.2.9",
      "from": "async@",
    }
  }
}

If the resolved is set then that means the tgz will be downloaded every install regardless whether it's in the cache or not.

@gabrielf
Copy link
Contributor

@gabrielf gabrielf commented Jul 4, 2013

I suspect the reason is that the package.json for a downloaded dependency looks different when installed from the cache and when it's installed fresh from the central repo. I made two folders and installed async in both, first with an empty cache and then from the cache. A diff between the two folders yield:

diff -r application-no-cache/node_modules/async/package.json application-from-cache/node_modules/async/package.json
41,45c41
<   "dist": {
<     "shasum": "1d805cdf65c236e4b351b9265610066e4ae4185c"
<   },
<   "_from": "async@",
<   "_resolved": "https://registry.npmjs.org/async/-/async-0.2.9.tgz"
---
>   "_from": "async@"
@gabrielf
Copy link
Contributor

@gabrielf gabrielf commented Jul 4, 2013

I wrote a small, but ugly, shell script that creates a shrinkwrap file that does not download unnecessary modules again on subsequent npm install.

It basically removed all modules from node_modules that experience the problem, runs npm install to make sure they get put in the npm cache. It then deletes problematic modules again and runs node_modules to make sure all modules are installed from the cache. At the end a shrinkwrap file is created where all dependencies have been installed from the cache which mean they will not have to be re-downloaded.

function deleteModulesThatWasInstalledStraightFromCentralRepo() {
  for module in  node_modules/*
  do
    MODULE_PATH="$module"
    if grep -q '"_resolved": "https://registry.npmjs.org' "$MODULE_PATH/package.json" ; then
      echo "Deleting $MODULE_PATH"
      rm -rf $MODULE_PATH
    fi
  done
}

deleteModulesThatWasInstalledStraightFromCentralRepo
rm npm-shrinkwrap.json
npm install
deleteModulesThatWasInstalledStraightFromCentralRepo
npm install
npm shrinkwrap --dev
@aseemk
Copy link

@aseemk aseemk commented Oct 24, 2013

Wow, after a lot of head-scratching and debugging, I finally arrived at this same thing. Two symptoms for me:

  1. Subsequent npm installs re-install dependencies. This is particularly bad for large apps as they grow. I discovered this because Heroku is starting to cache node_modules, but even so this kicks in:

    heroku/heroku-buildpack-nodejs@83104b4#commitcomment-4413053

  2. Re-installing (cleanly, i.e. deleting node_modules first) from shrinkwrap then re-shrinkwrapping causes random, undeterministic changes in the shrinkwrap, due to these resolved lines appearing and disappearing.

    You'd expect shrinkwrap to be idempotent!

aseemk referenced this issue in heroku/heroku-buildpack-nodejs Oct 24, 2013
Ensure npm scripts are run after restoring cache
@mgol
Copy link

@mgol mgol commented Nov 9, 2013

@aseemk Completely agree, I've just been hit by those 2 issues you pointed out.

@vvo
Copy link

@vvo vvo commented Nov 9, 2013

due to these resolved lines appearing and disappearing.

This is so annoying that coworkers suggested to modify shrinkwrap.json manually or removing shrinkwrap.

@aseemk
Copy link

@aseemk aseemk commented Nov 12, 2013

I discovered a very simple workaround for now — just run this after every shrinkwrap:

var fs = require('fs');
var shrinkwrap = require('./npm-shrinkwrap.json');

function replacer(key, val) {
    if (key === 'resolved' && this.from && this.version) {
        return undefined;
    } else {
        return val;
    }
}

fs.writeFileSync('./npm-shrinkwrap.json', JSON.stringify(shrinkwrap, replacer, 2));

It takes advantage of JSON.stringify()'s replacer argument to recurse into every dependency and remove the resolved keys.

You can run that manually after every shrinkwrap, or wrap it in bash to shrinkwrap too:

npm shrinkwrap
echo '''
    // above JS goes here
''' | node

Hope this helps!

@mgol
Copy link

@mgol mgol commented Nov 12, 2013

@aseemk Thx for the script!

@vvo
Copy link

@vvo vvo commented Nov 18, 2013

I know we should not use npm while deploying BUT the current shrinkwrap problem might worsen the current npm instabilities issues.

If all projects using shrinkwrap are re-downloading packages I guess npm is under big load due to this bug (partly).

@vvo
Copy link

@vvo vvo commented Nov 19, 2013

Funny enough the number of expressjs downloads doubled 5 monthes ago, when this issue was created

selection_066

selection_067

http://npm-stat.vorba.ch/charts.html?package=express

This does not seems to be a summer effect since last year did not see the same raise.

@isaacs
Copy link
Member

@isaacs isaacs commented Nov 19, 2013

It's safe to say that this bug is not significantly affecting downloads. We're simply still in the exponential portion of npm's growth curve. Shrinkwrap usage is a rounding error.

That being said, this is indeed a bug, and should be fixed. Those "resolved" fields should be added in the cache data, not just in the installed package.json.

@vvo
Copy link

@vvo vvo commented Dec 4, 2013

I hope too it's not affecting downloads or not too much.

But given every company using npm install and shrinkwrap in deployments and given the hype for "continuous deployment" I thought that could be one of the tiny reason for the raise in npm downloads.

@moll
Copy link

@moll moll commented Dec 4, 2013

This bug, as might've said above, also affects unpublished modules that were installed manually with npm install *.tgz. Those are still looked up in NPM and therefore cause npm install to fail.

@moll
Copy link

@moll moll commented Dec 17, 2013

Seems that removing those resolved properties from the shrinkwrap isn't much of a workaround as it'll then not install modules with Git repository sources from the repositories, but still from the NPM repo.

@mgol
Copy link

@mgol mgol commented Dec 17, 2013

@moll Ah, right, that's why my package was misbehaving...

I guess I'll just be making sure that one resolved property remains...

@Raynos
Copy link
Contributor

@Raynos Raynos commented Jan 15, 2014

👍 any suggestions on how to fix the cache to write the resolved fields ?

@hurrymaplelad
Copy link

@hurrymaplelad hurrymaplelad commented Feb 5, 2014

Quick regex fix for me. Leaves resolved for git repos.

s/,\n^\s*"from": ".*registry.npmjs.org[^,\n]*(,?$)/$1/
s/,\n^\s*"resolved": ".*registry.npmjs.org[^,\n]*(,?$)/$1/

Were these from and resolved fields pointing at npm adding some value that I'm missing, or should shrinkwrap not generate them in the first place?

@Raynos
Copy link
Contributor

@Raynos Raynos commented Feb 5, 2014

Run npm cache clear before rm -rf ./node_modules && npm i && npm shrinkwrap

It will generate from and resolved fields properly.

shinkwrap just puts node_modules into a file. it's npm i that writes inconsistent package.json files into node_modules tree based on install from cache or install from full registry

@hurrymaplelad
Copy link

@hurrymaplelad hurrymaplelad commented Feb 10, 2014

Heads up, wrote a quick tool, clingwrap to simplify updating shrinkwrap. Interested in a PR to get this behavior in core npm?

@shaobos
Copy link

@shaobos shaobos commented Feb 12, 2014

if you attempt to remove 'from' and 'resolved' fields from shrinkwrap, be aware that ynpm install would fail if the version of dependent package is a git url instead of a regular semver, e.g.
https://github.com/jshint/jshint/blob/d5f6699a0269e2892629d22cf8e2941bd3374f74/package.json#L25

@hurrymaplelad
Copy link

@hurrymaplelad hurrymaplelad commented Feb 12, 2014

@tinyfrog, exactly. We only want to prune from and resolved if they point to registry.npmjs.org.

mgol referenced this issue in angular/angular.js Mar 12, 2014
We need to be able to build angular at older shas, without the lock file / shrinkwrap file
the dependencies will resolve differently on different machines and at different times.

This will help us avoid broken builds and hard to track down issues.

I had to manually edit this file after it was generated because `npm shrinkwrap` will install
optional dependencies as if they were hard dependencies.

See: npm/npm#2679 (comment)

My manual edit:

```
diff --git a/npm-shrinkwrap.json b/npm-shrinkwrap.json
index 756df44..dc157eb 100644
--- a/npm-shrinkwrap.json
+++ b/npm-shrinkwrap.json
@@ -3110,19 +3110,7 @@
         "chokidar": {
           "version": "0.8.1",
           "from": "https://registry.npmjs.org/chokidar/-/chokidar-0.8.1.tgz",
-          "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-0.8.1.tgz",
-          "dependencies": {
-            "fsevents": {
-              "version": "0.1.6",
-              "from": "fsevents@0.1.6",
-              "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-0.1.6.tgz"
-            },
-            "recursive-readdir": {
-              "version": "0.0.2",
-              "from": "recursive-readdir@0.0.2",
-              "resolved": "https://registry.npmjs.org/recursive-readdir/-/recursive-readdir-0.0.2.tgz"
-            }
-          }
+          "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-0.8.1.tgz"
         },
         "glob": {
           "version": "3.2.9",
```

Additionally chokidar doesn't list the dependencies above as optional, but that will hopefully
be soon fixed: paulmillr/chokidar#106

In the meantime the patch from the PR above needs to be applied to
node_modules/karma/node_modules/chokidar/package.json before running `npm shrinkwrap`

----

After this change is applied, angular core developers don't need to do anything differently,
except when updating dependencies we need to call `npm update && npm shrinkwrap --dev`
followed by reappling my patch above until npm's bug.

Closes #6653
IgorMinar added a commit to IgorMinar/angular.js that referenced this issue Mar 14, 2014
from our experiements it appears that the presense or absense of the from and resolved properties
makes no difference on the behavior of  but  updates these properties
with different values depending on different state of the cache and node_modules.

So in order to get clean diffs during updates, we are just going to drop these properties and have
a script to do this automatically.

Long term this should be fixed in npm: npm/npm#3581
IgorMinar added a commit to IgorMinar/angular.js that referenced this issue Mar 14, 2014
This is to deal with npm/npm#3581

See the previous commit for more info.

Closes angular#6672
IgorMinar added a commit to IgorMinar/angular.js that referenced this issue Mar 15, 2014
This is to deal with npm/npm#3581

See the previous commit for more info.

Closes angular#6672
IgorMinar added a commit to IgorMinar/angular.js that referenced this issue Mar 15, 2014
from our experiements it appears that the presense or absense of the from and resolved properties
makes no difference on the behavior of  but  updates these properties
with different values depending on different state of the cache and node_modules.

So in order to get clean diffs during updates, we are just going to drop these properties and have
a script to do this automatically.

Long term this should be fixed in npm: npm/npm#3581
@donaldpipowitch
Copy link

@donaldpipowitch donaldpipowitch commented May 12, 2015

We stumbled over the same problems as the angular guys (again) in socketio/engine.io-client#370. These issues seem to be related and they have long discussions about this problem as you can see in their commits.

@othiym23
Copy link
Contributor

@othiym23 othiym23 commented Jul 15, 2015

This has been moved to the npm roadmap, which we're using instead of the confusing next-* labels now.

@glenjamin
Copy link
Contributor

@glenjamin glenjamin commented Aug 5, 2015

fwiw, we've been using https://www.npmjs.com/package/shonkwrap to handle this for a while now and not really run into any issues.

maxlang pushed a commit to maxlang/cockroach that referenced this issue Sep 29, 2015
Unfortunately, npm shrinkwrap is inconsistent.

npm/npm#3581

Also the npm-shrinkwrap format has changed once more.
OliverJAsh added a commit to guardian/frontend that referenced this issue Oct 22, 2015
`npm shrinkwrap` spits out different metadata depending on the npm version, which makes our diffs much too difficult to read properly.

Browsing [the issue](npm/npm#3581) I found [a little script](https://github.com/angular/angular.js/blob/master/scripts/npm/clean-shrinkwrap.js) we can use to tidy up the shrinkwrap and keep it consistent.

I've also tried to document the npm install process.
@othiym23
Copy link
Contributor

@othiym23 othiym23 commented Nov 6, 2015

This thread is long and covers a lot of ground, but I believe the original issue of the opener (that is, resolved fields come and go in npm-shrinkwrap.json) has been addressed within npm@2 and npm@3. There are a bunch of other possible feature requests and inconsistencies mentioned in this thread, and that makes it tough to act on any of them in a coherent way, so I'm going to close this issue, and if you think there's an outstanding issue with shrinkwrap that isn't covered by issues with the shrinkwrap label, please open new issues.

@vkrol
Copy link

@vkrol vkrol commented Mar 7, 2016

The behavior from the issue description continues to occur in the npm@3.8.0. This is very annoying.

that is, resolved fields come and go in npm-shrinkwrap.json

@othiym23 please, can you reopen this issue?

@mockdeep
Copy link

@mockdeep mockdeep commented Mar 11, 2016

I'm seeing the same issue. Shrinkwrapping on one computer differs from another:

"dependencies": {
  "babel-code-frame": {
    "version": "6.7.2",
      "from": "babel-code-frame@>=6.7.2 <7.0.0"
    }
  },
  ...
}

Becomes:

"dependencies": {
  "babel-code-frame": {
    "version": "6.7.2",
    "from": "babel-code-frame@6.7.2",
    "resolved": "https://registry.npmjs.org/babel-code-frame/-/babel-code-frame-6.7.2.tgz"
  },
  ...
}

The shorter version is definitely preferable, since otherwise it ends up being TMI, and verifying changelogs on updated packages is more of a headache, especially since Github will often not even show the lengthy diff.

@brianpchsu
Copy link

@brianpchsu brianpchsu commented Aug 1, 2016

I also have the same issue with npm @3.10.3. Basically, I just updated my project number in package.json, but somehow the dependencies verion in shrinkwrap.json changed as well.

From:

"concat-stream": {
      "version": "1.4.10",
      "from": "concat-stream@>=1.4.7 <1.5.0",
      "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-1.4.10.tgz"
    },
    "connect-multiparty": {
      "version": "2.0.0",
      "from": "connect-multiparty@>=2.0.0 <3.0.0",
   ...
    },

To:

"concat-stream": {
      "version": "1.4.10",
      "from": "concat-stream@>=1.4.7 <1.5.0",
      "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-1.4.10.tgz"
    },
    "connect-multiparty": {
      "version": "2.0.0",
      "from": "connect-multiparty@latest",
    },
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.