Missing dependencies after running npm install a second time

[schmod]

I'm opening this issue because:

• npm is crashing.
• npm is producing an incorrect install.
• npm is doing something I don't understand.
• Other (see below for feature requests):

What's going wrong?

When there is an existing lockfile, npm5 does not install all necessary packages. (This issue specifically affects any project that depends on foundry-kue-scheduler)

The initial invocation of npm install installs the correct set of dependencies, and creates a lockfile as I would expect.

When I delete node_modules, and run npm install again (against the same lockfile), I get a different set of packages (several transitive dependencies are missing), and npm alters the contents package-lock.json.

How can the CLI team reproduce the problem?

This seems to be a minimal test-case that shows how npm install is broken for any package that depends on foundry-kue-scheduler:

# scaffold a minimal project
$ mkdir npm5-test
$ cd npm5-test
$ npm init -y

# install/save my first/only dependency
$ npm i foundry-kue-scheduler

npm WARN deprecated redlock@2.1.1: possible critical bug, see https://github.com/mike-marcacci/node-redlock/issues/31
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN npm5-test@1.0.0 No description
npm WARN npm5-test@1.0.0 No repository field.

added 203 packages in 11.601s

# npm installed my packages and creates a lockfile
$ ls -lh
total 88
drwxr-xr-x  194 andrew  staff   6.4K May 31 15:09 node_modules
-rw-r--r--    1 andrew  staff    38K May 31 15:09 package-lock.json
-rw-r--r--    1 andrew  staff   286B May 31 15:09 package.json

# one of my transitive dependencies is present
$ ls -lh node_modules/kue/package.json
-rw-r--r--  1 andrew  staff   2.0K May 31 15:09 node_modules/kue/package.json

# At this point, everything is OK.  
# NPM has installed the set of packages that I expected.
# I can run `npm install` again, and it correctly reports that there's nothing to do.

# now, let's remove node_modules and reinstall (as though I'm installing
# packages for the first time in a repo that has package-lock.json checked-in,
# and node_modules is listed in .gitignore )
$ rm -rf node_modules/
$ npm i

npm WARN npm5-test@1.0.0 No description
npm WARN npm5-test@1.0.0 No repository field.

added 192 packages in 4.73s

# note that fewer packages were installed, and npm MODIFIED the extant lockfile
$ ls -lh
total 88
drwxr-xr-x  185 andrew  staff   6.1K May 31 15:11 node_modules
-rw-r--r--    1 andrew  staff    36K May 31 15:11 package-lock.json
-rw-r--r--    1 andrew  staff   286B May 31 15:11 package.json

# also, my transitive dependency is gone.
$ ls node_modules/kue/package.json
ls: node_modules/kue/package.json: No such file or directory

# subsequent invocations of npm install make things even worse
$ npm i

npm WARN npm5-test@1.0.0 No description
npm WARN npm5-test@1.0.0 No repository field.

removed 190 packages in 3.026s

# ahhhhhh!
$ ls -lh
total 16
drwxr-xr-x  5 andrew  staff   170B May 31 15:15 node_modules
-rw-r--r--  1 andrew  staff   503B May 31 15:15 package-lock.json
-rw-r--r--  1 andrew  staff   286B May 31 15:15 package.json

Here's a gist that shows the package structure, as well as the changes to package-lock.json between the first and second invocation of npm install.

supporting information:

• npm -v prints: 5.0.0
• node -v prints: v6.10.3
• npm config get registry prints:
• Windows, OS X/macOS, or Linux?: macOS (also verified inside Docker w/ Linux)
• Network issues:
  • Geographic location where npm was run:
  • I use a proxy to connect to the npm registry.
  • I use a proxy to connect to the web.
  • I use a proxy when downloading Git repos.
  • I access the npm registry via a VPN
  • I don't use a proxy, but have limited or unreliable internet access.
• Container:
  • I develop using Vagrant on Windows.
  • I develop using Vagrant on OS X or Linux.
  • I develop / deploy using Docker.
  • I deploy to a PaaS (Triton, Heroku).

[schmod]

Some quick discussion:

---

The broken transitive dependency (kue) referenced in the reproduction steps above was resolved through a GitHub dependency (foundry-kue-scheduler depends on a GitHub-hosted fork of kue-scheduler). I haven't been able to identify if that's the cause of this regression, but it seems like a potential culprit.

---

The fact that the "bare" npm install command could modify an existing package-lock.json was surprising to me.

I would not be surprised if npm prune, npm update, or npm install <package> modify the lockfile, but definitely was not expecting npm install to make changes on its own.

This seems like a bug, but there doesn't seem to be much documentation about the contracts that NPM makes with regards to lockfiles, or how they're meant to be used in a typical workflow. For all I know, this could be an expected behavior. More documentation would be helpful.

My $0.02 is that the "bare" npm install command should strive to be idempotent, and should not make any changes (outside of node_modules or caches) when a lockfile is present.

[schmod]

Dropping a link to #16837 here, because it seems similar.

[kelsin]

Seeing these same results. We committed our package-lock.json after a brand new npm i and then on Jenkins we (running same node version, same npm version) see missing module errors. I also see the same results where the package-lock.json gets weird and smaller after every npm i run.

By weird, I'm seeing an issue where the repository URL ends up in the "version" field of package-lock.json.

[kelsin]

And to confirm I also see this only when locally removing the node_modules folder (which is also the environment in CI)

[ericeslinger]

I am seeing something very similar when updating a single package. For example, once everything is working fine, I can do this:

➜  models (master) ✔ npm i
up to date in 1.561s
➜  models (master) ✔ rm -rf node_modules/plump 
➜  models (master) ✔ npm i
added 1 package and removed 4 packages in 1.792s
➜  models (master) ✗ 

note that I just deleted one package that I directly depend on, and then running npm i a second time correctly installs the deleted package, but seems to delete the packages that were depended on by that package I manually deleted. So now (in this case), plump is properly installed, but rxjs and merge-options are both missing (both are in the dependencies entry for plump). I'm not entirely sure where the 4 removed packages total comes from - I assume they are symbol-observable and is-plain-obj, which are the sole declared dependencies of rxjs and merge-options respectively.

What is weirder is that if I do npm i --no-save rxjs, I get back "added 2 packages", but they don't actually add (npm ls rxjs shows it's not there, and just doing an ls -laR node_modules | grep rxjs confirms this).

If I do something silly like go into a working directory that's all properly installed and do npm i --no-save rxjs (or some other package that is a declared dependency of the thing I'm working on but was already installed), it goes nuts - in this case it removed 123 packages and updated 211 packages, when in fact it should have done nothing (rxjs was already installed and up to date).

In all cases, this is only repaired by deleting both package-lock.json and node_modules and doing a fresh npm i.

[mbland]

I was going to file my own bug, but I think this one covers my experience. I've managed to create a minimal experiment and demo repo that reproduces this behavior; the README has all the details:

https://github.com/mbland/npm-v5-package-lock-bug-demo

Hopefully it's helpful here. The upshot: the problem appears related to devDependency packages whose own dependency specs are set tolatest (at least, that's how it appeared in my project with a devDependency on live-server). It also only appears to manifest when the original node_modules directory is removed and recreated.

[zkat]

@mbland awesome work! Thanks for this 🎉

[mbland]

FYI, just re-ran all the scenarios with npm v5.0.2, and got essentially the same results. The only difference is that buffer-shims is no longer installed, because readable-stream v2.2.10 no longer depends on it.

[Glavin001]

I think I was able to work around it by manually installing the missing dependencies until there were none left and letting npm update the package-lock.json as I go along.

For example:

❯ npm test

module.js:487
    throw err;
    ^

Error: Cannot find module 'pretty-error'
    at Function.Module._resolveFilename (module.js:485:15)

❯ npm install pretty-error
npm WARN solution-manager@1.0.0 No repository field.

+ pretty-error@2.1.0
added 21 packages, removed 1 package and updated 14 packages in 3.072s

❯ npm run test

module.js:487
    throw err;
    ^

Error: Cannot find module 'wrap-ansi'
    at Function.Module._resolveFilename (module.js:485:15)

❯ npm install wrap-ansi
npm WARN solution-manager@1.0.0 No repository field.

+ wrap-ansi@2.1.0
added 1 package, removed 1 package and updated 14 packages in 2.519s

❯ npm test

IT WORKS!

And npm updated both package.json and package-lock.json.

Now to confirm it actually helped anything:

❯ rm node_modules

❯ npm test

module.js:487
    throw err;
    ^

Error: Cannot find module 'yargs'
    at Function.Module._resolveFilename (module.js:485:15)

❯ npm install

added 380 packages in 55.731s

❯ npm test

IT WORKS!

Looking good!

I'm going to do some more testing, however I wanted to report my strange findings here in case it helps anyone. Hope we can get this resolved soon!

[pusherman]

I can confirm the comment by @Glavin001. We handled it by having our developers that removed node_modules remove package-lock.json before running npm install and as long as npm produced package-lock.json rather than pulling from a git repo subsequent npm install commands did not modify the lock file.

[Babazon]

I upgraded to npm5 while using Expo (for react native, it installs about 700 packages within node_modules without adding them to package.json)

I did npm install for the first time to add one package and it pruned the entire node_modules directory, and I had to downgrade to npm4.x , create a new Expo project, and copy paste my src folder and configs.

Is it a feature that npm5 prunes by default on each install or was it an attempt to make it "better than yarn"? Because that didn't work at all, I went back to yarn after all.

[mbland]

The behavior of the mbland/npm-v5-package-lock-bug-demo scenarios still holds under 5.0.3. (Note: I haven't been pushing package-lock*.json updates, as they only reflect updated package versions, not fundamental changes in npm behavior.)

Though I've a lot of detail in the README of that repo already, I've a few more observations and thoughts I'll share here.

Problems arise when only one artifact is present

It seems npm v5 gets into trouble when one of package-lock.json or node_modules is present, but the other is missing. Upgrading from a previous npm version to npm v5 and running npm install within an existing working directory would appear to fit this pattern (i.e. node_modules is present, package-lock.json is missing). As one small data point, I can confirm that adding and removing http-server multiple times as a devDependencies package in my demo repo after starting from a clean state (where both package-lock.json and node_modules are generated for the first time) behaves as expected.

First problem: missing package-lock.json results in version values becoming URLs

As I noted in my repo's README, in the case where node_modules is present and package-lock.json is regenerated, running npm install results in some version strings becoming URLs instead of semver specs (from the fsevents entry from package-lock.json):

--- package-lock-00.json	2017-06-10 14:59:58.000000000 +0200
+++ package-lock-01.json	2017-06-10 15:04:20.000000000 +0200
@@ -263,688 +263,688 @@
       "optional": true,
       "dependencies": {
         "abbrev": {
-          "version": "1.1.0",
-          "bundled": true,
+          "version": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.0.tgz",
+          "integrity": "sha1-0FVMIlZjbi9W58LlrRg/hZQo2B8=",
           "dev": true,
           "optional": true
         },
         "ansi-regex": {
-          "version": "2.1.1",
-          "bundled": true,
+          "version": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz",
+          "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=",
           "dev": true
         },
         "ansi-styles": {
-          "version": "2.2.1",
-          "bundled": true,
+          "version": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-2.2.1.tgz",
+          "integrity": "sha1-tDLdM1i2NM914eRmQ2gkBTPB3b4=",
           "dev": true,
           "optional": true
         },
[ ...snipped... ]

This screws with some packages such as istanbul-lib-instrument, which performs computations based on the version property.

Would it be worthwhile to have npm install remove the old node_modules and regenerate both in this case? It's slightly annoying, but not as annoying as having some packages fail with cryptic errors.

Major problem: missing node_modules and dependencies of devDependencies with a version spec of latest

Also, as I mentioned before in my repo's README, the major problem of packages being removed seems to affect packages with a version spec of latest that appear as dependencies of packages from the project's top-level devDependencies list. (And, of course, dependencies of those packages, and so on.) At least, that's what I've observed so far; perhaps others can confirm whether this constraint holds in their projects, or if it happens to other packages that don't fit this pattern.

Here's a demonstration of what I observe after removing node_modules and running npm install twice:

$ cd node_modules
$ jq '.dependencies' live-server/package.json 
{
  "chokidar": "^1.6.0",
  "colors": "latest",
  "connect": "3.5.x",
  "cors": "latest",
  "event-stream": "latest",
  "faye-websocket": "0.11.x",
  "http-auth": "3.1.x",
  "morgan": "^1.6.1",
  "object-assign": "latest",
  "opn": "latest",
  "proxy-middleware": "latest",
  "send": "latest",
  "serve-index": "^1.7.2"
}

$ jq -r '.dependencies | keys | @sh' live-server/package.json | xargs ls -d                 
ls: colors: No such file or directory   
ls: cors: No such file or directory     
ls: event-stream: No such file or directory                                     
ls: object-assign: No such file or directory                                    
ls: opn: No such file or directory      
ls: proxy-middleware: No such file or directory                                 
ls: send: No such file or directory     
chokidar       connect        faye-websocket http-auth      morgan         serve-index

_requiredBy becomes empty

Below is a diff for node_modules/colors/package.json, a live-server dependency with a version spec of latest, generated within mbland/npm-v5-package-lock-bug-demo.

• The first version (-) is from a clean install where npm install generates both package-lock.json and node_modules.
• The second version (+) is generated from the first run of npm install when package-lock.json is present, but node_modules is not; when npm install is run once more, the package disappears.

What appears most significant to me about the diff is that:

• _requiredBy becomes empty in the second case (+), even though it's listed in the dependencies section of node_modules/live-server/package.json across all npm install runs
• _where is shortened from ../npm-v5.0.1-package-lock-bug-demo/node_modules/live-server to ../npm-v5.0.1-package-lock-bug-demo
• other live-server dependencies that have regular semver specs, though their package.json files demonstrate similar changes, are not removed from node_modules in subsequent runs.

diff -uNr node_modules-00/colors/package.json node_modules/colors/package.json
--- node_modules-00/colors/package.json 2017-06-10 15:25:40.000000000 +0200
+++ node_modules/colors/package.json  2017-06-10 15:26:12.000000000 +0200
@@ -1,35 +1,36 @@
 {
-  "_from": "colors@latest",
+  "_args": [
+    [
+      "colors@1.1.2",
+      "/Users/msb/src/mbland/npm-v5.0.1-package-lock-bug-demo"
+    ]
+  ],
+  "_from": "colors@1.1.2",
   "_id": "colors@1.1.2",
   "_inBundle": false,
   "_integrity": "sha1-FopHAXVran9RoSzgyXv6KMCE7WM=",
   "_location": "/colors",
   "_phantomChildren": {},
   "_requested": {
-    "type": "tag",
+    "type": "version",
     "registry": true,
-    "raw": "colors@latest",
+    "raw": "colors@1.1.2",
     "name": "colors",
     "escapedName": "colors",
-    "rawSpec": "latest",
+    "rawSpec": "1.1.2",
     "saveSpec": null,
-    "fetchSpec": "latest"
+    "fetchSpec": "1.1.2"
   },
-  "_requiredBy": [
-    "/live-server"
-  ],
+  "_requiredBy": [],
   "_resolved": "https://registry.npmjs.org/colors/-/colors-1.1.2.tgz",
-  "_shasum": "168a4701756b6a7f51a12ce0c97bfa28c084ed63",
-  "_spec": "colors@latest",
-  "_where": "/Users/msb/src/mbland/npm-v5.0.1-package-lock-bug-demo/node_modules/live-server",
+  "_spec": "1.1.2",
+  "_where": "/Users/msb/src/mbland/npm-v5.0.1-package-lock-bug-demo",
   "author": {
     "name": "Marak Squires"
   },
   "bugs": {
     "url": "https://github.com/Marak/colors.js/issues"
   },
-  "bundleDependencies": false,
-  "deprecated": false,
   "description": "get colors in your node.js console",
   "engines": {
     "node": ">=0.1.90

Conclusion

Right now the easiest workaround appears to be removing both artifacts and letting npm generate both—but checking package-lock.json into the repo and generating node_modules from it is the whole point. Currently, npm install does seem to install all the right packages and versions from package-lock.json the first time you run npm install (so Travis is usually happy), but it behaves strangely every time after that until you remove both artifacts and start from scratch.

Hopefully some of these clues help point to the source of the problem.

[Qriva]

I can also confirm this - using private repo aka git+ssh act very strange.
After runing npm update, sometimes it disappears for some reason although It appeared for a moment during instalation. I am not able to say what cause that but it act totaly random.

Is it possible that during typing keyphrase for ssh, the time used to do it can change anything?

[Rauttis]

Running into a similar issue with npm prune --production.

A git+ssh dependency that is defined in dependencies disappears after npm prune --production.

[Reeywhaar]

Same problem.

After installing dependency, like "npm install -D $depname", gulp appears missing. Gulp is defined if package.json as "github:gulpjs/gulp#4.0". "npm install" then fixes the problem.

[evenfrost]

Same here on npm v5.7.1 with cyrillic-to-translit-js being defined in package.json as

    "cyrillic-to-translit-js": "github:greybax/cyrillic-to-translit-js#master"

Seems the issue is still relevant. @zkat mind reopening?

[SudoPlz]

This issue happened to me when I requested from the Development server a bundle without the platform and dev flags.

I was requesting the bundle like http://192.168.192.30:8081/index.bundle

Once I changed it to http://192.168.192.30:8081/index.bundle?platform=ios&dev=true it worked well again.

[greysteil]

I've just seen this on a Dependabot pull request and dug into it. I think I have a fix, but my knowledge of the npm codebase / JS isn't good enough to implement it and (particularly) write tests for it.

First up, here are some reproduction steps:

# Get set up with a problematic update
git clone https://github.com/GilbertGobbels/GAwesomeBot.git
cd GAwesomeBot
git checkout indev-4.0.2

# Generate a bad package-lock.json
npm install null@0 --package-lock-only # null@0 could be anything here

git diff
# Gaze in horror at all the git-source dependencies having been removed (example: canvas-prebuilt)

The --package-lock-only modifier isn't necessary in the above (it just speeds things up), and the argument npm@5.8.0 could be anything - the problem is caused by passing an argument

Next up, I think I've isolated the problem in npm:
The cause of the above is here. The top part of the conditional is only run if an argument is passed, but the problem is that these lines are only run if an argument isn't passed. For whatever reason, git dependencies don't make it into the tree unless they're loaded.

Finally here is a PR that I'm pretty sure fixes this
I've put the above together into a PR at #20198, but JavaScript isn't my strongest language and it could really do with a test. There might also be performance optimisations available that I'm not thinking about (and were the reason for the conditional to start with, although this seems unlikely given the codepath).

I'd love help from a maintainer to get it tested and over the line.

[user3323]

The same problem!

What is the need for npm install something being so destructive ?

[user3323]

Well, I've found a reason of this "remove all of them" behavior of npm.

It's all about package.json file and dependencies / devDependencies fields.

In the process of installation of ANY package locally, npm will auto-scan versions of every package recorded in the package.json file, and will compare all of them with versions of packages that you have in node_modules folder. And, in case when you take someone's app (with package.json file, where recorded versions ^4.x.x) and leave your node_modules folder with newer (versions 5.x.x) packages - npm will delete all the packages, that didn't match one with other.

I will just remind, that version ^4.x.x does not match with 5.x.x, because this triangle caret ^ takes "upper effect" only on minor and patch numbers of version. (https://bytearcher.com/articles/semver-explained-why-theres-a-caret-in-my-package-json/)

So, I think it can be confusing for a lot of people. When, for example they know, that Angular 5 doesn't has "cataclysmic" changes in compare to Angular 4.
But "wise" npm thinks otherwise.

And if your package.json looks like this:

"dependencies": {
    "@angular/animations": "^4.0.0",
    "@angular/common": "^4.0.0",
    "@angular/compiler": "^4.0.0",
    "@angular/core": "^4.0.0",
    "@angular/forms": "^4.0.0",
    "@angular/http": "^4.0.0",
    "@angular/platform-browser": "^4.0.0",
    "@angular/platform-browser-dynamic": "^4.0.0"
}

... and all your node_modules/@angular/ packages have versions 5.x.x - all of them will be removed by npm. Furthermore - npm will remove all other packages, that are bound with /@angular/ packages. Thus, if node_modules folder has ~800 subfolders, when you replace package.json with lower versions, and simply install, for example, bootstrap (npm install bootstrap) - there will be ~150 subfolders in node_modules folder, already. Yep, minus 650.

It was like that couple days ago with npm v5.6.0 ... and yesterday they released version 6.0.0, and it still works in this not very smart order.

So, be careful, folks! 😎

[fnlctrl]

git+ssh dependencies are still removed in v6.1.0....