Missing dependencies after running `npm install` a second time

[schmod]

I'm opening this issue because:

• npm is crashing.
• npm is producing an incorrect install.
• npm is doing something I don't understand.
• Other (see below for feature requests):

What's going wrong?

When there is an existing lockfile, npm5 does not install all necessary packages. (This issue specifically affects any project that depends on foundry-kue-scheduler)

The initial invocation of npm install installs the correct set of dependencies, and creates a lockfile as I would expect.

When I delete node_modules, and run npm install again (against the same lockfile), I get a different set of packages (several transitive dependencies are missing), and npm alters the contents package-lock.json.

How can the CLI team reproduce the problem?

This seems to be a minimal test-case that shows how npm install is broken for any package that depends on foundry-kue-scheduler:

# scaffold a minimal project
$ mkdir npm5-test
$ cd npm5-test
$ npm init -y

# install/save my first/only dependency
$ npm i foundry-kue-scheduler

npm WARN deprecated redlock@2.1.1: possible critical bug, see https://github.com/mike-marcacci/node-redlock/issues/31
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN npm5-test@1.0.0 No description
npm WARN npm5-test@1.0.0 No repository field.

added 203 packages in 11.601s

# npm installed my packages and creates a lockfile
$ ls -lh
total 88
drwxr-xr-x  194 andrew  staff   6.4K May 31 15:09 node_modules
-rw-r--r--    1 andrew  staff    38K May 31 15:09 package-lock.json
-rw-r--r--    1 andrew  staff   286B May 31 15:09 package.json

# one of my transitive dependencies is present
$ ls -lh node_modules/kue/package.json
-rw-r--r--  1 andrew  staff   2.0K May 31 15:09 node_modules/kue/package.json

# At this point, everything is OK.  
# NPM has installed the set of packages that I expected.
# I can run `npm install` again, and it correctly reports that there's nothing to do.

# now, let's remove node_modules and reinstall (as though I'm installing
# packages for the first time in a repo that has package-lock.json checked-in,
# and node_modules is listed in .gitignore )
$ rm -rf node_modules/
$ npm i

npm WARN npm5-test@1.0.0 No description
npm WARN npm5-test@1.0.0 No repository field.

added 192 packages in 4.73s

# note that fewer packages were installed, and npm MODIFIED the extant lockfile
$ ls -lh
total 88
drwxr-xr-x  185 andrew  staff   6.1K May 31 15:11 node_modules
-rw-r--r--    1 andrew  staff    36K May 31 15:11 package-lock.json
-rw-r--r--    1 andrew  staff   286B May 31 15:11 package.json

# also, my transitive dependency is gone.
$ ls node_modules/kue/package.json
ls: node_modules/kue/package.json: No such file or directory

# subsequent invocations of npm install make things even worse
$ npm i

npm WARN npm5-test@1.0.0 No description
npm WARN npm5-test@1.0.0 No repository field.

removed 190 packages in 3.026s

# ahhhhhh!
$ ls -lh
total 16
drwxr-xr-x  5 andrew  staff   170B May 31 15:15 node_modules
-rw-r--r--  1 andrew  staff   503B May 31 15:15 package-lock.json
-rw-r--r--  1 andrew  staff   286B May 31 15:15 package.json

Here's a gist that shows the package structure, as well as the changes to package-lock.json between the first and second invocation of npm install.

supporting information:

• npm -v prints: 5.0.0
• node -v prints: v6.10.3
• npm config get registry prints:
• Windows, OS X/macOS, or Linux?: macOS (also verified inside Docker w/ Linux)
• Network issues:
  • Geographic location where npm was run:
  • I use a proxy to connect to the npm registry.
  • I use a proxy to connect to the web.
  • I use a proxy when downloading Git repos.
  • I access the npm registry via a VPN
  • I don't use a proxy, but have limited or unreliable internet access.
• Container:
  • I develop using Vagrant on Windows.
  • I develop using Vagrant on OS X or Linux.
  • I develop / deploy using Docker.
  • I deploy to a PaaS (Triton, Heroku).

[schmod]

Some quick discussion:

---

The broken transitive dependency (kue) referenced in the reproduction steps above was resolved through a GitHub dependency (foundry-kue-scheduler depends on a GitHub-hosted fork of kue-scheduler). I haven't been able to identify if that's the cause of this regression, but it seems like a potential culprit.

---

The fact that the "bare" npm install command could modify an existing package-lock.json was surprising to me.

I would not be surprised if npm prune, npm update, or npm install <package> modify the lockfile, but definitely was not expecting npm install to make changes on its own.

This seems like a bug, but there doesn't seem to be much documentation about the contracts that NPM makes with regards to lockfiles, or how they're meant to be used in a typical workflow. For all I know, this could be an expected behavior. More documentation would be helpful.

My $0.02 is that the "bare" npm install command should strive to be idempotent, and should not make any changes (outside of node_modules or caches) when a lockfile is present.

[schmod]

Dropping a link to #16837 here, because it seems similar.

[kelsin]

Seeing these same results. We committed our package-lock.json after a brand new npm i and then on Jenkins we (running same node version, same npm version) see missing module errors. I also see the same results where the package-lock.json gets weird and smaller after every npm i run.

By weird, I'm seeing an issue where the repository URL ends up in the "version" field of package-lock.json.

[kelsin]

And to confirm I also see this only when locally removing the node_modules folder (which is also the environment in CI)

[ericeslinger]

I am seeing something very similar when updating a single package. For example, once everything is working fine, I can do this:

➜  models (master) ✔ npm i
up to date in 1.561s
➜  models (master) ✔ rm -rf node_modules/plump 
➜  models (master) ✔ npm i
added 1 package and removed 4 packages in 1.792s
➜  models (master) ✗ 

note that I just deleted one package that I directly depend on, and then running npm i a second time correctly installs the deleted package, but seems to delete the packages that were depended on by that package I manually deleted. So now (in this case), plump is properly installed, but rxjs and merge-options are both missing (both are in the dependencies entry for plump). I'm not entirely sure where the 4 removed packages total comes from - I assume they are symbol-observable and is-plain-obj, which are the sole declared dependencies of rxjs and merge-options respectively.

What is weirder is that if I do npm i --no-save rxjs, I get back "added 2 packages", but they don't actually add (npm ls rxjs shows it's not there, and just doing an ls -laR node_modules | grep rxjs confirms this).

If I do something silly like go into a working directory that's all properly installed and do npm i --no-save rxjs (or some other package that is a declared dependency of the thing I'm working on but was already installed), it goes nuts - in this case it removed 123 packages and updated 211 packages, when in fact it should have done nothing (rxjs was already installed and up to date).

In all cases, this is only repaired by deleting both package-lock.json and node_modules and doing a fresh npm i.

[mbland]

I was going to file my own bug, but I think this one covers my experience. I've managed to create a minimal experiment and demo repo that reproduces this behavior; the README has all the details:

https://github.com/mbland/npm-v5-package-lock-bug-demo

Hopefully it's helpful here. The upshot: the problem appears related to devDependency packages whose own dependency specs are set tolatest (at least, that's how it appeared in my project with a devDependency on live-server). It also only appears to manifest when the original node_modules directory is removed and recreated.