[BUG] dev dependency binaries not linked after upgrading package-lock.json from v1 to v2 using npm install --production and then running npm install

[KThompso]

Current Behavior:

Upgrading package-lock.json from lockfileVersion: 1 to lockfileVersion: 2 using npm install --production causes future calls to npm install to not link the binaries of the dev dependencies under node_modules/.bin. This seems like it may be caused by the dev dependencies in package-lock.json not being updated.

Expected Behavior:

Subsequent calls to npm install should correctly link the dev dependency binaries in node_modules/.bin if needed.

Steps To Reproduce:

1. Using npm version 6
   1.1. Create a project (npm init --yes).
   1.2. Install a dev dependency that has a binary, also generating a package-lock.json file in lockfileVersion 1 (npm install --save-dev typescript).
   1.3. Delete node_modules (rm -rf node_modules).
2. Using npm version 7
   2.1. Install only production dependencies, also causing package-lock.json to be upgraded to lockfileVersion 2 (npm install --production).
   2.2 Install all dependencies (npm install)
   2.3 Try executing one of the binaries that should have been linked (npx -c 'tsc -v'), it produces error /bin/sh: 1: tsc: not found because the binary has not been linked via node_modules/.bin/.

Docker Reproduction

This Dockerfile can be used to reproduce an environment where the corrupted package-lock.json has been created. You can build and run it using docker build -t node-issue . && docker run node-issue

FROM node:14 as npm6
WORKDIR /app
# Create a node project using npm 6 and install a dev dependency
# that contains a binary.
RUN npm init --yes && \
    npm install --save-dev typescript

FROM node:15 as npm7
COPY --from=npm6 /app/package*.json /app/
WORKDIR /app
# Install production dependencies, then all dependencies. This should
# link the binaries for typescript in (e.g. tsc) under node_modules/.bin.
RUN npm install -g npm@7.10.0 && \
    npm install --production && \
    npm install

# Causes error, tsc not found.
CMD ["npx", "-c", "tsc --version"]

Example output of running this container:

$ docker build -q -t node-issue . && docker run node-issue
sha256:f0e75b67500f84e122cf73351cf63048bebbefe6f60d0436b87dc0b9db7a1686
sh: 1: tsc: not found

More info:

# start container
$ docker run -it node-issue bash

# Check node_modules, there's no .bin directory
root@5c272cfb33dd:/app# ls -al node_modules/
total 16
drwxr-xr-x 3 root root 4096 Apr 22 19:22 .
drwxr-xr-x 1 root root 4096 Apr 22 19:22 ..
-rw-r--r-- 1 root root  390 Apr 22 19:22 .package-lock.json
drwxr-xr-x 5 root root 4096 Apr 22 19:22 typescript

# Try running install again
root@5c272cfb33dd:/app# npm install

up to date, audited 2 packages in 428ms

found 0 vulnerabilities

# Nothing has changed, still no .bin directory
root@5c272cfb33dd:/app# ls -al node_modules/
total 24
drwxr-xr-x 1 root root 4096 Apr 22 19:22 .
drwxr-xr-x 1 root root 4096 Apr 22 19:22 ..
-rw-r--r-- 1 root root  390 Apr 22 19:27 .package-lock.json
drwxr-xr-x 5 root root 4096 Apr 22 19:22 typescript

# Remove the package-lock.json and node_modules and re-install
root@5c272cfb33dd:/app# rm -rf node_modules/ package-lock.json 
root@5c272cfb33dd:/app# npm install

added 1 package, and audited 2 packages in 1s

found 0 vulnerabilities

# Finally, .bin is created and typescript binaries are linked.
root@5c272cfb33dd:/app# ls -al node_modules/
total 24
drwxr-xr-x 4 root root 4096 Apr 22 19:28 .
drwxr-xr-x 1 root root 4096 Apr 22 19:28 ..
drwxr-xr-x 2 root root 4096 Apr 22 19:28 .bin
-rw-r--r-- 1 root root  529 Apr 22 19:28 .package-lock.json
drwxr-xr-x 5 root root 4096 Apr 22 19:28 typescript
root@5c272cfb33dd:/app# ls node_modules/.bin
tsc  tsserver

Example files

Here is an example of a corrupted package-lock.json and its corresponding package.json:

package-lock.json

{
  "name": "app",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "version": "1.0.0",
      "license": "ISC",
      "devDependencies": {
        "typescript": "^4.2.4"
      }
    },
    "node_modules/typescript": {
      "version": "4.2.4",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.2.4.tgz",
      "integrity": "sha512-V+evlYHZnQkaz8TRBuxTA92yZBPotr5H+WhQ7bD3hZUndx5tGOa1fuCgeSjxAzM1RiN5IzvadIXTVefuuwZCRg==",
      "dev": true
    }
  },
  "dependencies": {
    "typescript": {
      "version": "4.2.4",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.2.4.tgz",
      "integrity": "sha512-V+evlYHZnQkaz8TRBuxTA92yZBPotr5H+WhQ7bD3hZUndx5tGOa1fuCgeSjxAzM1RiN5IzvadIXTVefuuwZCRg==",
      "dev": true
    }
  }
}

package.json

{
  "name": "app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "typescript": "^4.2.4"
  }
}

Workaround

If you use npm ci or delete package-lock.json and reinstall it should update the binary links correctly.

Environment:

Host

• OS: Ubuntu 20.04.2 LTS
• Docker: 20.10.6, build 370c289

Docker

NPM 6

• OS: Debian 9 stretch
• Node: 14.16.1
• npm: 6.14.12

NPM 7

• OS: Debian 9 stretch
• Node: 15.14.0
• npm: 7.10.0

[darcyclarke]

This should be resolved in the next version of arborist which should then get pulled in to the next npm release (~next week); ref. npm/arborist#287