[BUG] Platform-specific optional dependencies not being included in package-lock.json when reinstalling with node_modules present

[JustinChristensen]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'

I'm working on a team that utilizes a mix of x64-based and m1-based macs, and has CI build processes that uses musl. We're seeing that npm is skipping platform-specific optional dependencies for packages such as @swc/core as a result of the package-lock.json file being generated without all of them included. In our case, this then causes linting to throw an exception, because one of our eslint plugins depends on @swc, which depends on having the platform specific @swc package also installed.

There seems to be at least two stages of cause to this. Firstly, when installing @swc/core from a clean slate working directory npm generates a package-lock.json with all of the optional dependencies for @swc/core listed:

[user@host:foo] $ npm install @swc/core
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-android-arm-eabi": {
    "node_modules/@swc/core-android-arm64": {
    "node_modules/@swc/core-darwin-arm64": {
    "node_modules/@swc/core-darwin-x64": {
    "node_modules/@swc/core-freebsd-x64": {
    "node_modules/@swc/core-linux-arm-gnueabihf": {
    "node_modules/@swc/core-linux-arm64-gnu": {
    "node_modules/@swc/core-linux-arm64-musl": {
    "node_modules/@swc/core-linux-x64-gnu": {
    "node_modules/@swc/core-linux-x64-musl": {
    "node_modules/@swc/core-win32-arm64-msvc": {
    "node_modules/@swc/core-win32-ia32-msvc": {
    "node_modules/@swc/core-win32-x64-msvc": {

And it only installs the platform specific package:

[user@host:foo] $ ls -l node_modules/@swc/
total 0
drwxr-xr-x  22 user  staff  704 Apr 29 15:39 core
drwxr-xr-x   6 user  staff  192 Apr 29 15:39 core-darwin-arm64

If I then remove my package-lock.json, leave my node_modules directory as-is, and then reinstall, I get:

[user@host:foo] $ rm -rf package-lock.json
[user@host:foo] $ npm install
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-arm64": {

That is, it then generates a package-lock.json with only the platform-specific dependency that was installed on this machine, and not with the other optional dependencies that should also be listed.

If you delete both node_modules AND package-lock.json, and then re-run npm install, it generates the correct lockfile with all of those optional dependencies listed.

The problem is that then, If the package-lock.json with the missing optional platform-specific dependencies gets checked into git and an x64 user pulls it down, or vice-versa, npm fails to detect that your platform's optional dependencies are missing in the lockfile and just silently skips installing the platform-specific dependency. For example, when I've got a package-lock.json that only contains the x64 @swc package because of the above problem (generated by my coworker on his x64 machine):

[user@host:foo] $ node
Welcome to Node.js v16.14.2.
Type ".help" for more information.
> process.arch
'arm64'
>
[user@host:foo] $ grep 'node_modules/@swc/core-*' package-lock.json
    "node_modules/@swc/core": {
    "node_modules/@swc/core-darwin-x64": {
[user@host:foo] $ ls
package-lock.json package.json

And I then install:

[user@host:foo] $ npm install
added 1 package in 341ms

1 package is looking for funding
  run `npm fund` for details
[user@host:foo] $ ls node_modules/@swc/
core

You can see that it fails to install the arm64 dependency or warn me in any way that the package-lock.json is missing my platform's dependency.

So yeah, two problems:

1. npm is generating an inconsistent package-lock.json when node_modules has your platform-specific dependency installed.
2. When installing from this inconsistent package-lock.json, npm fails to try to correct the problem by comparing the optional dependencies to what's listed upstream

Expected Behavior

1. npm should preserve the full set of platform-specific optional deps for a package like @swc when rebuilding package-lock.json from an existing node_modules tree
2. npm install should warn if the package-lock.json becomes inconsistent because of the first case

Steps To Reproduce

See above.

Environment

• npm: 8.8.0
• Node.js:
• OS Name: OSX
• System Model Name: Macbook Pro

[user@host:foo] $ npm -v
8.8.0
[user@host:foo] $ node -v
v16.14.2
[user@host:foo] $ uname -a
Darwin host.foo.com. 21.3.0 Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T8101 arm64

[user@host] $ npm config ls
; "user" config from /Users/user/.npmrc
; node bin location = /Users/user/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/user/Development/foo
; npm version = 8.8.0
; cwd = /Users/user/Development/foo
; HOME = /Users/user
; Run `npm config ls -l` to show all defaults.

[JustinChristensen]

@nlf

Sorry to ping you out of the blue, but this issue has been open for 11 days now without any movement. Is there anyone working on npm right now that might have the bandwidth to at least validate that this is indeed a problem as I've described it?

Just so that when someone does become available to do some development work they know that this is in the queue?

Please and thank you.

[JustinChristensen]

Bump

[RobbieClarken]

I'm also encountering this issue with a Next.js project:

• Deleting package-lock.json and running npm install on an M1 Mac results in a package-lock.json file that is no longer able to build the app on x86.
• This can be fixed by deleting package-lock.json and node_modules and re-running npm install.

Unfortunately developers often don't realise the package-lock.json file is broken because everything continues to run fine on their machine. It is only when the build runs in CI that we learn it is broken.

Here is a reproduction:

$ node --version
v16.13.0
$ npm --version
8.12.1
$ npx create-next-app@latest
What is your project named? … my-app
Creating a new Next.js app in /Users/robbie/demo/my-app.
$ cd my-app/
$ npm install

up to date, audited 223 packages in 480ms

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ git status
On branch main
nothing to commit, working tree clean
$ rm package-lock.json
$ npm install

up to date, audited 223 packages in 579ms

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ # ************ package-lock.json is now incompatible with x86 ************
$ git diff
diff --git a/package-lock.json b/package-lock.json
index cbbf946..a87c1e5 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -96,36 +96,6 @@
         "glob": "7.1.7"
       }
     },
-    "node_modules/@next/swc-android-arm-eabi": {
-      "version": "12.1.6",
-      "resolved": "https://registry.npmjs.org/@next/swc-android-arm-eabi/-/swc-android-arm-eabi-12.1.6.tgz",
-      "integrity": "sha512-BxBr3QAAAXWgk/K7EedvzxJr2dE014mghBSA9iOEAv0bMgF+MRq4PoASjuHi15M2zfowpcRG8XQhMFtxftCleQ==",
-      "cpu": [
-        "arm"
-      ],
-      "optional": true,
-      "os": [
-        "android"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
-    "node_modules/@next/swc-android-arm64": {
-      "version": "12.1.6",
-      "resolved": "https://registry.npmjs.org/@next/swc-android-arm64/-/swc-android-arm64-12.1.6.tgz",
-      "integrity": "sha512-EboEk3ROYY7U6WA2RrMt/cXXMokUTXXfnxe2+CU+DOahvbrO8QSWhlBl9I9ZbFzJx28AGB9Yo3oQHCvph/4Lew==",
-      "cpu": [
-        "arm64"
-      ],
-      "optional": true,
-      "os": [
-        "android"
-      ],
-      "engines": {
-        "node": ">= 10"
-      }
-    },
[...]
$ rm -r package-lock.json node_modules
$ npm install

added 222 packages, and audited 223 packages in 2s

68 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
$ # ************ package-lock.json is now ok again ************
$ git status
On branch main
nothing to commit, working tree clean

[pete55104]

I am also having this issue. I'm trying to run tests using jest with swc. The test runner is a linux image, but my dev machine is darwin. I can get it to work by either using --force to install the linux dependency, or I can install packages from inside the container... but github CI stands up the docker container in such a way that I can't easily install packages from in there, and that also prevents me from maintaining a cached node modules etc.

[johnculviner]

bump

[nikkhn]

bump - cannot get optional dependencies (namely @swc/core-linux-arm64-gnu) to install on my linux distro

[sgoodluck]

bump

[alcuadrado]

Confirming that this issue is still present. It's particularly important for projects using NAPI modules, as tons of them use platform-specific packages.

[AboldUSER]

Ran into this issue when creating a CI process for a repo where I use a Windows machine and the CI process is using Linux. My quick "fix" for now is to start the CI process by deleting the package-lock.json and running npm install instead of npm ci. I know this is not good practice, so looking forward to a real fix to come through.

[eliotSmithNYC]

bump

[douglassllc]

I am having a similar issue. My project uses @ffmpeg-installer/ffmpeg. While using npm v6 all optional dependencies (arch specific) are installed. After my upgrade to npm v8 the optional dependencies no longer install. Per the npm documentation I attempted using --include=optional, but this did not resolve the issue.

What has changed between v6 and v8 and is there an npm config option that will have v8 work similar to v6 when it comes to optional dependencies?

[MrAlucardDante]

@arudenkoofficial that depends on how you setup your package.json. I make it a habit to pin important dependencies to specific versions, so running npm install should generate the exact same versions in the lock file.

For updating dependencies, i use ncu. https://www.npmjs.com/package/npm-check-updates

We do something similar too. We pin every package, since not everyone use semver. And we use renovate to automatically create PRs when an update is available.

[ilhamksyuriadi-csgi]

bump, this still happen to me

[studentrk]

The bug is also still happening for me. The suggested workaround to remove both package-lock.json and node_modules directory created another error(when starting my ionic capacitor application I got the error: [Error] SyntaxError: Unexpected token '{' promiseReactionJob ionic ) --> this error probably occured, because installing the node_modules is currently for me only possible with this solution nodejs/node-gyp#2972 (comment)).

Current Workaround for me:
So to avoid creating another error I installed "@rollup/rollup-darwin-arm64" directly.

[ruizdiazever]

REWRITE JS IN RUST

[glad2os]

Still exists Oct 3, 2024

turbo-trading-react-1  | 
turbo-trading-react-1  | > turbo-trading-react@0.0.0 dev
turbo-trading-react-1  | > vite
turbo-trading-react-1  | 
turbo-trading-react-1  | /app/node_modules/rollup/dist/native.js:59
turbo-trading-react-1  |                throw new Error(
turbo-trading-react-1  |                      ^
turbo-trading-react-1  | 
turbo-trading-react-1  | Error: Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.
turbo-trading-react-1  |     at requireWithFriendlyError (/app/node_modules/rollup/dist/native.js:59:9)
turbo-trading-react-1  |     at Object.<anonymous> (/app/node_modules/rollup/dist/native.js:68:76)
turbo-trading-react-1  |     at Module._compile (node:internal/modules/cjs/loader:1546:14)
turbo-trading-react-1  |     at Module._extensions..js (node:internal/modules/cjs/loader:1691:10)
turbo-trading-react-1  |     at Module.load (node:internal/modules/cjs/loader:1317:32)
turbo-trading-react-1  |     at Module._load (node:internal/modules/cjs/loader:1127:12)
turbo-trading-react-1  |     at TracingChannel.traceSync (node:diagnostics_channel:315:14)
turbo-trading-react-1  |     at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
turbo-trading-react-1  |     at cjsLoader (node:internal/modules/esm/translators:329:5)
turbo-trading-react-1  |     at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:260:7) {
turbo-trading-react-1  |   [cause]: Error: Cannot find module '@rollup/rollup-linux-x64-gnu'
turbo-trading-react-1  |   Require stack:
turbo-trading-react-1  |   - /app/node_modules/rollup/dist/native.js
turbo-trading-react-1  |       at Module._resolveFilename (node:internal/modules/cjs/loader:1248:15)
turbo-trading-react-1  |       at Module._load (node:internal/modules/cjs/loader:1074:27)
turbo-trading-react-1  |       at TracingChannel.traceSync (node:diagnostics_channel:315:14)
turbo-trading-react-1  |       at wrapModuleLoad (node:internal/modules/cjs/loader:217:24)
turbo-trading-react-1  |       at Module.require (node:internal/modules/cjs/loader:1339:12)
turbo-trading-react-1  |       at require (node:internal/modules/helpers:135:16)
turbo-trading-react-1  |       at requireWithFriendlyError (/app/node_modules/rollup/dist/native.js:41:10)
turbo-trading-react-1  |       at Object.<anonymous> (/app/node_modules/rollup/dist/native.js:68:76)
turbo-trading-react-1  |       at Module._compile (node:internal/modules/cjs/loader:1546:14)
turbo-trading-react-1  |       at Module._extensions..js (node:internal/modules/cjs/loader:1691:10) {
turbo-trading-react-1  |     code: 'MODULE_NOT_FOUND',
turbo-trading-react-1  |     requireStack: [ '/app/node_modules/rollup/dist/native.js' ]
turbo-trading-react-1  |   }
turbo-trading-react-1  | }
turbo-trading-react-1  | 
turbo-trading-react-1  | Node.js v22.8.0
turbo-trading-react-1 exited with code 1
glad2os@glad2oss-MacBook-Pro turbo-trading % 

Dockerfile

FROM node:22.8.0

WORKDIR /app

COPY . .

RUN npm install

ENTRYPOINT [ "npm" ]
CMD [ "run", "dev" ]

docker-compose.yml

  turbo-trading-react:
    platform: linux/amd64
    build:
      context: turbo-trading-react
      dockerfile: Dockerfile.dev
    volumes:
      - ./turbo-trading-react:/app

package.json

{
  "name": "turbo-trading-react",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc -b && vite build",
    "lint": "eslint .",
    "preview": "vite preview"
  },
  "dependencies": {
    "react": "^18.3.1",
    "react-dom": "^18.3.1"
  },
  "devDependencies": {
    "@eslint/js": "^9.9.0",
    "@types/react": "^18.3.3",
    "@types/react-dom": "^18.3.0",
    "@vitejs/plugin-react": "^4.3.1",
    "eslint": "^9.9.0",
    "eslint-plugin-react-hooks": "^5.1.0-rc.0",
    "eslint-plugin-react-refresh": "^0.4.9",
    "globals": "^15.9.0",
    "typescript": "^5.5.3",
    "typescript-eslint": "^8.0.1",
    "vite": "^5.4.1"
  }
}

[enietos]

Bump - experiencing this as well:

[builder] Paketo Buildpack for Node Run Script 1.0.31
[builder]   Executing build process
[builder]     Running 'npm run build'
[builder]       
[builder]       > nodebook@1.0.0 build
[builder]       > vite build
[builder]       
[builder]       /layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js:59
[builder]                       throw new Error(
[builder]                             ^
[builder]       
[builder]       Error: Cannot find module @rollup/rollup-linux-x64-gnu. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). Please try `npm i` again after removing both package-lock.json and node_modules directory.
[builder]           at requireWithFriendlyError (/layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js:59:9)
[builder]           at Object.<anonymous> (/layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js:68:76)
[builder]           ... 3 lines matching cause stack trace ...
[builder]           at Module._load (node:internal/modules/cjs/loader:1024:12)
[builder]           at cjsLoader (node:internal/modules/esm/translators:348:17)
[builder]           at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:297:7)
[builder]           at ModuleJob.run (node:internal/modules/esm/module_job:222:25)
[builder]           at async ModuleLoader.import (node:internal/modules/esm/loader:316:24) {
[builder]         [cause]: Error: Cannot find module '@rollup/rollup-linux-x64-gnu'
[builder]         Require stack:
[builder]         - /layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js
[builder]             at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
[builder]             at Module._load (node:internal/modules/cjs/loader:986:27)
[builder]             at Module.require (node:internal/modules/cjs/loader:1233:19)
[builder]             at require (node:internal/modules/helpers:179:18)
[builder]             at requireWithFriendlyError (/layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js:41:10)
[builder]             at Object.<anonymous> (/layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js:68:76)
[builder]             at Module._compile (node:internal/modules/cjs/loader:1358:14)
[builder]             at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
[builder]             at Module.load (node:internal/modules/cjs/loader:1208:32)
[builder]             at Module._load (node:internal/modules/cjs/loader:1024:12) {
[builder]           code: 'MODULE_NOT_FOUND',
[builder]           requireStack: [
[builder]             '/layers/paketo-buildpacks_npm-install/build-modules/node_modules/rollup/dist/native.js'
[builder]           ]
[builder]         }
[builder]       }
[builder]       
[builder]       Node.js v20.15.0
[builder] exit status 1
[builder] ERROR: failed to build: exit status 1

Tried the above 'delete package_lock.json and node_modules" but encountered again. Would appreciate a fix, very confusing as a novice web developer getting into vite/npm.

[uweDuesing]

sigh...

Is there actually any news with reagrds to this quite fundamental bug? Have I missed anything?

[maidl]

For everyone encountering this in their CI/CD-Run (and came here via the link in the error message, see this comment):
Issuing a npm i @rollup/rollup-linux-x64-gnu before npm ci seems to do the trick, for now. Of course, this might be considered as a workaround, but could be the cleanest way so far, since it does not alter the project's dependencies and make the run/build work (again).

---

EDIT: Of course installing @rollup/rollup-linux-x64-gnu only helps if the platform you are running the CI/CD on is linux based. For other platforms, pay attention to the mentioned missing module in the error message.

[azerum]

Sorry for potentially unrelated comment, but can somebody explain what is the reasoning behind deleting package-lock.json? I would expect that for predictable installs, one should always have package-lock.json present. What is the use case of deleting it?

[paulrutter]

The bug occurs when npm install is called on modules that contain native code in optional dependencies. It then only adds the platform specific versions of the platform running the command to the lock file. When the lock file is absent, it generates it properly for all available platforms, so it works cross platform.

[uweDuesing]

This is getting worse.
So far every time I wanted build my Angular library I needed to delete the node_modules folder, package.json lock files and re-install everything, but now nothing works anymore.It shows this error

root@8841072121b8:/container# npm i
npm error code EBADPLATFORM
npm error notsup Unsupported platform for @rollup/rollup-darwin-arm64@4.24.0: wanted {"os":"darwin","cpu":"arm64"} (current: {"os":"linux","cpu":"arm64"})
npm error notsup Valid os: darwin
npm error notsup Actual os: linux
npm error notsup Valid cpu: arm64
npm error notsup Actual cpu: arm64