[BUG] npm ci still does not work with overrides correctly #4942

jchiem · 2022-05-25T19:33:54Z

Is there an existing issue for this?

I have searched the existing issues

This issue is related to:
#4684, #4732, #4732

This issue exists in the latest npm version

I am using the latest npm

Current Behavior

The release of npm@8.7.0 has made a fix for the aforementioned issues, but the same error occurs in my example repository on npm@8.10.0, and for any version above 8.6.0 for that matter, I tested #4732 with 8.7.0 and the error presented in that specific issue is resolved with any newer version of npm.

The bug is that if npm ci is used with certain dependencies and override combination the following error occurs:

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

This happens with any version of npm above 8.5.5, if 8.5.5 is used the error does not occur.

Furthermore, if you follow the instruction provided in the error to run npm install the security vulnerabilities that were originally resolved with the "overrides" are restored and the package-lock is updated. If you run npm ci after that the overrides are still not taken into consideration, so the overrides have been ignored.

I've set up a repository where the behavior is described in more detail which is listed under Steps To Reproduce

Expected Behavior

With npm ci I expect that the latest version of npm is able to use npm ci with the defined "overrides" without errors.

With npm install or npm install --package-lock-only I expect that the "overrides" configured are still respected and the package-lock is not updated and ignoring the overrides.

Steps To Reproduce

To test this do the following:

Have npm 8.6.0 or greater installed (bug occurs for any version above 8.5.5)

Clone the repository git clone git@github.com:jchiem/npm-overrides-bug.git
Run npm ci
Run npm install or npm install --package-lock-only

For Step 2 you should get an error like:

For Step 3, it results in no errors, but the package-lock is updated and the node modules installed were done without overrides since security vulnerabilities that should have been resolved using the "overrides" in package.json are no longer 0.

If you run npm ci at this point, the installed packages retain the vulnerabilities from Step 3's npm install and the `"overrides" are ignored.

Environment

npm: version above 8.5.5 (for example, 8.6.0 or 8.10.0)
Node.js: 16.14.2
OS Name: mac OS 12.3.1 (Monterey)
System Model Name: MacBook Pro (16-inch, 2019)
npm config:

; node bin location = /Users/joakim.chiem/.nvm/versions/node/v16.14.2/bin/node
; node version = v16.14.2
; npm local prefix = /Users/joakim.chiem/git/npm-overrides-bug
; npm version = 8.10.0
; cwd = /Users/joakim.chiem/git/npm-overrides-bug
; HOME = /Users/joakim.chiem
; Run `npm config ls -l` to show all defaults.

The text was updated successfully, but these errors were encountered:

darmentrout · 2022-06-03T16:10:15Z

I just upgraded npm from 8.5.0 to 8.12.1 and I'm getting the same errors. Removing package-lock.json, deleting node_modules, and rerunning npm install do not help.

$ npm ci
npm ERR! code EUSAGE
npm ERR!
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR!
npm ERR! Invalid: lock file's type-fest@0.20.2 does not satisfy type-fest@0.13.1
npm ERR! Missing: type-fest@0.20.2 from lock file
npm ERR! Missing: type-fest@0.20.2 from lock file
npm ERR! Missing: type-fest@0.20.2 from lock file
npm ERR! Missing: type-fest@0.20.2 from lock file
npm ERR!
npm ERR! Clean install a project
npm ERR!
npm ERR! Usage:
npm ERR! npm ci
npm ERR!
npm ERR! Options:
npm ERR! [--no-audit] [--foreground-scripts] [--ignore-scripts]
npm ERR! [--script-shell <script-shell>]
npm ERR!
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR!
npm ERR! Run "npm help ci" for more info

darmentrout · 2022-06-03T16:48:30Z

Running npm install twice makes the npm ci error go away.
#4859 (comment)

jchiem · 2022-06-03T19:51:32Z

Oh interesting, perhaps this will be solved with #4859 then? 🤔 for now I've just kept using npm 8.5.5 and the package-lock generated with that version, it works with my specific dependencies at least, I'll try out if npm install twice as mentioned in the other issue works for my project as well.

jchiem · 2022-06-07T07:24:33Z

@darmentrout

could you elaborate how npm install resolves the issue for you? For me, doing
rm -rf package-lock.json
rm -rf node_modules
npm install
npm install

Still results in vulnerabilities being installed with the 2nd npm install command, which to me indicates that the npm overrides still do not work correctly even with that method for me at least, using npm 8.12.1

MRobertEvers · 2022-06-07T12:12:17Z

npm install does not resolve the issue for us. npm >= 8.6 fails with the same errors.

krasingneves · 2022-07-01T20:33:16Z

I've experienced this with npm v8.11.0 as well

tvsbrent · 2022-07-04T22:04:03Z

The issue as I've encountered is when I've tried to pass in a range that I'd like to override. For example:

"overrides": {
  "got@<11.8.5": "11.8.5",
  "got@>=12.0.0 <12.1.0": "12.1.0"
}

The above will fail. In looking at the generated package-lock.json, it looks like the dependency that was pulling in the overridden version was not updated. For example:

"package-json": {
  "version": "6.5.0",
  "resolved": "<snip>",
  "integrity": "<snip>",
  "dev": true,
  "requires": {
    "got": "^9.6.0",
    "registry-auth-token": "^4.0.0",
    "registry-url": "^5.0.0",
    "semver": "^6.2.0"
  }
}

However, if I do this in the package.json, it does work:

"overrides": {
  "package-json@6.5.0": {
    "got": "11.8.5"
  }
}

The entry in the package-lock.json is the following in that case:

"package-json": {
  "version": "6.5.0",
  "resolved": "<snip>",
  "integrity": "<snip>",
  "dev": true,
  "requires": {
    "got": "11.8.5",
    "registry-auth-token": "^4.0.0",
    "registry-url": "^5.0.0",
    "semver": "^6.2.0"
  }
},

krasingneves · 2022-07-04T22:31:09Z

I was using

"overrides": {
  "package-to-override": {
    "got": "11.8.5"
  }
}

So, specifying the package but not the version. So I may give this a try

Joinatic · 2022-08-03T06:55:02Z

The second npm install seems to ignore the overrides, so no solution for me

bellis-ai · 2022-08-09T13:53:14Z

Do the overrides actually work?

Doing an npm install on this package.json

{
  "dependencies": {
    "better-docs": "^2.7.2"
  },
  "overrides": {
    "lodash": "^4.17.21"
  }
}

Still ends up with an install of lodash@4.17.4

DzianisMakeichyk · 2022-08-11T12:22:52Z

Double npm i still fix the issue. npm v8.6.0

jchiem · 2022-08-12T14:07:19Z

Double npm i still fix the issue. npm v8.6.0

Double npm i with 8.6.0 does not resolve the issue, it just introduces a new package-lock in which overrides are no longer being followed, please check the package-lock, please provide example of where the package-lock actually follows the override conditions after doing npm i twice if this is actually working for you.

jchiem · 2022-08-12T15:18:32Z

The problem persists with npm 8.17.0

With npm 8.5.5:

(1 vulnerability has been introduced since this issue for the repo mentioned in the description of this issue: https://github.com/jchiem/npm-overrides-bug)

npm/cli#4942

* Make a terrible rxdb implementation * Fix issues with updating patients * Style fixes * Small fixes * Lay groundwork for GraphQL sync * Add push replication * Fix bug with unknown consult entity * Change uid to id, connect RxDB pull * Fix issues with push-replication, DOB field * Fix a few errors * Fix serialization issues * Never lose a patient again * Fix deletedAt * Fix enums... again * !!Fix vulnerabilities (includes run of npm update) * Fix npm CI(?) * roll back node version for CD npm/cli#4942 * whoops wrong version * Fix frontend tests * Refactor for readability * Fix issues * Automatically select first tab where available * Add required fields * Ugly hack to reduce traffic * match project file structure * fix nitpick * match project file structure * Fix error with patient ordering in feed * move usePatients hook Co-authored-by: Jackson Chadfield <jcha686@aucklanduni.ac.nz>

jchiem · 2022-09-23T11:03:33Z

The problem persists in npm version 8.19.2

(cf https://stackoverflow.com/questions/69984660/npm-ci-can-only-install-packages-with-an-existing-package-lock-json-or-npm-shrin/71845794#71845794 and npm/cli#4942)

jchiem · 2022-11-07T16:56:18Z

I've ran some more tests regarding this and found that it seems to be specifically when < is used for the overrides for the condition.

Writing it without the < seems to allow the overrides to be interpreted correctly even with the latest version of npm, I think if it shouldn't work for overrides declared using < sign for the version, then that should be added to the documentation and this issue can be closed, if it should work however, this bug is still relevant for this specific type of override.

grigorii-merck · 2023-01-04T12:57:58Z

Facing similar issue, specifying version without < does not seem to make any difference.

pawan-bobbili-ntnx · 2023-02-06T07:24:18Z

is this issue fixed in any latest npm versions ?

20BBrown14 · 2023-02-07T21:52:52Z

is this issue fixed in any latest npm versions ?

I tested npm@8.19.2 and npm@8.19.3 today and am still seeing this issue that i'm pretty sure is related to the issues brought up here.

10hendersonm · 2023-05-10T14:10:25Z

I believe my team is seeing this in npm@9.5.1

m-nathani · 2023-06-19T07:34:25Z

i had the same problem, was overrding @babel/core from lib, after install my lib to my app, it wasnt overriding the @babel/core,

untill i had to remove node_modules and package-lock.json , and run npm i .

on which when i did npm ls @babel/core it showed the correct version.

petruki · 2023-06-25T01:32:08Z

This works for me (npm 9.5.1).
I was fixing a vulnerability introduced by a downstream dependency in angular/localize and I could utilize npm ci with overrides.

Here is how I did:

Remove your overrides from the package.json and run a clean npm install
Move your overrides back to the package.json and run npm update
If you use shrinkwrap, run this after updating

Do not use "$" dollar sign as a version reference from your dependencies or devDependencies. Currently, npm ci cannot interpret when verifying if package-json and lock files are synced.

npm/cli#4942

gaving · 2023-08-10T13:40:20Z

Switching from legacy yarn back to npm on a large project and getting burned by this one, not particularly sure how to proceed. npm ci still moans about package-lock.json not being in sync even with the previous post solution.

victorgois · 2023-09-12T13:25:06Z

I am dealing with the same issue. My application stopped running on docker in the google cloud build. Using node@18.17.1 and npm@9.6.7. However, when I add a npm i before the npm ci in the dockerfile, it builds. I do not use any override config in my package.

netishix · 2024-01-02T22:19:45Z

Still experiencing this issue with npm@9.8.1. Downgrading to npm@8.5.5 fixed the problem for me.

Sweetog · 2024-01-09T01:11:34Z

Still experiencing this issue with npm@9.5.1, npm@8.19.4, always resolved by downgrading to npm@8.5.5. PLEASE FIX THIS, the issue is absolutely problematic and should not be happening.

I believe it has to be changes in v8.6.0 release that have caused this regression:
https://github.com/npm/cli/releases/tag/v8.6.0

edwincoronado · 2024-01-09T01:14:44Z

Experiencing this issue too.

Momotoculteur · 2024-03-08T13:14:34Z

I had this issue with nodejs 20.11.1 and npm 10.2.4. Had to double npm install to work...

alexgomezlf · 2024-03-08T15:11:43Z

We are experiencing the same issue. Using npm install instead of npm ci worked for us, but it is not ideal and the package-lock.json changes every time we run npm install. Our only other option was to downgrade npm to npm@8.5.5

Sweetog · 2024-03-08T22:40:45Z

@petruki What is the purpose of your reply? I ask, because I don't want anyone to be confused by your comment and think there is any type of work around for any OP here because there is not. This issue is actually ridiculous, it is a fundamental failure. When a user declares overrides and runs npm ci, any version of npm > 8.5.5 borks out.

petruki · 2024-03-09T21:37:38Z

@Sweetog , oh, sorry my friend, I didn't mean to expose any hack for this issue. I just wanted to share what worked for me and perhaps help someone who cannot wait for a fix. I removed the reply anyway as you may be right. Thanks for the advice. 😉

yaronguez · 2024-04-05T20:38:31Z

It looks like this issue is over 2 years old. Are there any plans to fix it?

jchiem added Bug thing that needs fixing Needs Triage needs review for next steps Release 8.x work is associated with a specific npm 8 release labels May 25, 2022

github-matthieu-wipliez mentioned this issue May 30, 2022

[BUG] Overrides not working for some transitive dependencies, cause ERR ELSPROBLEMS in npm ls #4889

Closed

2 tasks

acicovic mentioned this issue Jun 13, 2022

GitHub Workflows using npm ci fail Parsely/wp-parsely#924

Closed

darcyclarke added the config:overrides Issues dealing with the overrides feature label Jul 11, 2022

ghost pushed a commit to University-of-Auckland-SOFTENG761/supervision that referenced this issue Sep 13, 2022

roll back node version for CD

084e0c2

npm/cli#4942

julienrbrt mentioned this issue Oct 10, 2022

ci: fix build docs cosmos/cosmos-sdk#13490

Merged

19 tasks

jcscottiii mentioned this issue Nov 8, 2022

Replace or fix lit static analysis tool GoogleChrome/chromium-dashboard#2449

Open

frankhinek mentioned this issue Jun 23, 2023

Fix for CVE-2022-25883 TBD54566975/web5-js#136

Merged

keskiju added a commit to TaitoUnited/full-stack-template that referenced this issue Jul 4, 2023

fix(client): remove npm overrides because of 'npm ci' incompatibility

b495c5a

npm/cli#4942

shazron mentioned this issue Jul 13, 2023

workaround: @oclif/core in @oclif/plugins-plugin@^2.x is not compatible with node-14 adobe/aio-cli#468

Closed

10 tasks

ericjperry mentioned this issue Jul 21, 2023

CVE-2023-37466 - VM2 - Sandbox Escape Vulnerability snowflakedb/snowflake-connector-nodejs#571

Closed

mindctrl mentioned this issue Nov 8, 2023

Prepare 0.25.0 release wpengine/atlas-content-modeler#677

Merged

1 task

chrisl8 mentioned this issue Mar 8, 2024

generateConfig[fn] is not a function for antd package ukrbublik/react-awesome-query-builder#1033

Closed

silverwind mentioned this issue Apr 24, 2024

Reproducible builds and strict dependency verification go-gitea/gitea#30685

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] npm ci still does not work with overrides correctly #4942

[BUG] npm ci still does not work with overrides correctly #4942

jchiem commented May 25, 2022 •

edited

darmentrout commented Jun 3, 2022

darmentrout commented Jun 3, 2022

jchiem commented Jun 3, 2022

jchiem commented Jun 7, 2022

MRobertEvers commented Jun 7, 2022

krasingneves commented Jul 1, 2022

tvsbrent commented Jul 4, 2022

krasingneves commented Jul 4, 2022 •

edited

Joinatic commented Aug 3, 2022

bellis-ai commented Aug 9, 2022

DzianisMakeichyk commented Aug 11, 2022

jchiem commented Aug 12, 2022

jchiem commented Aug 12, 2022

jchiem commented Sep 23, 2022 •

edited

jchiem commented Nov 7, 2022 •

edited

grigorii-merck commented Jan 4, 2023

pawan-bobbili-ntnx commented Feb 6, 2023

20BBrown14 commented Feb 7, 2023

10hendersonm commented May 10, 2023

m-nathani commented Jun 19, 2023

petruki commented Jun 25, 2023

gaving commented Aug 10, 2023

victorgois commented Sep 12, 2023

netishix commented Jan 2, 2024 •

edited

Sweetog commented Jan 9, 2024 •

edited

edwincoronado commented Jan 9, 2024

Momotoculteur commented Mar 8, 2024

alexgomezlf commented Mar 8, 2024

Sweetog commented Mar 8, 2024

petruki commented Mar 9, 2024

yaronguez commented Apr 5, 2024

[BUG] npm ci still does not work with overrides correctly #4942

[BUG] npm ci still does not work with overrides correctly #4942

Comments

jchiem commented May 25, 2022 • edited

Is there an existing issue for this?

This issue exists in the latest npm version

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

darmentrout commented Jun 3, 2022

darmentrout commented Jun 3, 2022

jchiem commented Jun 3, 2022

jchiem commented Jun 7, 2022

MRobertEvers commented Jun 7, 2022

krasingneves commented Jul 1, 2022

tvsbrent commented Jul 4, 2022

krasingneves commented Jul 4, 2022 • edited

Joinatic commented Aug 3, 2022

bellis-ai commented Aug 9, 2022

DzianisMakeichyk commented Aug 11, 2022

jchiem commented Aug 12, 2022

jchiem commented Aug 12, 2022

jchiem commented Sep 23, 2022 • edited

jchiem commented Nov 7, 2022 • edited

grigorii-merck commented Jan 4, 2023

pawan-bobbili-ntnx commented Feb 6, 2023

20BBrown14 commented Feb 7, 2023

10hendersonm commented May 10, 2023

m-nathani commented Jun 19, 2023

petruki commented Jun 25, 2023

gaving commented Aug 10, 2023

victorgois commented Sep 12, 2023

netishix commented Jan 2, 2024 • edited

Sweetog commented Jan 9, 2024 • edited

edwincoronado commented Jan 9, 2024

Momotoculteur commented Mar 8, 2024

alexgomezlf commented Mar 8, 2024

Sweetog commented Mar 8, 2024

petruki commented Mar 9, 2024

yaronguez commented Apr 5, 2024

jchiem commented May 25, 2022 •

edited

krasingneves commented Jul 4, 2022 •

edited

jchiem commented Sep 23, 2022 •

edited

jchiem commented Nov 7, 2022 •

edited

netishix commented Jan 2, 2024 •

edited

Sweetog commented Jan 9, 2024 •

edited