Node.js 14 / NPM 6 - Using dep with npm: prefix can result in invalid lockfiles

[patrickleet]

What Renovate type, platform and version are you using?

Github

Describe the bug

The generated package-lock file from renovate is sometimes invalid.

I've been using renovate for years, only seeing this for the first time in the past couple days.

When running npm ci in the CI process, and when cloning it down locally to verify, I get this error:

npm ci        
npm WARN prepare removing existing node_modules/ before installation
npm ERR! Cannot read property 'fetchSpec' of undefined

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/patrickleet/.npm/_logs/2021-04-20T15_08_26_091Z-debug.log

The logs from that log file are:

cat /Users/patrickleet/.npm/_logs/2021-04-20T15_08_26_091Z-debug.log0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'ci' ]
2 info using npm@6.14.13
3 info using node@v14.16.1
4 verbose npm-session 49f46ed1129803aa
5 info prepare initializing installer
6 verbose prepare starting workers
7 verbose prepare installation prefix: /Users/patrickleet/dev/rivi/engineering-meta/apps/projects/connect
8 verbose prepare using package-lock.json
9 warn prepare removing existing node_modules/ before installation
10 verbose checkLock verifying package-lock data
11 verbose teardown shutting down workers.
12 info teardown Done in 0s
13 verbose stack TypeError: Cannot read property 'fetchSpec' of undefined
13 verbose stack     at /usr/local/lib/node_modules/npm/node_modules/lock-verify/index.js:41:52
13 verbose stack     at Array.forEach (<anonymous>)
13 verbose stack     at /usr/local/lib/node_modules/npm/node_modules/lock-verify/index.js:25:25
14 verbose cwd /Users/patrickleet/dev/rivi/engineering-meta/apps/projects/connect
15 verbose Darwin 20.3.0
16 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "ci"
17 verbose node v14.16.1
18 verbose npm  v6.14.13
19 error Cannot read property 'fetchSpec' of undefined
20 verbose exit [ 1, true ]

Running npm i again patches the lock file and it then works.

Have you created a minimal reproduction repository?

No, I'm seeing it on multiple repositories, but they are private.

In the example I'm seeing now it's babel-monorepo being updated, but I don't think that's anything to do with the issue as I saw it on another dep as well.

If it's needed I can try to create something, but I don't even know that it'll happen. I use renovate on a lot of repos in many orgs to do the same thing.

Did the way lockfiles get generated change recently? Maybe something to do with npm6/7 which use different lockfile versions? It doesn't change the lockfile version so that seems unlikely..

Please read the minimal reproductions documentation to learn how to make a good minimal reproduction repository.

I do have an .npmrc for this project but I don't think that's the issue either?

• I have provided a minimal reproduction repository
• I don't have time for that, but it happens in a public repository I have linked to
• I don't have time for that, and cannot share my private repository
• The nature of this bug means it's impossible to reproduce publicly

Additional context

[patrickleet]

oddly this is literally the only diff in the package-lock.json file after cloning down the branch and running npm i

And that makes npm ci work again.

So maybe it's something to do with this dependency format:

"react-dom": "npm:@hot-loader/react-dom@16.14.0",

I'll see about changing that?

[github-actions]

Hi there,

The Renovate team needs your help! Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on minimal reproductions to understand what is needed.

We may close the issue if you have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[rarkins]

Unfortunately there's nothing about your description or the npm ci logs which would get us any close to reproducing or fixing. If you have the debug logs from when such a "bad" commit is made then maybe we can guess at it, but otherwise not.

[patrickleet]

There was a chance you've seen it before.

I think it has something to do with that "react-dom": "npm:@hot-loader/react-dom@16.14.0", format, as it's happening in a couple projects that all use that, which are all newly renovate enabled and all based on the same template.

I'll mess with it more.

[patrickleet]

Do you just use npm i to create the lockfile? It's odd to me that my result and renovate's would be slightly different.

[rarkins]

It defaults to use the --package-lock-only flag, which sometimes does produce incorrect results (eg often when file: refs are used). Perhaps npm: is another scenario where it's buggy. We have logic to detect existing known buggy scenarios for npm and switch to a full node modules install instead.

[patrickleet]

That's it.

I deleted node_modules, then ran npm i --package-lock-only which reproduced the error

➜  rm -rf node_modules 
➜  npm i --package-lock-only                   
added 2200 packages in 6.937s
➜  npm ci                   
npm ERR! Cannot read property 'fetchSpec' of undefined

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/patrickleet/.npm/_logs/2021-04-20T16_55_07_686Z-debug.log

[rarkins]

Any chance if running it twice fixes it?

[patrickleet]

just tried - it does not

[patrickleet]

so basically it seems more like an npm bug

[rarkins]

It is, but we should be able to work around the bug like we do with file: references. See

renovate/lib/manager/npm/extract/index.ts

Lines 347 to 357 in 10e5119

	if (skipInstalls === null) {
	if (hasFileRefs) {
	// https://github.com/npm/cli/issues/1432
	// Explanation:
	// - npm install --package-lock-only is buggy for transitive deps in file: references
	// - So we set skipInstalls to false if file: refs are found *and* the user hasn't explicitly set the value already
	logger.debug('Automatically setting skipInstalls to false');
	skipInstalls = false;
	} else {
	skipInstalls = true;
	}

[rarkins]

Can you create a minimal reproduction so that we can verify any fix works?

[patrickleet]

I tried, but the error didn't happen in it - will need to spend more time on it

[patrickleet]

https://github.com/CloudNativeEntrepreneur/renovate-9654-reproduction/pull/4/checks?check_run_id=2394884645

[rarkins]

private repo?

[renovate-release]

🎉 This issue has been resolved in version 24.119.6 🎉

The release is available on:

• GitHub release
• 24.119.6

Your semantic-release bot 📦🚀

[patrickleet]

oh my bad - it's public now

[rarkins]

BTW although I submitted a hopefully automatic workaround, I realized that we allow a configurable skipInstalls which you can set to false in config, so that might have worked right away for you.