Separate npm/yarn/pnpm managers

[rarkins]

What would you like Renovate to be able to do?

Have separate Renovate "managers" for npm/yarn/pnpm/lerna instead of the combined one today.

Describe the solution you'd like

If they were separated into different managers then it could simplify the code a lot.

Describe alternatives you've considered

Keeping as-is, with "npm" as a single manager.

Additional context

Today we support more than one type of lock file with the same package.json, e.g. package-lock.json and yarn.lock. This isn't a recommended config but some of our users do have it. This would mean we'd need two managers active on the same package file.

A challenge is how to know which is which, and not duplicating work:

• Yarn, pnpm and lerna managers could "abort" if they don't find their respective lock or config files
• npm could abort if it finds config files of the others or package-lock.json is in use

The above would mean that the managers all have to be "aware" of each other though, which can reduce part of the benefits of separating them.

[viceice]

That's really a challenge, because those managers are somehow equivalent. 🤔

What about using the npm manager to only update package.json if required.

add other lock-file manager to update lock files. eg. yarn-lockfile, pnpm-lockfile, npm-lockfile ...
So we run them if we find lockfiles for them. So the only challenge would be to check if we need to run a lockfile update.

[rarkins]

I don’t think it could work with them being separate. One thing that maybe could work is if more than one manager can do the same work and it doesn’t cause a problem

[rarkins]

Ignoring workspaces for now, here is how we would decide whether to extract or not:

• Lerna: if there's a lerna.json
• Yarn: if there's a yarn.lock and no lerna.json
• pnpm: if there's a pnpm-lock.yaml
• npm: if there's a package-lock.json or no lock file at all

Some other thoughts:

• It doesn't really matter if they each try to update the same values in package.json - they will all succeed
• Duplicates could potentially be filtered out during the "branchify" logic too
• There is a problem though if npm or yarn are not aware of "internal" dependencies and try to update then instead of ignore them
• Maybe we could let lerna return a list of ignored dependencies that is applied globally to other managers rather than to only its own deps
• If we can't get it perfect, but any problems can be overcome by the user disabling a conflicting package manager, then it could still be ok
• npm and yarn could potentially use "find-up" to look for lerna.json in a parent directory

[rarkins]

As @viceice suggested earlier, maybe we could trigger pnpm, yarn and lerna based on their lock/config files instead of package.json. So e.g. the packageFile would be lerna.json or yarn.lock but the returned results could come from multiple package.json files (we might add a new optional field like contentFile to help with this). This might be the start of a new concept where the fileMatch is used to find related package files but not necessarily the source of the actual dependency definitions.

Using the "centralized" files like yarn.lock lets us handle the case of when there's a 1:1 ratio of lock to package.json or 1:many.

The "npm" manager would then match all package.json files regardless of whether it has a package-lock.json or not. The problem.. it could update "internal" dependencies within workspaces of yarn or lerna. Possibility: "findUp" for lerna.json or a Yarn workspaces definition. If found, skip the package.json.

[roni-frantchi]

Would having a way to specify which lock files or managers to use help?
Being able to say - for this repository, or even path, update only target pnpm files

[rarkins]

I think what we need to do is this:

• Separate yarn, pnpm and use their lock files as fileMatch (extractAllPackageFiles)
• They return package.json as the packageFiles, not the lock file
• The npm manager would still return one packageFiles entry per package file
• We would need to add some type of logic to "skip" npm results which overlap with yarn and pnpm and don't have their own lock file (note: we still support having both yarn and npm lock files as an example)

One possibility I'd thought about was separating out non-locked package.json as manager "npm" and then having locked npm as a different manager like "packagelock" but I think that would be too confusing to people. Ultimately we're best if the package managers are npm, yarn, pnpm.

I'm not sure what to do about lerna though. Sometimes it just falls back to Yarn behavior. Maybe the lerna manager can "skip" itself if it can fall back to Yarn, but if it doesn't skip itself then it overrides Yarn using a similar method as how Yarn can override npm.

[rarkins]

A useful question: if we completely dropped lerna awareness today, which functionality would break? Lerna has potentially changed a lot since we first added it.

Lerna now supports pnpm (which I don't think we do), however it also looks like none of the Lerna awareness we use (e.g. bootstrap) is used, so maybe it doesn't matter: https://lerna.js.org/docs/recipes/using-pnpm-with-lerna

Lerna's documentation implies that useWorkspaces isn't needed anymore, but is supported: https://lerna.js.org/docs/api-reference/configuration#useworkspaces--packages

This page implies that bootstrap itself is legacy: https://lerna.js.org/docs/features/legacy-package-management#legacy-bootstrap-command

[viceice]

maybe feature flag lerna support and disable on app. if we get issues we can easily re-enable?

[rarkins]

What would feature flagging mean? I'm leaning towards:

• Dropping it in a major, as part of this refactor, so self-hosted folks can stay on an earlier major if they need
• Trying to deal quickly with any gaps we find for those in the app

[viceice]

Ok, let's do that. feature flag would make things more complicated again.

[rarkins]

It looks like from Lerna v7 onwards it drops the bootstrap command anyway and we can just fall back to using package managers. So I suggest in a new major release we say we drop Lerna support / support only Lerna v7

[viceice]

this will probably break a lot of lerna users which are not using the latest lerna version?

[rarkins]

Blocked by #23241

[rarkins]

Here's what we should do to make this separation possible:

• Remove lerna-specific code which was needed before Lerna v7. There is no specific Renovate handling needed for v7 onwards due to the lack of lerna bootstrap
• Remove support for multiple lock files. This means e.g. that if we find a package.json with yarn.lock then we ignore the npm manager
• Split npm/pnpm/yarn managers. Use package.json for npm fileMatch but the lock files for yarn and pnpm

[rarkins]

We are now ignoring Lerna altogether if we detect >=7.0.0. It seems to work fine just by using native npm/pnpm/yarn concepts (no more lerna bootstrap, etc).

[rarkins]

Blocked by #23779

[rarkins]

The npm manager will retain the default fileMatch of package.json

Yarn and pnpm will have their lock files as the fileMatch

All managers will use extractAllPackageFiles() and return package.json files (not lock files)

npm will:

• parse every file
• extract dependencies (common)
• extract npm workspaces
• post-process files
• return array of package.json packageFiles
  (essentially same as today but drop all Yarn/Lerna/pnpm awareness, export the common dependency parsing function)

Yarn/pnpm will:

• Find the lock file's sibling package.json
• Parse it to get workspaces info
• Find all matching package.json in workspaces
• Parse all package.json
• extract dependencies (common)
• post-process files
• return array of package.json packageFiles (not lock files)

Renovate then needs to de-duplicate in case a package.json is matched by npm and yarn/pnpm

[rarkins]

Extras to help the migration:

• Support "npm": {} config if using yarn or pnpm
• Support matchManagers=npm if using yarn or pnpm
• Support enabledManagers
• Check vulnerability fixing

[deviantintegral]

Could the warning message added at #23241 be improved?

When I saw this, I thought it meant we could no longer have multiple independent package.json and associated lock files in a single project. In fact, what this found was an accidentally committed package-lock.json file for a project using Yarn.

Instead of:

Updating multiple npm lock files is deprecated and support will be removed in future versions

How about:

Multiple lock files were found at . Supporting multiple JavaScript packages managers is deprecated and support will be removed in future versions. To fix this, choose only one package manager for the corresponding package.json and remove the other lock files from the repository.