A universal lockfile format for all package managers

[jonkoops]

I hope this is the right place to bring forward such a proposal, but I am looking for a place to discuss a possible standardization of the lock file format for package managers in the JavaScript ecosystem. There are many package managers out there, all with their own lock file formats:

• NPM — package-lock.json (JSON)
• PNPM — pnpm-lock.yaml (YAML)
• Yarn — yarn.lock (YAML)
• Deno — deno.lock (JSON)
• Bun — bun.lock (JSON)

Having this many different formats impacts portability for projects, essentially locking users into choosing a single package management solution and sticking with it. Moving between package managers is still possible, but could have unintended side effects, such as accidental upgrades of transitive dependencies.

Considering that all formats essentially serve the same purpose with only minor functional differences and is essentially a solved problem space, it seems like now would be a good time to unify efforts and come up with a common standard for lock files that could be implemented by all package managers.

There are some pre-existing efforts in other ecosystems that might serve as inspiration, for example Python recently adopted a new proposal to standardize a lock file format.

[zkochan]

Considering that all formats essentially serve the same purpose with only minor functional differences and is essentially a solved problem space, it seems like now would be a good time to unify efforts and come up with a common standard for lock files that could be implemented by all package managers.

I don’t think that is true. We had a major change in the lockfile format in pnpm v9, and v10 introduced some smaller changes as well. A shared format would make it really hard to innovate. We couldn’t even agree on simple things in the past — let alone on a shared lockfile format.

pnpm's lockfile is designed specifically to make installation in its "linked" node_modules directory structure easier. For a node_modules with hoisted directory structure a different format makes more sense to provide optimal performance.

Having this many different formats impacts portability for projects, essentially locking users into choosing a single package management solution and sticking with it. Moving between package managers is still possible, but could have unintended side effects, such as accidental upgrades of transitive dependencies.

I don’t quite understand why this is considered a problem. This is an area that is in constant development. Even switching between different versions of the same package manager is hard. Allowing different package managers on a single project is only introducing problems.

[ljharb]

If anything, I’d expect the common thing to be npm’s hidden lockfile (which doesn’t get committed). If every package manager CLI kept it accurate, then they could all coordinate without hampering innovation.

[zkochan]

But that hidden file seem to be just a copy of the non-hidden one and it also reflects that hoisted node_modules directory structure.

I guess we could share a separate file that would just be a map of all the packages used inside node_modules. It would be only the package names, versions and integirty checksums. We store that currently in the "packages" field of "pnpm-lock.yaml". However, we store less information than npm and yarn. We don't include the version specs of the dependencies. Except for peer dependencies.

[arcanis]

Speaking for Yarn I don't feel very attracted by this proposal. Package managers, like linters or bundlers, are different, with different tradeoffs, feature sets, and the content of the lockfile is the least of these differences.

A second reason is that lockfile format aren't compatible between them for fundamental reasons related to the package managers' designs. As an example, Yarn simply cannot support an exact same range to resolve to multiple versions - and thus our lockfile doesn't support it. Npm doesn't follow this rule, so their lockfile doesn't prevent it.

You're free to implement a lockfile converter (Yarn even had one in the past, which we discontinued for the reasons described above), but that's not a job for Yarn.

[ljharb]

@zkochan yes, when using eg pnpm, you wouldn't respect the structure, but you'd still ensure that all tools had a common file to read - it might not have value, but if so that seems like the path to me.

[selfisekai]

Speaking for related prior art:

Having this many different formats impacts portability for projects, essentially locking users into choosing a single package management solution and sticking with it.

package.json is already not an actual standard either. Some incompatibilities that I remember:

1. overrides and resolutions. npm uses the former, yarn uses the latter, bun uses both, pnpm has its own. It's not just this keyword name, the format of the contents is incompatible too.
2. Dependencies between workspace members. The explicit workspace: protocol is 2 completely different things between package managers - either a path (workspace:ligma-api) or a version range (workspace:^1.0.0) is expected.
3. GitHub repository specifiers (isaacs/minimatch#v10.0.1) are by design ambiguous as they can be resolved to either git+https, git+ssh, or archive tarballs. Oh, and even if you specify your dependency via one of these, some package managers will use a different source if it's on github.com specifically.
4. Generally the same strings resolving to different things or nothing. Like a https URL will work in most package managers as a tarball, but error in some because its path does not end with .tgz. Or git@github.com:npm/node-semver.git#semver:^7.5.0 will be interpreted as an npm tag.
5. Extensions to JSON syntax. Yes.

---

Re: separate standard file - I see a few things:

1. An installation might not create a node_modules directory by design. If that's the place where it'd land. Otherwise that'd require either every project to adjust their ignore files, or making it opt-in some way.
2. It gets created after a local installation is run - no static analysis before attempting an install.
3. It's read-only (changes won't be applied back).

I very much like one thing about this idea though: it'd allow to provide metadata that might too verbose for a lockfile, or not even be known before an installation by PM design. node_modules/.yarn-state.yml (yarn >=2) lists module locations, for example:

# Warning: This file is automatically generated. Removing it is fine, but will
# cause your node_modules installation to become invalidated.

__metadata:
  version: 1
  nmMode: classic

"@a/verboden(name~'!*)@npm:1.0.0::__archiveUrl=https%3A%2F%2Fs.lnl.gay%2F%40a%2Fverboden(name~'!*)%2F-%2Fverboden(name~'!*)-1.0.0.tgz":
  locations:
    - "node_modules/@a/verboden(name~'!*)"

"verboden(root)(name~'!*)@workspace:.":
  locations:
    - ""

[benjamincburns]

Having this many different formats impacts portability for projects, essentially locking users into choosing a single package management solution and sticking with it. Moving between package managers is still possible, but could have unintended side effects, such as accidental upgrades of transitive dependencies.

I don’t quite understand why this is considered a problem. This is an area that is in constant development. Even switching between different versions of the same package manager is hard. Allowing different package managers on a single project is only introducing problems.

Replying specifically to the first and last sentences quoted above from @zkochan here. I don't dispute the middle two.

I'm a maintainer for LangChain & LangGraph. When a user encounters an issue in one of our packages, they often want to dive into the code and contribute a fix. Somewhere early on in the project (before my time, anyway) some member of our team decided it'd be a good idea to use yarn as the package manager because running yarn for our monorepo is significantly faster than running npm install.

Because of that inward-facing choice, users who want to contribute to our projects must use yarn when working with our project. That doesn't seem like a big deal, and for many it isn't, but for many people who aren't familiar with yarn, it adds substantial friction to the process of digging in to figure out what's wrong, let alone the process of making a contribution.

We also have a similar problem internally, where some members of our team prefer yarn and some prefer pnpm, so rather than keeping things related to one product in the monorepo for that product we wind up with "side projects" in separate repos that use a different package management & build config and become burdensome for a small team to deal with when they stop being side projects. Sure we can say "knock that off" and ask them to exercise more discipline around this, but wouldn't you rather live in a world where you're free to use the tools that you like? Maybe you wouldn't, but I think most people participating in this ecosystem who aren't maintaining a package manager certainly would like to.

But even on purely technical merit, I'm sure that you can point to a ton of ways in which NPM, yarn, and PNPM are functionally different (some subtle, some not so subtle), but there are so many projects out there where you can literally just delete the lock file and the packageManager key in the package.json (if there even is one) and then run $PACKAGE_MANAGER install and it'll work fine. For the projects where this isn't the case many of us would seek to actively change that if there were a common lock file standard that we could adopt, as the reduction in "onboarding friction" for new contributors would be substantial.

[benjamincburns]

A second reason is that lockfile format aren't compatible between them for fundamental reasons related to the package managers' designs. As an example, Yarn simply cannot support an exact same range to resolve to multiple versions - and thus our lockfile doesn't support it. Npm doesn't follow this rule, so their lockfile doesn't prevent it.

@arcanis There's probably history here that I'm not aware of, so please forgive my ignorance, but is this example a mutually exclusive requirement, or just an incompatibility in the current formats? I'm not familiar with this specific rule, but from what you wrote here it sounds like it's a case where NPM can be more permissive, so it is, but if there was a standard format that imposed a stricter requirement on NPM, they could follow it when consuming/producing the standard file? Or is there something fundamental to the way NPM resolves version ranges that would break it if this stricter rule were imposed?

I've seen this idea of a common lockfile format batted around a bunch and it always seems to be quickly dismissed by the package manager maintainers. I just can't help but wonder what would happen if people from the three major package managers got together and came up with a list of mutually exclusive conflicting requirements that make a standard lockfile format infeasible. Maybe those mutually exclusive requirements really are immovable objects and a compatibility mode would just suck for everyone, but I (admittedly, very naïvely) wonder how long of a list it would be, and whether the compromises necessary to support a compatibility mode for a standard format would be above or below the "worth it" threshold for users wishing this could exist.

Also I'm not sure if this makes things any easier, but if transitioning to a standard lockfile format meant that I couldn't keep the version resolution from my current yarn.lock for my dependency tree, that's totally fine, IMO. I can take that one-time hit as long as the installed dependency tree is consistent across package managers going forward after switching to the standard format.

[ljharb]

@benjamincburns the latest solution to that problem is to use devEngines.packageManager. npm will respect it.

[benjamincburns]

@benjamincburns the latest solution to that problem is to use devEngines.packageManager. npm will respect it.

@ljharb Sorry, which problem does that solve? I mentioned a few problems in my last two comments 😅

[ljharb]

The first comment, where you want people to use yarn instead of npm.

[benjamincburns]

The first comment, where you want people to use yarn instead of npm.

@ijharb I don't want that, though. That's the whole point of my first comment. I want them to be able to use whatever package manager they prefer without imposing a particular one upon them.

[ljharb]

Ah. Then this issue is the thing, but it’s not currently possible, and without the support of ⅔ of the primary cli option maintainers, it likely won’t ever happen.

[RDIL]

At the end of a day, the point of a lockfile is to have reproducible installs. Unless every package manager behaves the exact same way, you're not going to get that. And if every package manager behaves the same way... what's the point of having more than one implementation?

---

Speaking of - different package managers have their own feature sets. Say for example (humor me here) I write a Yarn plugin that gives it the ability to download packages from RubyGems and link them as npm dependencies (in node_modules). If my package.json has "thor": "rubygems:thor@^1.3.2", a common lockfile format could say that I've got the exact resolution of thor@rubygems:https://rubygems.org/downloads/thor-1.3.2.gem as the resolution, but that doesn't mean any package manager can just download it, unpack the Gem file, and know how to make it into a folder.

Obviously this is quite an "out there" idea, but the only reason I bring it up is because I do know of at least 1 company who uses this exact system already, so, 🙃