Verifying the integrity of a lockfile

[zkochan]

We cannot guarantee that Git or other version control system merges changes to lockfiles correctly. Issues may happen sometimes (see #3188, #3137). Hence, we need pnpm to verify the correctness of the data which is present in the lockfile.

Currently, pnpm blindly trusts any lockfile that is up-to-date with the package.json file of the repository. This allows pnpm to be extremely fast. However, this also means that in case of an invalid lockfile, pnpm will create an invalid modules directory.

Possible solution: we could add a new field to the lockfile called lockfileChecksum. This field will contain a hash of the lockfile. When the hash will be out-of-date, pnpm will not run the fast install algorithm but will instead revalidate the data in the lockfile.

Cons of this solution. After any merge/rebase that modifies the lockfile, pnpm install will have to be executed to revalidate the lockfile.

cc @pnpm/collaborators

[statianzo]

Currently, pnpm blindly trusts any lockfile that is up-to-date with the package.json file of the repository.

Meaning it do a shallow comparison of top-level modules?

---

Rather than rebuilding the whole lockfile, does something like a Merkle tree make sense here? Could pnpm detect a branch that's out of date without discarding everything?

In the above, root-package is invalidated by the tampered F, but the A@1.0 and B@2.0 don't need to get rebuilt because their child hashes are still valid.

Whatever the implementation, having a means to verify this during CI and fail on checksum mismatch, would be useful. Otherwise, if a poor merge is made, every dev who pulls will be subject to pnpm in "slow mode".

[zkochan]

It would cause even bigger git diffs and the merges would result in conflicts more frequently.

[statianzo]

Good point. Stability of lock file was a major reason I adopted pnpm.

[zkochan]

Another solution might be to always check the actual name/version (or the package ID?) of the package inside the tarball with the name/version that is present in the lockfile. If they don't much, the lockfile is broken.

To implement this, we will probably need to add the name and version of the package to the package index file. The package index file is a file in the store that contains mappings of the package files to their locations in the content-addressable store (for example).

[statianzo]

Certainly a positive that a valid merge wouldn't force checksum recreation because integrity is not stored in the lockfile. What is the downside of this approach? Storage? Migrating from older versions of the structure?

or the package ID?
That would help with git based dependencies, right?

[statianzo]

Cons of this solution. After any merge/rebase that modifies the lockfile, pnpm install will have to be executed to revalidate the lockfile.

Having to pnpm install after any non-pnpm operations on the lockfile would be troublesome.

1. Dev A merges PR into main with pnpm-lock
2. Dev B merges PR into main with pnpm-lock, git merges fine
3. Devs C,D,E pull main and pnpm i
4. C, D, E all have a locally modified pnpm-lock with updated integrity

[vis97c]

Any update? I'm migrating from yarn which has "yarn check --integrity" that I use on my project as a precommit hook, quite useful when working with monorepos.

[cafesanu]

pnpm install --frozen-lockfile does the trick