[rush] "rush update" does not recover from ERR_PNPM_TARBALL_INTEGRITY

[octogonz]

Summary

If an NPM version is republished with a different tarball hash, then PNPM correctly detects this and reports ERR_PNPM_TARBALL_INTEGRITY. However rush update does not handle it correctly; it waits for 1 minute and then ignores the problem. The lockfile can only be repaired by using rush update --full.

This seems like a bug. Invoking rush update should repair the lockfile.

Repro steps

This repro relies on republishing a tarball to Verdaccio. A simpler repro is possible, but it's useful to show how this problem arose in real life.

1. Shell window A: Start the Verdaccio service

"lockfile-explorer-demos" folder, "main" branch

git clone https://github.com/microsoft/lockfile-explorer-demos lockfile-explorer-demos

cd lockfile-explorer-demos
pnpm install

pnpm start

2. Shell window B: Publish the test package

"lockfile-explorer-demos" folder, "main" branch

cd lockfile-explorer-demos

pnpm publish-all

3. Shell window C: Install the test package

"lockfile-explorer-demos-checkout" folder, "demo/sbs-0" branch

git clone https://github.com/microsoft/lockfile-explorer-demos lockfile-explorer-demos-checkout

cd lockfile-explorer-demos-checkout

git checkout demo/sbs-0

rush update --full

4. Shell window A: Modify the test package

"lockfile-explorer-demos" folder, "main" branch

# (Kill the Verdaccio service with CTRL+C)
# cd lockfile-explorer-demos

# Change the content of package @rushstack/p@2.0.3
rm demo-packages/p@2.0.3/index.js

# Clear the Verdaccio storage
rm -Rf temp

# Restart the service
pnpm start

5. Shell window B: Republish the test package

"lockfile-explorer-demos" folder, "main" branch

# cd lockfile-explorer-demos

# Republish @rushstack/p@2.0.3 with a new tarball hash
pnpm publish-all

6. Shell window C: Try installing again

"lockfile-explorer-demos-checkout" folder, "demo/sbs-0" branch

# cd lockfile-explorer-demos-checkout

rush purge
rush update

Final state

rush update takes a very long time to complete (because of the "1 minute" delay):

Running "pnpm install" in C:\t\lockfile-explorer-demos-checkout\common\temp

Scope: all 3 workspace projects
 WARN  GET http://localhost:54321/@rushstack/p/-/p-2.0.3.tgz error (ERR_PNPM_TARBALL_INTEGRITY). Will retry in 10 seconds. 2 retries left.
.                                        |   +1 +
 WARN  GET http://localhost:54321/@rushstack/p/-/p-2.0.3.tgz error (ERR_PNPM_TARBALL_INTEGRITY). Will retry in 1 minute. 1 retries left.
.                                        |  WARN  Got unexpected checksum for "http://localhost:54321/@rushstack/p/-/p-2.0.3.tgz". Wanted "sha512-hWg90O494I5DyrbPxYR2Ml7zUrlXHXYo8KLK8AwYYEuAMcjjUe1zUzeBFt+X07arsGhgPrPabkALMnltNWB2OQ==". Got "sha512-EJaQNhLBn0Fa+b0Xgg9AuMT6/5fkptdqjZ322P26UyvvHgERiz2FQndv1HXSXf5AFrO9m1X+bhy/JJ4mqvSwhg==".
 ERR_PNPM_ERR_PNPM_TARBALL_INTEGRITY  The lockfile is broken! A full installation will be performed in an attempt to fix it.
.                                        |   +1 +
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: C:\t\lockfile-explorer-demos-checkout\common\temp\pnpm-store\v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 2, reused 1, downloaded 0, added 1, done
Done in 1m 10.6s

Afterwards it claims to have succeeded (Rush update finished successfully.) and running rush install will finish immediately in a green state. However the common/config/rush/pnpm-lock.yaml file is NOT updated. If you run rush purge, then rush update will fail again.

rush update --full does fix the problem.

Details

Using Process Monitor to trace the kernel calls, I was able to determine that the cached state is being read from this location:

C:\Users\Owner\AppData\Local\pnpm-cache\metadata\localhost+54321\

At first I mistakenly thought that this was the cause, however I later determined that it is unrelated. Similar to the --store parameter, maybe we should use --cache-dir to isolate this state?

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
@microsoft/rush globally installed version?	5.83.3
rushVersion from rush.json?	5.83.3
useWorkspaces from rush.json?	true
Operating system?	Windows
Node.js version (node -v)?	v16.15.1

[octogonz]

Maybe --fix-lockfile would help?

[dmichon-msft]

Maybe --fix-lockfile would help?

We should probably consider including --fix-lockfile in the default parameters to pnpm when running rush update, aye.