[turborepo] error installing depencies from pruned lock file

[JamesAlexanderHill]

What version of Turborepo are you using?

1.6.3

What package manager are you using / does the bug impact?

Yarn v1

What operating system are you using?

Mac

Describe the Bug

In my docker file I have the following step:
RUN turbo prune --scope=$APP_NAME --docker
This generates the .out directory with the following structure

• /yarn.lock
• /json/
• /full/

I then copy the yarn.lock + json and install dependencies with these steps:

COPY --from=builder /app/out/json/ .
COPY --from=builder /app/out/yarn.lock ./yarn.lock
RUN yarn install --production --frozen-lockfile

I get the following error:
error Your lockfile needs to be updated, but yarn was run with --frozen-lockfile.

Looks like prune is not correcctly pruning the yarn.lock

Expected Behavior

Dependencies can be installed with yarn install --production --frozen-lockfile without error.

yarn.lock should match package.json

To Reproduce

install dependencies from pruned repo with docker flag.

Reproduction Repo

No response

[keawade]

I'm running into the npm equivalent of this over in npm land.

COPY --from=pruner /app/out/json/ .
COPY --from=pruner /app/out/package-lock.json ./package-lock.json
RUN --mount=type=cache,target=/root/.npm \
  --mount=type=secret,id=npmrc,target=/root/.npmrc \
  npm ci --no-audit --no-fund --prefer-offline

which results in

#18 [builder  8/12] RUN --mount=type=secret,id=npmrc,target=/root/.npmrc   npm ci --no-audit --no-fund --prefer-offline
#18 sha256:55bf88e06459432be0618ed1ad56c8f11a7c3c95727718aa617f3026dac17123
#18 6.535 npm notice 
#18 6.535 npm notice New major version of npm available! 8.19.2 -> 9.1.1
#18 6.535 npm notice Changelog: <https://github.com/npm/cli/releases/tag/v9.1.1>
#18 6.535 npm notice Run `npm install -g npm@9.1.1` to update!
#18 6.535 npm notice 
#18 6.556 npm ERR! code EUSAGE
#18 6.584 npm ERR! 
#18 6.584 npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
#18 6.584 npm ERR! 
#18 6.584 npm ERR! Missing: amqplib@0.10.3 from lock file
#18 6.584 npm ERR! Missing: avsc@5.7.7 from lock file
#18 6.584 npm ERR! Missing: bunyan@1.8.15 from lock file
...

The list of missing dependencies keeps going for 245 (all?) of my dependencies.

I'm using node:18-alpine as my base image.

[keawade]

I did some more digging and determined that the pruned lockfile created by turbo prune is identical to a lockfile generated by npm install with npm@9 (which also uses lockfileVersion 3). So I'm super baffled but my issue doesn't seem a result of turbo prune spitting out the wrong npm lockfile, at least.

EDIT: This is inaccurate. I compared the wrong samples. Comparing the correct samples reveals some differences. The vast majority of things are inconsequential-to-installation stripping of package metadata like funding fields. While I am seeing some node_modules/* packages missing from the pruned version those seem like they should also be inconsequential.

Spot-checking items from the list npm ci claims is missing has mixed results. Several items do actually show up in the lockfile but several items only appear as peerDependencies and I'm not sure if that's consequential or not.

For reference, the lockfile samples I'm comparing are the package-lock.json file generated by turbo --prune --scope=app --docker and the package-lock.json file generated by running npm install in a barebones folder only containing the out/json/ folder contents generated by turbo --prune --scope=app --docker.

[keawade]

Okay some folks I work with took a look with me and it does appear that the npm issue, while likely still related, isn't the same issue as the yarn issue.

I've opened #2739 to track the npm issue.

@JamesAlexanderHill I tried to reproduce the yarn issue here: https://github.com/keawade/turbo-wat/tree/yarn but it appears to either be something else or not as directly translatable to the npm issue I ran into. Feel free to fork that repo/branch if you're looking to create a reproduction repo.

[chris-olszewski]

Hi @JamesAlexanderHill, could you provide a reproduction? If that's not possible, getting at least the original and pruned lockfile would me debug the issue.

[lecstor]

@chris-olszewski I've hit the same issue and it appears to have been triggered by our use of yarn resolutions on immer and having deps on multiple [major] versions of immer.

Our repo currently only has a single package which I am pruning to (trying to prepare for soe apps to be added and don't want CI installing all the deps just to publish the lib).

in root package.json..

  "resolutions": {
    "immer": "^9.0.6",
  }

in root yarn.lock

"immer@npm:^9.0.6":
  version: 9.0.16
  resolution: "immer@npm:9.0.16"
  checksum: e9a5ca65c929b329da7a3b7beccf7984271cda7bdd47b2cab619eac3277dcd56598c211b55cc340786b6eff0c06652ac018808d9fd744443f06882364dece6bc
  languageName: node
  linkType: hard

immer@npm does not appear in the pruned yarn.lock

Running yarn install in the pruned repo adds immer to the yarn.lock.

When I remove the resolutions entry for immer both our root yarn.lock and the pruned yarn.lock contain the following entries for immer and running yarn install in the pruned repo does not modify it..

"immer@npm:8.0.1":
  version: 8.0.1
  resolution: "immer@npm:8.0.1"
  checksum: 63d875c04882b862481a0a692816e5982975a47a57619698bc1bb52963ad3b624911991708b2b81a7af6fdac15083d408e43932d83a6a61216e5a4503efd4095
  languageName: node
  linkType: hard

"immer@npm:^8.0.4":
  version: 8.0.4
  resolution: "immer@npm:8.0.4"
  checksum: 9d3b28df1ac5bf6918c611e71c15bdb136588c21a3431100448f21325fef9b055cc9a44fe8b023f0c5ecbc66a2ba38f403c9a67d581f613b49d0e4ff15564f79
  languageName: node
  linkType: hard

"immer@npm:^9.0.7":
  version: 9.0.16
  resolution: "immer@npm:9.0.16"
  checksum: e9a5ca65c929b329da7a3b7beccf7984271cda7bdd47b2cab619eac3277dcd56598c211b55cc340786b6eff0c06652ac018808d9fd744443f06882364dece6bc
  languageName: node
  linkType: hard

[chris-olszewski]

I expect this to be fixed in the same PR that solves #2791. Closing this in favor of that issue.