Explore pnpm and yarn v3

[tschaffter]

Yarn v1 is in maintenance mode since 2020. yarn v3 and pnpm are the most promising candidates to replace yarn v1. They should also provide a solution to the need (?) to remove node_modules after a git merge.

Node
Yarn v1 is still extensively used because v2 introduced many broken changes, which lead to a controversy. Not sure about the state of v3.

[tschaffter]

This article (2022-01) says that import features like go-to-definition are missing from yarn v3.

[tschaffter]

• Angular is still using yarn v1.
• Nx is still using yarn v1.
  • feat(core): upgrade yarn v1 => v3 [berry] && replace npx with yarn dlx nrwl/nx#7918

[tschaffter]

Upgrading to yarn berry (v3)

I upgraded to yarn v3 using the instruction in this article and was able to successfully build and run the challenge registry after clearing the nx cache.

Yarn stores every package in a global cache in your user directory on the file system.

TODO:

• Configure yarn 3 caching in CI workflow (https://github.com/actions/cache/blob/main/examples.md#node---yarn-2)

[tschaffter]

Benchmark

Hardware

• Intel(R) Core(TM) i7-9700K CPU @ 3.60GHz (8 cores)
• Gigabit internet

Yarn 1.22.19

Install:

$ hyperfine --warmup 1 --prepare 'rm -fr node_modules' --runs 5 'rm -fr yarn.lock && yarn install'
Benchmark 1: rm -fr node_modules && rm -fr yarn.lock && yarn install
  Time (mean ± σ):     86.102 s ±  5.088 s    [User: 68.002 s, System: 61.143 s]
  Range (min … max):   81.725 s … 93.151 s    5 runs

Install with lock file:

$ hyperfine --warmup 1 --prepare 'rm -fr node_modules' --runs 5 'yarn install --frozen-lockfile'
Benchmark 1: rm -fr node_modules && yarn install --frozen-lockfile
  Time (mean ± σ):     63.509 s ±  2.573 s    [User: 63.028 s, System: 60.387 s]
  Range (min … max):   59.299 s … 65.834 s    5 runs

Install the package colors:

$ hyperfine --prepare 'yarn remove colors' --runs 5 'yarn add colors'
Benchmark 1: yarn add colors
  Time (mean ± σ):     19.879 s ±  0.108 s    [User: 19.846 s, System: 18.475 s]
  Range (min … max):   19.753 s … 20.035 s    5 runs

Yarn 3.2.2

Install:

$ hyperfine --warmup 1 --prepare 'rm -fr node_modules' --runs 5 ' rm -fr yarn.lock && yarn install'
Benchmark 1: rm -fr node_modules && rm -fr yarn.lock && yarn install
  Time (mean ± σ):     52.966 s ±  1.156 s    [User: 66.253 s, System: 33.588 s]
  Range (min … max):   51.778 s … 54.747 s    5 runs

Install with lock file:

$ hyperfine --warmup 1 --prepare 'rm -fr node_modules' --runs 5 'yarn install --immutable'
Benchmark 1: rm -fr node_modules && yarn install --immutable
  Time (mean ± σ):     38.957 s ±  5.146 s    [User: 47.455 s, System: 34.421 s]
  Range (min … max):   35.419 s … 47.780 s    5 runs

Install the package colors:

$ hyperfine --prepare 'yarn remove colors' --runs 5 'yarn add colors'
Benchmark 1: yarn add colors
  Time (mean ± σ):      5.356 s ±  0.040 s    [User: 5.953 s, System: 0.789 s]
  Range (min … max):    5.307 s …  5.409 s    5 runs

[tschaffter]

Switching between Yarn versions

vscode@ade7130aff68:/workspaces/challenge-registry$ yarn set version 1.22.17
➤ YN0000: Retrieving https://github.com/yarnpkg/yarn/releases/download/v1.22.17/yarn-1.22.17.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-1.22.17.cjs
➤ YN0000: Done in 1s 330ms

vscode@ade7130aff68:/workspaces/challenge-registry$ yarn set version 1.22.19
Resolving 1.22.19 to a url...
Downloading https://github.com/yarnpkg/yarn/releases/download/v1.22.19/yarn-1.22.19.js...
Saving it into /workspaces/challenge-registry/.yarn/releases/yarn-1.22.19.cjs...
Updating /workspaces/challenge-registry/.yarnrc...
Done!

vscode@ade7130aff68:/workspaces/challenge-registry$ yarn set version 3.2.2  
➤ YN0000: Retrieving https://repo.yarnpkg.com/3.2.2/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-3.2.2.cjs
➤ YN0000: Done in 0s 490ms

Global and local yarn versions

vscode@ade7130aff68:/workspaces/challenge-registry$ yarn --version
3.2.2
vscode@ade7130aff68:/workspaces/challenge-registry$ cd ~
vscode@ade7130aff68:~$ yarn --version
1.22.19

[tschaffter]

Install project dependencies with PnP

• Comment out the line nodeLinker: node-modules in .yarnrc.yml.
• yarn install generates the following error:

# This file contains the result of Yarn building a package (challenge@workspace:.)
# Script name: postinstall

Error: EROFS: read-only filesystem, open '/node_modules/.ngcc_lock_file'
    at makeError$1 (/workspaces/challenge-registry/.pnp.cjs:33270:24)
    at EROFS (/workspaces/challenge-registry/.pnp.cjs:33297:10)
    at ZipFS.prepareWriteFile (/workspaces/challenge-registry/.pnp.cjs:35131:13)
    at ZipFS.writeFileSync (/workspaces/challenge-registry/.pnp.cjs:35115:53)
    at /workspaces/challenge-registry/.pnp.cjs:36197:20
    at /workspaces/challenge-registry/.pnp.cjs:36413:60
    at ZipOpenFS.getZipSync (/workspaces/challenge-registry/.pnp.cjs:36542:14)
    at ZipOpenFS.makeCallSync (/workspaces/challenge-registry/.pnp.cjs:36413:17)
    at ZipOpenFS.writeFileSync (/workspaces/challenge-registry/.pnp.cjs:36194:17)
    at VirtualFS.writeFileSync (/workspaces/challenge-registry/.pnp.cjs:35527:24)

Related:

• https://dev.to/arcanis/introducing-yarn-2-4eh1
• Nx does not support PnP yet.
• Storybook does not support PnP yet.

Yarn 2+ without PnP already provides a significant boost in performance and other improvements over Yarn 1. Further exploration of PnP will be left for another day/task.

[pongells]

Hi @tschaffter, I am a bit confused by the lack of official documentation -- does the result of this exploration mean nx supports yarn3 (with node-modules) without issues?

[tschaffter]

Yes, we successfully migrated this Nx workspace to Yarn 2+ three weeks ago. Another team in our org then tried to migrate their non-nx project to Yarn 2+ and bumped into a couple of issues listed below

• GitHub dependabot support (hopefully coming in the next few weeks)
• Storybook Vite Builder support

[ld210]

Yarn PnP mode (zero install) brought a significant boost in performance in our CI. 15 to 20mn before, 5 to 8mn with PnP, which is a huge improvement. A good chunk of those 5 to 8 minutes is taken by pods waiting to be ready, so even if the code base continues to grows, CI time will not grow as fast. It's a huge incentive towards PnP, and i don't see myself rollback to old node_modules, my teammates would lose their mind.

[tschaffter]

@ld210 I need to read more about zero install. While zero install is recommended and used by the Yarn team, it is usually not recommended to track binary and/or large files on Git(Hub) such as the .zip files that comes with zero install.

I just gave another try to PnP by removing nodeLinker: node-modules and now am able to run yarn install for this monorepo. Without PnP, I can execute binaries that live in node_modules/bin directly, e.g. nx, serverless, etc. When using PnP, it seems that I have to prefix these command with yarn for them to work, e.g. yarn nx, yarn serverless. I find it slightly painful to prefix the commands. Is there a workaround?

EDIT: I added node_modules/.bin to the path, this is why I can run nx instead of yarn nx, for instance. But that seems more difficult with PnP given that the path to the binary may change anytime (is it true?).

$ yarn bin nx
/workspaces/challenge-registry/.yarn/unplugged/nx-virtual-63eb908039/node_modules/nx/bin/nx.js

$ ls /workspaces/challenge-registry/.yarn/unplugged/nx*
/workspaces/challenge-registry/.yarn/unplugged/nx-virtual-5e8d409805:
node_modules

/workspaces/challenge-registry/.yarn/unplugged/nx-virtual-63eb908039:
node_modules

[tschaffter]

Interesting discussion related to my previous comment: yarnpkg/berry#180

[ld210]

@tschaffter
the .zip files generated by yarn install are pretty small files. On our project, dependencies went from 2Go to 300Mo, each zip files are relatively small (lots of 10/50ko, biggest one is 7Mo). Some of our tracked assets files are bigger than that, so it is not an issue for us, but it could be for projects with large dependencies.
About the "yarn" prefix, it is indeed annoying ! I didn't found a workaround, but considering the gains, i can live with it.
I found that with PnP, there is a fairly large amount of configuration at the beginning, specific to each project, that can be painful. On our own project, it was relatively easy to make it work. Not having many dependencies, and zero non-Ivy Angular libraries, helped a lot.
You are right, the path to the binaries may change, at each upgrade.
Thanks for the link to the discussion, very interesting, indeed !
All in all, PnP mode in its current state may not be for everyone, it is very project-specific. But the gains can be huge and worth the trouble. Node modules are such a pain, it's a relief not to have to deal with them anymore.
The time saved on CI alone was worth it.