eventually move to Yarn 2+ (currently 4)

[warner]

@katelynsills and @btulloh noticed that the Yarn docs encourage folks to install Yarn 2, but this tree only works with Yarn 1. A quick survey showed that many (all?) of us have Yarn 1 installed locally, so we haven't noticed the problem before.

Ideally our tree should work with both versions. Fallback position is to work with Yarn 2 and make sure the docs point people at installing it rather than yarn 1 (with a stretch goal of somehow detecting when you're using yarn 1 and printing a useful error message).

[katelynsills]

I think we can specify the version of yarn to use by entering (yarn policies set-version $VERSION) and committing the results. I have yarn 2 installed and used by default, but after making this change in a package (to yarn 1.21.1), it runs yarn 1.

Also here's a very informative blog post on what's new in Yarn 2: https://dev.to/arcanis/introducing-yarn-2-4eh1

[warner]

Hm, I did that, and the two files it added were .yarnrc (makes sense) and a 5MB .yarn/releases/yarn-1.21.1.js (which seems overkill). I'm guessing it's copying the entire Yarn executable into our source tree. Any idea how we might make this better?

Cc @jfparadis @michaelfig in case you've got some ideas

[warner]

Hrm, reading https://legacy.yarnpkg.com/en/docs/cli/policies it seems like this is the intended behavior (and they don't find it to be a big deal). If that seems reasonable to everyone else, I'll make a PR for it.

[katelynsills]

Sounds good to me

[warner]

Ok, we're pinned to yarn-1.21.1 now. I'll rename the ticket to reflect a low-priority goal to move to Yarn 2 once it's stable and ready.

[warner]

https://yarnpkg.com/getting-started/migration/ has notes on how one might make this transition.

[dckc]

@turadg , I bemoaned the fact that yarn build rebuilds everything in our tree, even things where the inputs haven't changed. You referred to this issue as a solution. I'm struggling to confirm that yarn 2 is supposed to have this "build only stuff whose inputs have changed" feature. I'm running into measurements that show yarn 2 is slower, in fact:

Speed winner: Yarn 1 by a long shot.
-- Yarn 1 vs Yarn 2 vs NPM... 2020

Help me find docs on the build performance optimizations in yarn 2+, please?

[turadg]

Help me find docs on the build performance optimizations in yarn 2+, please?

Cached is the most common case. The doc you linked to shows 14s vs 29s for 2.0-rc.27 vs v1 (whatever version was ~4 yrs ago).

For more recent data: https://yarnpkg.com/features/performances

yarn build rebuilds everything in our tree, even things where the inputs haven't changed. You referred to this issue as a solution.

Correct. Are you also asking for confirmation of that? Check out #8461 (in a new directory avoid borking your current node_modules) and run yarn install a couple times.

Then to see what it's like changing between branches, bump a dep. E.g. "typescript" to 5.3.3 in the root package.json. yarn install again to warm the cache. Then reset hard to simulate switching from a feature branch back to master.

With v3 it completes in ~1s:

➤ YN0000: ┌ Resolution step
➤ YN0002: │ @agoric/wallet@workspace:packages/wallet doesn't provide eslint (p148d7), requested by babel-eslint
➤ YN0002: │ @agoric/wallet@workspace:packages/wallet doesn't provide eslint (pcc945), requested by eslint-plugin-eslint-comments
➤ YN0002: │ @endo/eslint-plugin@npm:2.0.0 doesn't provide eslint (pa1d77), requested by @typescript-eslint/utils
➤ YN0002: │ @endo/pass-style@npm:0.1.7 doesn't provide ava (pa91ef), requested by @fast-check/ava
➤ YN0002: │ @endo/pass-style@npm:1.0.1 doesn't provide ava (pf2e1c), requested by @fast-check/ava
➤ YN0002: │ @lerna/version@npm:5.6.2 doesn't provide nx (p20b2d), requested by @nrwl/devkit
➤ YN0060: │ agoric@workspace:packages/agoric-cli provides @yarnpkg/core (pd0bee) with version 3.5.4, which doesn't satisfy what @yarnpkg/cli and some of its descendants request
➤ YN0002: │ type-coverage@npm:2.26.3 doesn't provide typescript (pf744c), requested by type-coverage-core
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 0s 257ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0008: │ @agoric/sdk@workspace:. must be rebuilt because its dependency tree changed
➤ YN0000: └ Completed in 0s 778ms
➤ YN0000: Done with warnings in 1s 368ms

Whereas with v1 (master) it takes ~7s:

❯ time yarn install
yarn install v1.22.19
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
[4/5] 🔗  Linking dependencies...
warning " > @agoric/eslint-config@0.4.0" has incorrect peer dependency "eslint-plugin-jsdoc@^40.1.0".
[5/5] 🔨  Building fresh packages...
$ patch-package
patch-package 6.5.1
Applying patches...
@agoric/wallet-ui@0.1.3-solo.0 ✔
@lerna/conventional-commits@5.6.2 ✔
@lerna/version@5.6.2 ✔
acorn@8.10.0 ✔
ava@5.3.0 ✔
bufferfromfile@0.4.377 ✔
depd@2.0.0 ✔
encoding@0.1.13 ✔
express@4.18.1 ✔
external-editor@3.1.0 ✔
node-fetch@2.6.12 ✔
protobufjs@6.11.3 ✔
readable-stream@3.6.2 ✔
rxjs@7.5.5 ✔
✨  Done in 6.84s.
yarn install  6.40s user 2.47s system 124% cpu 7.107 total

There are other advantages upgrading to v2+:

• workspace protocol (so we don't have to maintain version numbers in all deps)
• patch protocol (so we don't need patch-package)
• constraints (e.g. to enforce layering)
• path to adopt pnpm-style linker (without changing the UI)

[dckc]

yarn build rebuilds everything in our tree, even things where the inputs haven't changed. You referred to this issue as a solution.

Correct. Are you also asking for confirmation of that?

I'm trying to understand how yarn 2 is supposed to help with build performance. I thought you might have a quick pointer to docs on how yarn 2 uses something like make's dependency tracking to avoid work that yarn 1 does.

Check out #8461 (in a new directory avoid borking your current node_modules) and run yarn install a couple times.

OK, so this is about yarn install? I suppose building xsnap is in that pile...

But I'm also concerned about yarn build. Should I expect yarn 2 to help there too?

Then to see what it's like changing between branches,

yeah; what's necessary in that case is a mystery to me...

With v3 it completes in ~1s:

A step that takes 1s is easier than thinking.

There are other advantages upgrading to v2+:

thanks for all the info.

[turadg]

concerned about yarn build. Should I expect yarn 2 to help there too?

Not much. It does add the ability to run a task only in workspaces in topological order and/or only that have changed since some ref. However that won't be new to the repo as Lerna does that now. What it could mean is getting off Lerna if we can get back with yarn+changesets.