feat: stop ADO extension reliance on pipeline-installed Node and Yarn versions (by migrating to yarn v3)

[dbjorge]

Details

Leaving this as a draft pending feedback on design + testing as-deployed in a test extension

This PR migrates the repo to yarn v3, both for development of the extension and also for our customers. Notable impacts of this are:

• A copy of yarn 3 is now checked into the repo
• The same copy of yarn 3 is now bundled with the ADO extension and invoked directly. This means that users will no longer need to include any explicit npm install -g yarn step, and it insulates us from issues due to from differences in yarn versions based on customer environment.
• Because of this above, it is easier to invoke our bundled yarn using the same node binary that our script runs within (the node 16 version guaranteed to exist on the agent regardless of user pipeline tasks). This means that users will no longer need to include any explicit NodeTools@0 step, and it insulates us from issues due to differences in Node versions based on customer environment (I suspect this should fix Accessibility-insights for azure devops not working and throw exception #1551).
  • This has the very significant downside of relying on the agent node version to be recent enough to support our task. That's true today, but we've had issues with that in the past - agents didn't add node16 support for tasks until well after we were having trouble with the previously-supported version being EOL and unsupported by many of our dependencies
• We get to drop our lerna dependency in favor of the yarn workspaces plugin, which substantially reduces our overall dependency tree size
• Pre-empts the possibility of package-squatting attacks against workspace-internal packages

Motivation

• Removes a few steps from onboarding
• Prevents several classes of errors we've seen customers hit recently
• Enables more precise yarn resolutions
• Pre-empts package-squatting attacks

Context

Reviewer note: Most of the individual changes are more or less mechanical and not super interesting. However, these three files are substantive changes to the way the extension functions and need deeper review:

• packages/ado-extension/prepare-package-dir.js
• packages/ado-extension/src/install-runtime-dependencies.ts
• packages/shared/build-runtime-package-metadata.js

Some reference material I used while putting together this PR:

• Service repo's yarn v3 migration PR for reference
• Yarn's v1 -> v2+ migration guide that I followed for the first few commits

Pull request checklist

• Addresses an existing issue: Fixes Reduce onboarding complexity by replacing dependency on yarn #1109 and maybe also Accessibility-insights for azure devops not working and throw exception #1551
• [n/a] Added relevant unit test for your changes. (yarn test)
• [n/a] Verified code coverage for the changes made. Check coverage report at: <rootDir>/test-results/unit/coverage
• Ran precheckin (yarn precheckin)

[dbjorge]

js-yaml is here because bundling a working .yarnrc.yml required manipulating our base one a bit at build time. I picked this over other yaml libraries (particularly, yaml) by asking "among widely-used and recently-maintained options, which one do we already have the latest version of in our transitive dependency tree anyway?"

[katydecorah]

@dbjorge Overall, I think this would be a big improvement to the onboarding flow!

A few small questions:

• What does the maintenance of having yarn checked into the repo look like? Should we expect to update this periodically?
• Curiosity question: I noticed that the package.json files now have added devDependencies, was that a result of switching from lerna to yarn 3?
• I think this pull request could remove lerna.json at the root, correct?

[dbjorge]

• What does the maintenance of having yarn checked into the repo look like? Should we expect to update this periodically?

Yes, we should expect to update this periodically, probably at a pace comparable to updating node versions we use.

• Curiosity question: I noticed that the package.json files now have added devDependencies, was that a result of switching from lerna to yarn 3?

This is a result of yarn v2+ being more strict about importing/using dependencies that aren't explicitly listed. Particularly, yarn v2+ doesn't place entries in packages/subpackage/node_modules/.bin for devDependencies that aren't explicitly listed by the subpackage, which breaks most dev tools we use in practice.

• I think this pull request could remove lerna.json at the root, correct?

Yep, good catch, will push an update.

[dbjorge]

Test run against 8e89098. Ran a matrix of test cases (all combinations of hosted agent images ubuntu-18.04, ubuntu-20.04, ubuntu-22.04, windows-2019, windows-2022 each with all combinations of "no explicit node/yarn install steps", "install node 16", "install node 16 + yarn", "install node 18"), all worked as expected.

[katydecorah]

This will be a big onboarding improvement! Nice work, @dbjorge!