CI build fails on master for current and previously successful builds

[danieljurek]

• Package Name: n/a
• Package Version: n/a (repros on master)
• Operating system:
• nodejs
  • version:
• browser
  • name/version:
• typescript
  • version:
• Is the bug related to documentation in
  • README.md
  • source code documentation
  • SDK API docs on https://docs.microsoft.com

Describe the bug
Running local or CI builds for master are failing consistently after yesterday afternoon (Pacific time).

The first failure we saw in master happened at this build: https://dev.azure.com/azure-sdk/public/_build/results?buildId=42923

Some of the logs from the "Build libraries" step:

[@azure/template] started
>>> @azure/template
tsc -p . && rollup -c 2>&1 
[@azure/keyvault] started

dist-esm/src/index.js → dist/index.js...
created dist/index.js in 45ms

dist-esm/src/index.js → browser/azure-template.js...
[!] (Rollup Core plugin) Error: Could not load events (imported by /home/vsts/work/1/s/sdk/template/template/dist-esm/src/index.js): ENOENT: no such file or directory, open 'events'
Error: Could not load events (imported by /home/vsts/work/1/s/sdk/template/template/dist-esm/src/index.js): ENOENT: no such file or directory, open 'events'
    at Object.openSync (fs.js:443:3)
    at Object.readFileSync (fs.js:343:35)
    at Object.load (/home/vsts/work/1/s/common/temp/node_modules/.registry.npmjs.org/rollup/1.13.1/node_modules/rollup/dist/rollup.js:11224:23)
    at Promise.resolve.then (/home/vsts/work/1/s/common/temp/node_modules/.registry.npmjs.org/rollup/1.13.1/node_modules/rollup/dist/rollup.js:15429:25)


0 of 11: [@azure/template] failed to build!

Looks as if rollup cannot find events and in other packages, buffer

A build from an earlier commit SHA which succeeded in the past (successful build: https://dev.azure.com/azure-sdk/public/_build/results?buildId=42761) now fails when built after this time (failing build: https://dev.azure.com/azure-sdk/public/_build/results?buildId=43082)... The failed build produces a similar error message.

To Reproduce
Steps to reproduce the behavior:

1. Clone the repository (optionally set head to a commit SHA for a known good commit (e.g. 6687005)
2. Install required packages (rush install)
3. Build (rush build --verbose)

Expected behavior
Build succeeds without errors

Screenshots
Build fails with errors

Additional context
We have tried a few things and have not solved the build problems:

• Different dev machines succeed and fail (one developer's repo does successfully build) ... This rules out DevOps as the only source of the issue
• Building in a node:lts docker container on a fresh clone of the repo does not succeed
• Diffing the dependencies from the install step shows no differences between passing and failing builds using the same SHA

[danieljurek]

#3456 mitigated this issue

[mikeharder]

The root cause of this specific issue was that a new version of a transitive dependency (which contained breaking changes for our repo) was published to npmjs.com. We assumed that all dependency versions (including transitive) were pinned via the Rush/PNPM pnpm-lock.yaml, but this was not actually the case. We're working with the Rush team on next steps (microsoft/rushstack#1142).