ci: publish tarballs for all Angular packages as build artifacts on PR builds (angular#33321)

Previously, when one wanted to try out the changes from a PR before it
was merged, they had to check out the PR locally and build the Angular
packages themselves (which is time-consuming and wasteful given that the
packages have already been built on CI).

This commit persists all Angular packages on each build as `.tgz` files,
which can be used to install dependencies on an project (supported by
both [npm][1] and [yarn][2]). In addition to individual `.tgz` files for
each package, a `.tgz` file including all packages is also stored, which
can be used to test the packages locally by overwriting the ones in the
`node_modules/` directory of a project.

CircleCI [build artifacts][3] an be used for longer-term storage of the
outputs of a build and are designed to be useful around the time of the
build, which suits our needs.

[1]: https://docs.npmjs.com/cli/install.html
[2]: https://yarnpkg.com/lang/en/docs/cli/add
[3]: https://circleci.com/docs/2.0/artifacts

PR Close angular#33321

Author: gkalpak

.circleci/config.yml

@@ -559,6 +559,26 @@ jobs:
       # amount of container nodes for this job is controlled by the "parallelism" option.
       - run: ./integration/run_tests.sh ${CIRCLE_NODE_INDEX} ${CIRCLE_NODE_TOTAL}
 
+  # This job creates compressed tarballs (`.tgz` files) for all Angular packages and stores them as
+  # build artifacts. This makes it easy to try out changes from a PR build for testing purposes.
+  # More info CircleCI build artifacts: https://circleci.com/docs/2.0/artifacts
+  #
+  # NOTE: Currently, this job only runs for PR builds. See `publish_snapshot` for non-PR builds.
+  publish_packages_as_artifacts:
+    executor: default-executor
+    environment:
+      NG_PACKAGES_DIR: &ng_packages_dir 'dist/packages-dist'
+      NG_PACKAGES_ARCHIVES_DIR: &ng_packages_archives_dir 'dist/packages-dist-archives'
+    steps:
+      - custom_attach_workspace
+      - init_environment
+      - run:
+          name: Create artifacts
+          command: ./scripts/ci/create-package-archives.sh $CI_PULL_REQUEST $CI_COMMIT $NG_PACKAGES_DIR $NG_PACKAGES_ARCHIVES_DIR
+      - store_artifacts:
+          path: *ng_packages_archives_dir
+          destination: angular
+
   # This job updates the content of repos like github.com/angular/core-builds
   # for every green build on angular/angular.
   publish_snapshot:
@@ -823,6 +843,11 @@ workflows:
       - integration_test:
           requires:
             - build-npm-packages
+      - publish_packages_as_artifacts:
+          # Only run on PR builds. (For non-PR builds, the `publish_snapshot` job.)
+          <<: *only_on_pull_requests
+          requires:
+            - build-npm-packages
       - publish_snapshot:
           # Note: no filters on this job because we want it to run for all upstream branches
           # We'd really like to filter out pull requests here, but not yet available:

docs/DEVELOPER.md

@@ -139,8 +139,58 @@ automatically publishes build artifacts to repositories in the Angular org, eg.
 package is published to https://github.com/angular/core-builds.
 
 You may find that your un-merged change needs some validation from external participants.
-Rather than requiring them to pull your Pull Request and build Angular locally, you can publish the
-`*-builds` snapshots just like our CircleCI build does.
+Rather than requiring them to pull your Pull Request and build Angular locally, they can depend on
+snapshots of the Angular packages created based on the code in the Pull Request.
+
+### Getting Packages from Build Artifacts
+Each CI run for a Pull Request stores the built Angular packages as
+[build artifacts](https://circleci.com/docs/2.0/artifacts). The artifacts are not guaranteed to be
+available as a long-term distribution mechanism, but they are guaranteed to be available around the
+time of the build.
+
+You can access the artifacts for a specific CI run by going to the workflow page, clicking on the
+`publish_packages_as_artifacts` job and then switching to the "Artifacts" tab.
+(If you happen to know the build number of the job, the URL will be something like:
+https://circleci.com/gh/angular/angular/<build-number>#artifacts)
+
+#### Archives for each Package
+On the "Artifacts" tab, there is a list of links to compressed archives for Angular packages. The
+archive names are of the format `<package-name>-pr<pr-number>-<sha>.tgz` (for example
+`core-pr12345-a1b2c3d.tgz`).
+
+One can use the URL to the `.tgz` file for each package to install them as dependencies in a
+project they need to test the Pull Request changes against. Both
+[npm](https://docs.npmjs.com/cli/install.html) and [yarn](https://yarnpkg.com/lang/en/docs/cli/add)
+support installing dependencies from URLs to `.tgz` files, for example by updating the dependencies
+in `package.json` to point to the artifact URLs and then running `npm/yarn install`:
+
+```json
+"dependencies": {
+  "@angular/common": "https://<...>.circle-artifacts.com/0/angular/common-pr12345-a1b2c3d.tgz",
+  "@angular/core": "https://<...>.circle-artifacts.com/0/angular/core-pr12345-a1b2c3d.tgz",
+  "...": "..."
+}
+```
+
+#### Download all Packages
+In addition to the individual package archives, a `.tgz` file including all packages is also
+available (named `all-pr<pr-number>-<sha>.tgz`). This can be used if one prefers to download all
+packages locally and test them by either of the following ways:
+
+1. Update the dependencies in `package.json` to point to the local uncompressed package directories.
+
+2. Directly copy the local uncompressed package directories into the `node_modules/` directory of a
+   project.
+
+Note that (while faster) the second approach has limitations. For example:
+a. Any transitive dependencies of the copied packages will not be automatically updated.
+b. The packages need to be copied over every time `npm/yarn install` is run.
+c. Some package managers (such as `pnpm` or `yarn pnp`) might not work correctly.
+
+### Publishing to GitHub repos
+You can also manually publish `*-builds` snapshots just like our CircleCI build does for upstream
+builds. Before being able to publish the packages, you need to build them locally by running the
+`./scripts/build-packages-dist.sh` script.
 
 First time, you need to create the GitHub repositories:
 

scripts/ci/create-package-archives.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+
+set -eu -o pipefail
+
+readonly prNumber="$1"
+readonly prLastSha="${2:0:7}"
+readonly inputDir="$PROJECT_ROOT/$3"
+readonly outputDir="$PROJECT_ROOT/$4"
+readonly fileSuffix="-pr$prNumber-$prLastSha.tgz"
+
+echo "Creating compressed archives for packages in '$inputDir'."
+
+# Create or clean-up the output directory.
+echo "  Preparing output directory: $outputDir"
+rm -rf "$outputDir"
+mkdir -p "$outputDir"
+
+# Create a compressed archive containing all packages.
+# (This is useful for copying all packages into `node_modules/` (without changing `package.json`).)
+outputFileName=all$fileSuffix
+echo "  Creating archive with all packages --> '$outputFileName'..."
+tar --create --gzip --directory "$inputDir" --file "$outputDir/$outputFileName" --transform s/^\./packages/ .
+
+# Create a compressed archive for each package.
+# (This is useful for referencing the path/URL to the resulting archive in `package.json`.)
+for dir in $inputDir/*
+do
+  packageName=`basename "$dir"`
+  outputFileName="$packageName$fileSuffix"
+  outputFilePath="$outputDir/$outputFileName"
+
+  echo "  Processing package '$packageName' --> '$outputFileName'..."
+
+  tar --create --gzip --directory "$dir" --file "$outputFilePath" --transform s/^\./package/ .
+done
+
+echo "Done creating compressed archives."