ci: share data between jobs using workspaces

[filipesilva]

PR Checklist

Please check if your PR fulfills the following requirements:

• The commit message follows our guidelines: https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no api changes)
• Build related changes
• CI related changes
• Documentation content changes
• angular.io application / infrastructure changes
• Other... Please describe:

What is the new behavior?

This PR expands usage of the CircleCI workspaces feature to share artifacts between jobs of the same workflow.

The current mindset WRT to artifacts is that we cache some artifacts at the end of the build-npm-packages job, which is then restored on each job. This works well even for jobs prior to that specific cache because of partial cache hits.

With workspaces the mindset is slightly different. There is an initial job that creates a workspace from the cache, which other jobs use and might enhance.

The key difference is that subsequent jobs do not repeat setup, which can yield different results since part of setup is merging a PR with latest master. Another desirable characteristic of this setup is that it enables easier sharing of artifacts between jobs, and lessens repetition in the circleci configuration.

Here is a sample of comparison of workflows taken roughly at the same time. It's important to capture this comparison at similar times because CircleCI workload varies during the day:

Before: https://circleci.com/workflow-run/155fc8ea-2ea5-45ec-8f75-05f3297f4052
After: https://circleci.com/workflow-run/97e4f222-75f3-426c-bcdf-a583f1524045

Of these we can compare specific jobs:

• lint
  • before https://circleci.com/gh/angular/angular/218672
  • after https://circleci.com/gh/angular/angular/218906
  • the checkout/post checkout/restoring cache/yarn install steps (total 51s) are replaced by the single attaching workspace step that takes 15s
• build-ivy-npm-packages
  • before https://circleci.com/gh/angular/angular/218673
  • after https://circleci.com/gh/angular/angular/218901
  • the checkout/post checkout/restoring cache/yarn install steps (total 27s) are replaced by the single attaching workspace step that takes 12s

The gains in individual job setup are lessened by the new common install step though. In https://circleci.com/gh/angular/angular/218899 we can see it took 1m58s.

Of these, 27s seem to be due to yarn behaviour where --frozen-lockfile with a cached directory can still fetch dependencies in some cases (yarnpkg/yarn#5840 (comment)). Updating the AIO lockfile should reduce it to ~2s.

The gain on correctness cannot be compared. Under the previous setup it was possible for individual jobs to run over different code.

Further additions to persisted workspace files can also improve performance of subsequent runs.

It's also important to note that this PR also adds caching of AIO node_modules, which wasn't done before.

Does this PR introduce a breaking change?

• Yes
• No

Other information

[IgorMinar]

the rest looks great. we can rename the name in a follow up PR. I want this change now because it's so good ! :-)

[IgorMinar]

is install the right name? how about init or setup or prepare? we do much more than just "install"

[IgorMinar]

merge-assistance: if this fails to cherry-pick to patch and the conflict can't be easily resolved, please merge it to master-only and ask @filipesilva to send a separate PR for the patch branch. thanks!

[gkalpak]

I think it would be a good idea to capture in the commit message (and PR description) why we did this change (and why "it is so good" 😛).
Also, how this affects total CI time (afaict, it increases the total time for certain jobs) and why the benefits justify any increase in total workflow duration.

[filipesilva]

@gkalpak I am a bit late to change the commits but added details and comparison to the PR message.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}