[RFC] Persistent build cache by default

[alan-agius4]

Author: Alan Agius (@alan-agius4)
Status: Open
Closing Date: 2021-08-24

Summary

In Angular CLI version 12.1, we introduced an experimental opt-in feature to persist build information to disk. This under the hood uses Webpack’s persistent cache feature.

Users can opt-in to the experimental build persistent cache using the NG_PERSISTENT_BUILD_CACHE=1 environment variable.

NG_PERSISTENT_BUILD_CACHE=1 ng <serve|test|build|...>

When opting to use this feature, information of the current build is persisted on disk, and reused in the subsequent reinitialization of the build pipeline via ng build, ng serve and other ng commands. The improvement in cold build times we've seen in the real applications is up to 68%. This does come at the cost of an increase in memory usage.

Proposal

We are proposing that in version 13, we enable persistent build cache by default, whilst still providing an option to opt-out. We also intend to introduce a new command ng cache which would facilitate enabling, disabling and purging the cache.

The persistent build cache typically speeds up local development workflows, because these workflows often depend on performing full, non-incremental builds.

However we found the impact negatively in CI environments because of the increase in memory usage and the cache which is located in node_modules/.cache/angular is typically not stored after each build and restored before each one. For this reason we propose that persistent build cache is disabled when the build is running in a CI environment. We can determine this by checking for the presence of the CI environment variable.

Community feedback

We’d like the community to weigh in and provide feedback on a number of points.

• How NG_PERSISTENT_BUILD_CACHE affects the development when building, serving and testing an application.

• If any, what is the observed memory increase.

Non Windows users can run the below to gather this data:

rm -rf node_modules/.cache
/usr/bin/time NG_PERSISTENT_BUILD_CACHE=0 ng <serve|test|build|...>
rm -rf node_modules/.cache
/usr/bin/time NG_PERSISTENT_BUILD_CACHE=1 ng <serve|test|build|...>

• Should it be possible to provide an alternate cache path, where possibly it could be leveraged on CI, to restore the cache directory before each run. node_modues/.cache is currently used as cache location.

[oguzhanogreden]

However we found the impact negatively in CI environments because of the increase in memory usage and the cache which is located in node_modules/.cache/angular is typically not stored after each build and restored before each one.

Notes:

• I'm not well aware of CI systems (and people who manage these systems). So I may be misjudging several things, including the negative impact you're talking about.
• First time commentor, let me know if anything's off.

I feel like this would reduce usage of the technology you're introducing. Wouldn't it be better to provide good documentation on this point? Would you propose defaulting to using persistent build caches in CI if only, you could assume the build pipelines would make good use of your technology?

In my (admittedly little) experience, when Angular team provides good documentation, the community follows. The community will either be using this directly, or requesting from other teams (e.g. a "devops team"). In both cases, they can be guided by documentation.

Practically, I think it's worth considering:

• default to persistent build caches without a built-in exception for CI environments
• communicate "stores/restore these folders in your build environment" instead of communicating "set environment variable in build environment".

If this works, you would have placed a default successfully that is a net positive for cloud costs (= reduced time + no negative impact), instead of (i) too easily letting people opt out of a potentially cost reducing technology or (ii) people potentially incorrectly using the build settings and not opting out, and neither using the tecnology, therefore experience cost increases.

[SanderElias]

I love to see this turned on by default.
Yes, I would like to be able to move the folder to a different place, as that can make it easier to store than in CI/CD settings.
Also, I would love to have those settings to be easily managed by the CLI itself.
So, here is my wishlist in no particular order:

• ng cache clear Clear out the cache
• ng cache statistics Gives the size, location and so
• ng cache folder shows the folder it is using
• ng cache folder /some/folder sets the folder used for caching
• ng cache off turns off caching
• ng cache on turns on caching

By adding those as commands to the CLI it becomes much easier to control for people not versed in DevOps.

[e-oz]

I vote for making it turned on by default but with the setting in angular.json to turn it off (similar to “buildOptimization: true”). This way, “production” and “ci” configurations could turn it off if needed, while dev builds will run significantly faster (and it's where it matters the most).

[yharaskrik]

I love this idea and definitely echo what @e-oz and @SanderElias mentioned. Having support for tools and configuration would be the only hurdle I see in enabling it by default. Also have clear documentation on how it works and how to configure it.

[ofirrifo]

@alan-agius4 Is a faster build work with Github Actions when I use NG_PERSISTENT_BUILD_CACHE=1 ng build ?

[alan-agius4]

@alan-agius4 Is a faster build work with Github Actions when I use NG_PERSISTENT_BUILD_CACHE=1 ng build ?

It depends how your Github action is configured and which paths are being cached and restored between each run.

[terencehonles]

I'm currently using effectively CI=1 NG_BUILD_CACHE=/ci-persisted-dir NG_PERSISTENT_BUILD_CACHE=1 ng build and I would like to be able to continue to use that (or something similar). I would like to be able to set the cache directory based on the environment, and an ENV VAR or command line option would accomplish that for me.

[ocombe]

If you turn this on by default, please add a config option in angular.json (build-angular schema) instead of a CLI parameter, it's much easier to use, especially when we call the CLI api programmatically, and it's easier to know that this option exists with the schema autocomplete.

[vash72]

not sure if changing the default behaviour is a good option, someone using the default config can get in trouble without knowing, i'd like more to have an hint when using the "ng serve/test/build" command

[Den-dp]

I like the idea of smart defaults, which effectively means more users will get faster rebuilds.

On my small projects, I didn't find any significant increase in memory/disk usage - so I'd leave this enabled even for CI's (at least this should fit well with TeamCity since it reuses working directory between builds).

Also, I found that NG_BUILD_CACHE variable already helps with moving the cache, that's nice.

---

And for those who are curious about how fast builds will be with persistent build cache, here are my numbers for a small-sized project

Clean build looks slightly slower but all others - x2 faster.

For serves all just good

[mattlewis92]

We've been using NG_PERSISTENT_BUILD_CACHE=1 both to speed up our local builds and CI, and it shaves off a solid 2.5m of CI time on our large (~1300 components) app, as well as makes local dev start times faster 🥳 The only problem we've been running into is that occasionally the cache gets corrupted and we start getting cryptic build failures like this:

./libs/core/shared-components/src/lib/doc-page-avatar/doc-page-avatar.component.ts:8:0-69 - Error: Module not found: Error: Can't resolve '../../../../../common/pipes/src/lib/trust-html' in '/home/circleci/repo/libs/core/shared-components/src/lib/doc-page-avatar'

This normally seems to be triggered by massive refactors of lots of files being moved around at once (in that example above the file it cannot find was moved to another directory). If you know to just clear the cache to fix it, then it's a simple solution, but if it's on by default it may cause some confusion to users and end up with a lot of issues reported to the CLI team. Maybe logging a hint or something at the end to try clearing the cache first could help with that.

Another minor issue we ran into was when npm hoisting issues cause the main webpack version in node_modules not to match the webpack version used by the angular CLI (for example by having @storybook/angular installed). When this happens you get lots cryptic of warnings from webpack like this:

[webpack.cache.PackFileCacheStrategy] Skipped not serializable cache item 'Compilation/modules|/home/circleci/repo/node_modules/@angular-devkit/build-angular/src/babel/webpack-loader.js??ruleSet[1].rules[2].use[0]!/home/circleci/repo/node_modules/@ngtools/webpack/src/ivy/index.js!/home/circleci/repo/libs/settings/core/src/lib/profile/profile-settings.component.ts': No serializer registered for ConstDependency
<w> while serializing webpack/lib/cache/PackFileCacheStrategy.PackContentItems -> webpack/lib/NormalModule -> Array { 57 items } -> ConstDependency

This could be solved with a simple check comparing the webpack version in node_modules/webpack/package.json with the one in node_modules/@angular-devkit/build-angular/node_modules/webpack/package.json. If it doesn't exist then log a warning to the console. Unfortunately the solution for the user is a bit complex, the only way I've found to resolve it is to run npm i -D webpack@<version used by the cli>, npm dedupe, npm uninstall webpack. Even clearing out node_modules and package-lock.json sometimes won't make it hoist the correct webpack version.

p.s. all these perf changes to the CLI have made an enormous difference to our CI times and productivity. When we first upgraded to 11.0 our build times were around ~20 minutes, and now with the latest 12.2 we're close to ~5 minutes. So thank you so much for making this a priority, it really does make a huge difference 😄

[alan-agius4]

This RFC is now closed, thank you all for participating. We'll review all the comments in the coming days, Overall though it seems that the community saw good improvements when turning persistent build cache including on CI environments.