Update Schematics to version 8 fails with Out-of-Memory exception

[ComradeCow]

Reproduction

Steps to reproduce:
0. Start with a "large" Angular v7 workspace

1. Update @angular/core etc to v8
2. Run ng update @angular/material

"Large" is a relative term, of course. We're using a monorepo style workspace (using the Nrwl's NX tools) for our Angular Applications. We have 390 *.component.ts and *.directive.ts files.

Expected Behavior

The update to successfully run the upgrade schematics.

Actual Behavior

After it installs the @angular/material and @angular/cdk packages this is displayed to the terminal:

    ** Executing migrations for package '@angular/cdk' **

<--- Last few GCs --->

[15728:0000017D08BC7A90]   978327 ms: Mark-sweep 2016.3 (2069.9) -> 2000.2 (2069.9) MB, 772.3 / 0.0 ms  (average mu = 0.273, current mu = 0.240) allocation failure scavenge might not succeed
[15728:0000017D08BC7A90]   979306 ms: Mark-sweep 2016.2 (2069.9) -> 2000.4 (2070.4) MB, 720.2 / 0.0 ms  (average mu = 0.269, current mu = 0.264) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF600E82C16]
Security context: 0x03f554a193e1 <JSObject>
    1: createScanner [000003B48610A701] [C:\git\Daktronics-Web-Platform\node_modules\typescript\lib\typescript.js:~7066] [pc=000002E34F358EE6](this=0x01dc41080509 <Object map = 0000028D62B90EE9>,1,0x02b136100709 <false>,0,0x03ce79b0e971 <String[425]\: `.control-container {\r\n  background: linear-gradient(var(--background-standard), var(--background-standa...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20190529.132332.15728.0.001.json
Node.js report completed
 1: 00007FF60033674F napi_wrap+120927
 2: 00007FF6002DFE26 uv_loop_fork+44550
 3: 00007FF6002E0CC6 uv_loop_fork+48294
 4: 00007FF600618BCE private: void __cdecl v8::internal::Parser::ReportMessageAt(struct v8::internal::Scanner::Location,enum v8::internal::MessageTemplate,class v8::internal::AstRawString const * __ptr64,enum v8::internal::ParseErrorType) __ptr64+206
 5: 00007FF6005ECC4A public: void __cdecl v8::internal::wasm::StreamingDecoder::NotifyCompilationEnded(void) __ptr64+666
 6: 00007FF60068309C public: class v8::internal::HeapObject __cdecl v8::internal::Heap::CreateFillerObjectAt(unsigned __int64,int,enum v8::internal::ClearRecordedSlots,enum v8::internal::ClearFreedMemoryMode) __ptr64+2972
 7: 00007FF60068AE0F public: void __cdecl v8::internal::LocalEmbedderHeapTracer::NotifyV8MarkingWorklistWasEmpty(void) __ptr64+5375
 8: 00007FF600681483 public: bool __cdecl v8::internal::Heap::CollectGarbage(enum v8::internal::AllocationSpace,enum v8::internal::GarbageCollectionReason,enum v8::GCCallbackFlags) __ptr64+1235
 9: 00007FF60067FB84 public: void __cdecl v8::internal::GCTracer::AddScopeSample(enum v8::internal::GCTracer::Scope::ScopeId,double) __ptr64+1316
10: 00007FF6006C07B3 public: class v8::internal::Handle<class v8::internal::HeapObject> __cdecl v8::internal::Factory::NewFillerObject(int,bool,enum v8::internal::AllocationSpace) __ptr64+67
11: 00007FF6009E76E6 public: void __cdecl v8::internal::OptimizingCompileDispatcher::Unblock(void) __ptr64+116966
12: 00007FF600E82C16 public: static class v8::internal::Vector<char const > __cdecl v8::internal::NativesCollection<0>::GetScriptsSource(void)+662454

I uploaded the report.json to a Gist here.

This ends up running for a long time. The NodeJS process continually increases memory usage until it crosses the 2GB threshold and then fails out. Unfortunately I can't share link to a reproducing repository.

I also tried running this on an old Mac laptop we have in the office (late 2011), and it failed with a similar Out of Memory error.

Since I can't upload a reproducing repo, please let me know if there's any other information that can help!

Environment

Angular CLI: 8.0.0
Node: 12.3.1
OS: win32 x64
Angular: 8.0.0
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, language-service, material, platform-browser
... platform-browser-dynamic, router

Package Version

@angular-devkit/architect 0.800.0
@angular-devkit/build-angular 0.800.0
@angular-devkit/build-ng-packagr 0.800.0
@angular-devkit/build-optimizer 0.800.0
@angular-devkit/build-webpack 0.800.0
@angular-devkit/core 8.0.0
@angular-devkit/schematics 8.0.0
@ngtools/json-schema 1.1.0
@ngtools/webpack 8.0.0
@schematics/angular 8.0.0
@schematics/update 0.800.0
ng-packagr 5.2.0
rxjs 6.5.2
typescript 3.4.5
webpack 4.30.0

[jymdman]

There are a couple of issues open in @angular/cli regarding memory issues.
Try to increase the memory limit for node like here angular/cli#14543

[ComradeCow]

Increasing the memory limit let the schematics finish out.

To save people a click, I ran this command:

node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng update @angular/material

I watched it run for about 1.5 hours before I had to leave at the end of the day. The node.exe process topped out around 5.7GB at one point, and finished with 145 files modified.

Since my issue is solved I'm fine closing out this bug - seems like this is more of an issue with angular/cli?

[intellix]

I had to do the above for @angular/cdk. It took around 10 minutes on 8gb old space and it only changed 2 files after all that compute: package.json, package-lock.json

Similar thing for @angular/material really. It almost takes a minute to scan my TS file to fix:

import { x } from '@angular/material';

to

import { x } from '@angular/material/accordion';

Not sure if it's related to the performance regression mentioned in TS 3.4 but there's definitely something weird. Felt like I needed to send it to IBM Watson to change 1 line in a couple of files of my project

[devversion]

Yeah that is definitely something we need to look into. At first glance this looks related to how the migrations leverage tslint to analyze source-files and perform the replacements.

We still use tslint because when ng update was in the early stages, the pattern for doing replacements was to use TSLint. Especially because the tslint rules could also be used within Google.

In the future.. I definitely want to switch away from tslint and just use the TypeScript compiler API. This gives us a couple of benefits (e.g. global analysis phase; files don't need to be re-checked multiple-times etc.)

[devversion]

Looked more into this and actually inspected what objects are allocated. It's definitely caused by tslint as it re-creates the whole TypeScript program for every updated source-file. This impacts performance significantly as re-creating the program w/ type checker etc. results in ~3seconds per source-file.

Additionally tslint executes all lint rules for a given source-file until there are no remaining fixes. This causes an additional slow-down. Additionally TypeScript 3.4.x seems to be currently affected by memory leaks which can cause OOM exceptions when creating a TypeScript program (e.g. see here)

[Xaz16]

Hi guys
@devversion any updates on this? It is really painful due to a big size of our project :)

[bcarson]

Chiming in with @Xaz16, I'm also running into this issue in a large project.

It's been 18 days, any updates or workarounds you can share?

Thank you @devversion, you're the best! We believe in you!! 💪🏻👍🏻

[devversion]

The workaround is to increase the memory #16154 (comment) unless there are some other known issues with this? I brought this up multiple times in our team meeting but we decided that it's not a priority for us to switch away from TSLint right now as the current focus is other stuff (see README.md). I wish we could switch away from TSLint as soon as possible but it just involves a lot of work and we decided to not prioritize this right now since there is a workaround.