ng build --prod OOM

[buu700]

Versions

OS: Debian Stretch

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 6.0.0-rc.7
Node: 8.11.1
OS: linux x64
Angular: 6.0.0-rc.6
... animations, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic
... platform-server, platform-webworker
... platform-webworker-dynamic, router, service-worker

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.5.10
@angular-devkit/architect-cli      0.5.10
@angular-devkit/build-angular      0.5.10
@angular-devkit/build-ng-packagr   0.5.10
@angular-devkit/build-optimizer    0.5.10
@angular-devkit/build-webpack      0.0.8
@angular-devkit/core               0.5.10
@angular-devkit/schematics         0.5.10
@angular-devkit/schematics-cli     0.5.10
@angular/cdk                       6.0.0-rc.13
@angular/cli                       6.0.0-rc.7
@angular/flex-layout               6.0.0-beta.15
@angular/material                  6.0.0-rc.13
@ngtools/json-schema               1.2.0
@ngtools/webpack                   6.0.0-rc.8
@schematics/angular                0.5.10
@schematics/package-update         0.3.2
@schematics/schematics             0.5.10
@schematics/update                 0.5.10
rxjs                               6.0.0
typescript                         2.7.2
webpack                            4.6.0

Repro steps

• git clone https://github.com/buu700/cyph-ng6-test.git
• cd cyph-ng6-test
• npm install .
• ng build --prod

Observed behavior

                                                                                                                              2                                                                                                                                                                             4                                                                                                                                                                             6                                                                                                                                                                             6                                                                                                                                                                           6                                                                                                                                      92% chunk asset optimization UglifyJSPlugin                                             s
<--- Last few GCs --->

[1712:0x3d4b6e0]   428216 ms: Mark-sweep 1472.2 (1607.6) -> 1472.1 (1607.6) MB, 1500.0 / 0.0 ms  allocation failure GC in old space requested
[1712:0x3d4b6e0]   430995 ms: Mark-sweep 1472.1 (1607.6) -> 1471.9 (1571.6) MB, 2777.9 / 0.0 ms  last resort GC in old space requested
[1712:0x3d4b6e0]   433430 ms: Mark-sweep 1471.9 (1571.6) -> 1471.9 (1568.6) MB, 2433.7 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2edfc9d257c1 <JSObject>
    1: def_variable [0x4d324d822d1 <undefined>:~5199] [pc=0x1c494a884151](this=0xfcaa76a1f81 <AST_Function map = 0x3c3d27db53c9>,symbol=0x10aafa3b15f1 <AST_SymbolVar map = 0x2493e4398469>,init=0x4d324d82201 <null>)
    2: visit [0x4d324d822d1 <undefined>:~4898] [pc=0x1c494a7874f8](this=0xeb7273e1649 <TreeWalker map = 0x3404dd422ac9>,node=0x10aafa3b15f1 <AST_SymbolVar map = 0x2493e4398469>,descen...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [ng]
 2: 0x11e7fec [ng]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [ng]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [ng]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [ng]
 6: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::NewInternal(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [ng]
 7: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag, v8::internal::MinimumCapacity) [ng]
 8: v8::internal::HashTable<v8::internal::StringTable, v8::internal::StringTableShape>::EnsureCapacity(v8::internal::Handle<v8::internal::StringTable>, int, v8::internal::PretenureFlag) [ng]
 9: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handle<v8::internal::String>) [ng]
10: 0x10a21ae [ng]
11: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [ng]
12: 0x1c4948d042fd
Aborted

Desired behavior

Successful build.

Mention any other details that might be useful (optional)

Failure occurs in a Docker container with 6.5 GB RAM allocated.

Building without --prod succeeds , and CLI version 1.7.4 works fine.

[filipesilva]

I'm having a look. Have noticed that Webpack 4 seems to churn a lot on Module Concatenation on larger projects.

Am also aware of some perf problems on node >8.9.4. Can you try with node 8.9.4, or node 10+ as well please?

[filipesilva]

Also, can I use this project for benchmarks? Sometimes I need big projects for perf testing and yours seems good for that purpose.

[filipesilva]

I don't think I can install this project on windows, it seems to fail on building rsa-keygen@1.0.6 with node-gyp.

I did notice your machine errors out at 1.5gb ram though. Are you increasing the default node memory limit?

I do that in my test project here: https://github.com/filipesilva/angular-cli-test-repo/blob/master/package.json#L12

[buu700]

Am also aware of some perf problems on node >8.9.4. Can you try with node 8.9.4, or node 10+ as well please?

Just tested with 8.9.4; looks like the output is about the same.

output.txt

Also, can I use this project for benchmarks? Sometimes I need big projects for perf testing and yours seems good for that purpose.

Cool, sure, not a problem at all. Out of curiosity, are you talking about public benchmarks or just testing for your own benefit? Either way, sounds good to me.

(Only thing to be aware of is that the license of Cyph is Ms-RSL, which allows using the source for debugging/development purposes like you're talking about, but not much else.)

I did notice your machine errors out at 1.5gb ram though. Are you increasing the default node memory limit?

Ah, forgot about that; I'll test that now too.

[buu700]

Not 100% sure yet, but it looks like this will unblock us, so thanks for the suggestion!

In the tests I just ran, it used a bit over 4 GB RAM, but did successfully finish without crashing. The builds themselves failed, but I think the issues are in this test repo rather than in our code itself; I'll run another test in our actual project to confirm.

[buu700]

It succeeded with the actual project with RAM consumption of 4.3 GB and total time of 465863ms (using Node 8.11.1).

Any thoughts on whether this should be considered a bug / is reasonably fixable without harming the output size/performance? A requirement of ~4 GB RAM for prod builds, while not ideal, is basically fine for me personally, although it'd be nice if the CLI could somehow automatically detect when a project is large enough to require this much RAM and handle it gracefully.

Also, not sure offhand what's different in the test project, but if you want to use it for benchmarking then I should be able to find some time to fix whatever's wrong with it.

[filipesilva]

Glad to hear this unblocks you. So this repo did not need as much RAM when built on 1.7.x? There might be a memory regression on webpack. I'm going to keep this issue open but move it to the 6.1.x milestone because I plan on doing more extensive perf benchmarking for that release.

I don't think we can really handle the increased memory requirement gracefully as it is neither known ahead of time, nor is it something that can be configurable on a already running node process. IIRC it can only be set as flags in node when starting the process.

Regarding the use I wanted for the repo: not really public benchmarks but mostly something I run locally when users report problems that don't happen on small applications. So nothing that would put it out there.

Will the github repo you linked remain public, or is it something you made public just for the repro? I ask in case I should make a fork to benchmark stuff later. If I fork and use the project for benchmark I would maintain it myself, so I don't think it would make any difference for you to spend time fixing it yourself.

[buu700]

Cool, all sounds good to me.

So this repo did not need as much RAM when built on 1.7.x? There might be a memory regression on webpack. I'm going to keep this issue open but move it to the 6.1.x milestone because I plan on doing more extensive perf benchmarking for that release.

Oops, sorry, apparently the same command in our master branch (Angular 5 / rxjs 5.5 / CLI 1.7.4) actually does also use a bunch of RAM and crash. I guess I'd just never had a reason to use ng build --prod directly, since for prod builds pre-v6 we were using a custom build script that ejected and split up the output into smaller chunks (#10550).

Will the github repo you linked remain public, or is it something you made public just for the repro? I ask in case I should make a fork to benchmark stuff later. If I fork and use the project for benchmark I would maintain it myself, so I don't think it would make any difference for you to spend time fixing it yourself.

It was just a one-off thing for the repro, but I also didn't have any specific plans to take it down. I'll make one last commit there to fix the build failure (it needs a small patch to fix an issue in a third-party library), then afterwards feel free to fork and maintain it for future benchmarks.

Edit: The latest version of my test project now builds with no errors.

[Enngage]

I'm having similar issue with the release of 6.0.0. Mine looks like:

<--- Last few GCs --->

[27528:00000210A60A45A0]   165099 ms: Mark-sweep 1416.9 (1528.7) -> 1416.6 (1528.2) MB, 1688.5 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 000002A64B425EE1 <JSObject>
    2: checkLVal [c:\dev\FitRockApp\node_modules\acorn\dist\acorn.js:1739] [bytecode=0000013E7AA2C7B9 offset=201](this=000002A70A17F669 <Parser map = 0000024C3F979FC9>,expr=000002578465F411 <Node map = 000001A179CFD931>,bindingType=000000B6B4315849 <String[3]: var>,checkClashes=000002578465F621 <Object map = 000000E7350023B9>)
    4: checkParams [c:\dev\FitRockApp\node_modules\acorn\dist\acorn...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewTransitionArray
 5: v8::internal::EhFrameIterator::DecodeSLeb128
 6: v8::internal::JSReceiver::GetOwnPropertyDescriptor
 7: v8::internal::JSReceiver::GetOwnPropertyDescriptor
 8: v8::internal::JSReceiver::GetOwnPropertyDescriptor
 9: v8::internal::JSReceiver::class_name
10: v8::internal::JSReceiver::GetOwnPropertyDescriptor
11: v8::internal::LookupIterator::PrepareTransitionToDataProperty
12: std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Reallocate
13: std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Reallocate
14: std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Reallocate
15: std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Reallocate
16: std::vector<v8::internal::compiler::MoveOperands * __ptr64,v8::internal::ZoneAllocator<v8::internal::compiler::MoveOperands * __ptr64> >::_Reallocate
17: 00000198269047A1

Any ideas how can I go around this? I have multiple apps in the same repo and the other ones are working quite ok with the aot so it does seem to be related to the size of the project. I have 16GB Ram and around 50% is free when running the build.

[aaronte]

Also running into the same problem:

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/
    

Angular CLI: 6.0.0
Node: 8.11.1
OS: darwin x64
Angular: 6.0.0
... animations, cdk, cli, common, compiler, compiler-cli, core
... forms, http, material, platform-browser
... platform-browser-dynamic, platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.0
@angular-devkit/build-angular     0.6.0
@angular-devkit/build-optimizer   0.6.0
@angular-devkit/core              0.6.0
@angular-devkit/schematics        0.6.0
@angular/flex-layout              5.0.0-beta.14
@angular/language-service         5.1.2
@ngtools/webpack                  6.0.0
@schematics/angular               0.6.0
@schematics/update                0.6.0
rxjs                              6.1.0
typescript                        2.7.2
webpack                           4.6.0

<--- Last few GCs --->

[96332:0x103000000]   175390 ms: Mark-sweep 983.4 (1417.3) -> 983.0 (1405.8) MB, 887.2 / 0.0 ms  allocation failure GC in old space requested
[96332:0x103000000]   176295 ms: Mark-sweep 983.0 (1405.8) -> 982.8 (1328.8) MB, 904.8 / 0.0 ms  last resort GC in old space requested
[96332:0x103000000]   177172 ms: Mark-sweep 982.8 (1328.8) -> 982.8 (1291.3) MB, 876.6 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x20e0493257c1 <JSObject>
    1: from [buffer.js:~175] [pc=0xf79400d3101](this=0x20e075636461 <JSFunction Buffer (sfi = 0x20e04937bba9)>,value=0x20e0fc062651 <Very long string[135937688]>,encodingOrOffset=0x20e049335799 <String[4]: utf8>,length=0x20e02d1022d1 <undefined>)
    2: arguments adaptor frame: 2->3
    3: writeOut(aka writeOut) [/Users/USER/Workspace/web-client/node_modules/webpack/lib/Compiler.js:279] [byte...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 5: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 6: v8::String::WriteUtf8(char*, int, int*, int) const [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 7: node::StringBytes::Write(v8::Isolate*, char*, unsigned long, v8::Local<v8::Value>, node::encoding, int*) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 8: node::Buffer::New(v8::Isolate*, v8::Local<v8::String>, node::encoding) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
 9: node::Buffer::(anonymous namespace)::CreateFromString(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/USER/.nvm/versions/node/v8.11.1/bin/node]
10: 0xf7940090647