Crashes node with JavaScript heap out of memory

[CarterLi]

Bug report or Feature request?

Bug report

Version (complete output of terser -V)

terser 3.10.11

Found on node v10.13.0 (ubuntu); verified on node v11.1.0 (macOS)

Complete CLI command or minify() options used

I'm using terser with terser-webpack-plugin,

const TerserWebpackPlugin = require('terser-webpack-plugin');
minimizer: [
  new TerserWebpackPlugin({
    test: /(vendors|runtime~main).*\.js(\?.*)?$/i
  }),
],

  optimization: {
    runtimeChunk: true,
    splitChunks: {
      cacheGroups: {
        commons: {
          test: /[\\/]node_modules[\\/]/,
          name: 'vendors',
          chunks: 'all',
        },
      }
     }
  }

terser input

Can't provide a minimal test case, but I'm minifying 3rd-party libraries including Angular 7

terser output or error

● Webpack █████████████████████████ additional chunk assets processing (91%)



<--- Last few GCs --->

[41382:0x104002400]   109170 ms: Scavenge 1306.3 (1416.4) -> 1305.5 (1416.4) MB, 6.1 / 0.0 ms  (average mu = 0.217, current mu = 0.137) allocation failure
[41382:0x104002400]   109192 ms: Scavenge 1306.4 (1416.4) -> 1305.6 (1416.4) MB, 7.4 / 0.0 ms  (average mu = 0.217, current mu = 0.137) allocation failure
[41382:0x104002400]   109209 ms: Scavenge 1306.5 (1416.4) -> 1305.7 (1416.4) MB, 6.8 / 0.0 ms  (average mu = 0.217, current mu = 0.137) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x296c1269d949 <JSObject>
    0: builtin exit frame: lastIndexOf(this=0x296c63c11401 <Very long string[2782918]>,0x296c06baa819 <String[1]\: \n>,0x296c63c11401 <Very long string[2782918]>)

    1: L(aka L) [0x296c0a513921] [/Users/Carter/sharplook/node_modules/terser/dist/bundle.js:44] [bytecode=0x296c0a558211 offset=15](this=0x296c309025b1 <undefined>)
    2: prepend_comments [0x296c0a513891] [/Users/Carter/sharplook/no...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100039ef7 node::Abort() [/usr/local/bin/node]
 2: 0x10003a0c8 node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node]
 3: 0x10016e23b v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x10016e1dc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10042e8d8 v8::internal::Heap::UpdateSurvivalStatistics(int) [/usr/local/bin/node]
 6: 0x100430347 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 7: 0x10042dc53 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10042ca39 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x100434937 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x100434984 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x100414d57 v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::PretenureFlag, v8::internal::Map*, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x1004168c5 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
13: 0x1004f4dae v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/bin/node]
14: 0x1005108e7 v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
15: 0x10020870e v8::internal::Builtin_Impl_StringPrototypeLastIndexOf(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
16: 0x804a4acfcdd

Expected result

Don't crash

[CarterLi]

If I replace var _terser = require('terser'); with var _terser = require('terser/tools/node'); inside node_modules/terser-webpack-plugin/dist/minify.js, I get

$ env NODE_ENV=production node_modules/.bin/webpack --config 'webpack/prod.js' --env=prod


● Webpack █████████████████████████ additional chunk assets processing (91%)



<--- Last few GCs --->

[42582:0x104800c00]   121946 ms: Scavenge 1278.7 (1391.5) -> 1277.9 (1392.0) MB, 6.1 / 0.0 ms  (average mu = 0.223, current mu = 0.196) allocation failure
[42582:0x104800c00]   121961 ms: Scavenge 1286.4 (1399.6) -> 1285.5 (1400.1) MB, 4.4 / 0.0 ms  (average mu = 0.223, current mu = 0.196) allocation failure
[42582:0x104800c00]   121980 ms: Scavenge 1301.6 (1415.3) -> 1300.7 (1416.3) MB, 4.0 / 0.0 ms  (average mu = 0.223, current mu = 0.196) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x0e74e801d949 <JSObject>
    0: builtin exit frame: lastIndexOf(this=0x0e74ac014229 <Very long string[3980731]>,0x0e741deaa819 <String[1]\: \n>,0x0e74ac014229 <Very long string[3980731]>)

    1: has_nlb(aka has_nlb) [0xe74a62a7bb1] [0x0e74a6d025b1 <undefined>:6056] [bytecode=0xe7474eaa531 offset=15](this=0x0e74a6d025b1 <undefined>)
    2: prepend_comments [0xe74a62a86b9] [0x0e74a6d025b1 <undefined>:~6060] [pc=0x1af3568...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100039ef7 node::Abort() [/usr/local/bin/node]
 2: 0x10003a0c8 node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node]
 3: 0x10016e23b v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x10016e1dc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10042e8d8 v8::internal::Heap::UpdateSurvivalStatistics(int) [/usr/local/bin/node]
 6: 0x100430347 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 7: 0x10042dc53 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x10042ca39 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x100434937 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x100434984 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x100414d57 v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::PretenureFlag, v8::internal::Map*, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x1004168c5 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
13: 0x1004f4dae v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/local/bin/node]
14: 0x1005108e7 v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
15: 0x10020870e v8::internal::Builtin_Impl_StringPrototypeLastIndexOf(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
16: 0x1af3544cfcdd
17: 0x1af35448e458
18: 0x1af356879d9f
19: 0x1af356871c5a
20: 0x1af3544876a6
fish: 'env NODE_ENV=production node_mo…' terminated by signal SIGABRT (Abort)

lastIndexOf in has_nlb

[kzc]

There's no actionable information here. The issue template was not followed and no terser input was provided for a reproduction.

This is most likely a webpack issue, not a terser issue. Your build was likely near the threshold of being out of memory before. Increase Node memory size or try different options.

Related issue very similar to this report: webpack-contrib/uglifyjs-webpack-plugin#188 (comment)

[fabiosantoscode]

@CarterLi can you try to extract what code is being minified here, and test it yourself against terser? This could be webpack-related.

[CarterLi]

I tested unminified code generated by webpack with terser. It did work.

I'll close this

[kzc]

I tested unminified code generated by webpack with terser. It did work.

No conclusions can be drawn from that. If you were near the Node.JS memory limit, it may work in development mode and not in production mode.

And then there's the issue that webpack generates different code for production and development modes. So it's difficult for its users to obtain unminified production output as a test case for Terser. Webpack really ought to have a CLI flag --no-minify (as Parcel does) which could be used in production mode to easily discern if there's a webpack issue or a minifier issue.

To create a test case for Terser follow these instructions and then run webpack in production mode, redirect stderr to a file, and then post that file into a gist.

[vladimiry]

Out of memory errors occur quite often If you run a relatively heavy build on Travis / Appveyor CI with free/open-source plans. Here is another relative issue webpack-contrib/terser-webpack-plugin#22.

[dep-deprecated]

I know this isn't a terser-js issue per se, but @CarterLi did you land on any solutions here? I'm having the same issue.

[CarterLi]

@dep I split large files into smaller chunks

https://webpack.js.org/plugins/split-chunks-plugin/#optimization-splitchunks

[dep-deprecated]

@CarterLi Thank you so much. This worked!

[fabiosantoscode]

@dep could you provide a minimal example of this crash? Then I can reopen the issue.

[dep-deprecated]

It ended up being a node memory issue, I fixed it by scoping a param in to node for my build job, like so:

package.json:

"scripts": {
  "build": "node --max-old-space-size=4069 scripts/build.js",
}

Leaving that here for posterity.

[vladimiry]

I fixed it by scoping a param in to node for my build job

This is not a fix but workaround.

[ngprnk]

this is not an issue in development mode, this arises in production mode, when it is doing minifying process, not sure if this issued should be closed.

[steven87vt]

@dep I split large files into smaller chunks

https://webpack.js.org/plugins/split-chunks-plugin/#optimization-splitchunks

As just an FYI, the OP stated they "split large files into smaller chunks". I used the following property to dial in what the minimum chunk size was for my app:
https://webpack.js.org/plugins/split-chunks-plugin/#splitchunksmaxsize

After dialing in the precise size requirements down to the nearest hundred bytes, webpack produced a set of chunked .js files which I could then manually analyze. The largest being the culprit in my specific case.

Thanks you OP @CarterLi for rasing and solving the issue.

[dadamssg]

@steven87vt could you describe what the "culprit" was? and how you ultimately fixed that?

[steven87vt]

So my short term fix was to set the optimizations.splitChunks.max size option. The result of this is more chunked production files for download.

The "cluprit" in my case is 8 different developers blazing through spa development not applying best practise to js imports (and the additions to lodash chaining).

The long term, more detailed fix is using a bundle analyzer to effectively target your applications needs, such as https://github.com/webpack-contrib/webpack-bundle-analyzer/blob/master/README.md

[trusktr]

My experience with UglifyJS and now with Terser, is that they usually crash my production builds.

I can build huge C/C++ libs (even if it takes hours) on my dinky 8GB RAM i5 laptop, yet JS minification stuff crashes after it reaches memory limits (usually after 10-15 minutes in my current cases).

On a MacBook pro, my production build needs 10GB of memory to succeed minification. My daily driver only has 8GB of RAM.

So it makes me think: how do decades-old C/C++ compilers handle large code transformation? I think there's something we can learn from them, and that there's room for improvement here, like re-factoring the minifier tool to use an async-chunked method of operation capable of saving chunks on disk (or something, I'm just imagining).

With this thought, I believe this issue (or a similar one) desires to be open.

[L2jLiga]

@trusktr feel free to open new issue, btw issues is one of ways to improve Terser :)

[fabiosantoscode]

There's no need. I'm working on memory usage already. But it's easier said than done, and it involves a separate project. You can't really stream a javascript file into an AST, and then cross-reference variables, inline, drop dead code, and keep mutating it in multiple passes. Especially if the project is really complex and hasn't been built to do that in the first place.

[ycjcl868]

+1

[fabiosantoscode]

Due to using the new source-map package, the latest Terser version is a bit more resistant to this issue.