Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[rwieruch]

My Storybook project has grown to a larger size and on

"storybook-build": "build-storybook -c .storybook -o dist"

I get the following Stack Trace [0] with a custom Webpack config [1]. It happens on the recent v5.1.4 but also on an older v5.0.x. I guess it isn't related to the Storybook version, but just to the size of my project. What's the recommended strategy to take from here?

1. Code Splitting? If yes, how to apply it in Storybook?
2. Deactivate Source Maps? [2]
3. Increase --max_old_space_size=4096?
4. ???

Happy to receive any hints on how to proceed if running into this issue.

Node: v10.16.0
Machine: MacOS

---

[1] Custom webpack.config.js for Storybook:

module.exports = ({ config }) => {
  config.module.rules.push(
    {
      test: /\.(js|jsx)$/,
      exclude: /node_modules/,
      use: ['babel-loader'],
    },
    {
      test: /stories\.(js|jsx)?$/,
      loaders: [require.resolve('@storybook/addon-storysource/loader')],
      enforce: 'pre',
    },
    // No other file loaders needed, because they are mocked for Jest
  );

  return config;
};

---

[0] Stack Trace:

info @storybook/react v5.1.4
info
info clean outputDir..
info => Copying prebuild dll's..
info => Building manager..
info => Loading manager config..
info => Loading presets
info => Loading custom addons config.
info => Compiling manager..
info => manager built (6.1 s)
info => Building preview..
info => Loading preview config..
info => Loading presets
info => Loading custom webpack config (full-control mode).
info => Using base config because react-scripts is not installed.
info => Compiling preview..
 92% chunk asset optimization TerserPlugin                         a <--- Last few GCs --->

[94004:0x103800000]    45158 ms: Mark-sweep 1356.8 (1440.3) -> 1344.0 (1441.8) MB, 519.1 / 0.0 ms  (average mu = 0.212, current mu = 0.173) allocation failure scavenge might not succeed
[94004:0x103800000]    45783 ms: Mark-sweep 1358.4 (1441.8) -> 1345.8 (1443.8) MB, 571.7 / 0.0 ms  (average mu = 0.152, current mu = 0.086) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x3c46fcedc01d]
Security context: 0x300cb971e681 <JSObject>
    1: getArg [0x300c61786e11] [/Users/mydspr/Developer/Repos/treact-ui/ui-components/node_modules/terser/node_modules/source-map/lib/util.js:~18] [pc=0x3c46fd842c10](this=0x300c61787159 <Object map = 0x300cd40d7341>,aArgs=0x300c3b50eaa1 <Object map = 0x300caa657b81>,aName=0x300c643c29e9 <String[4]: bias>,aDefaultValue=1)
    2: /* anonymous */(aka /* anonym...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003a08d node::Abort() [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 2: 0x10003a297 node::OnFatalError(char const*, char const*) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 3: 0x1001d2455 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 4: 0x10059d6d2 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 5: 0x1005a01a5 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 6: 0x10059c04f v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 7: 0x10059a224 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 8: 0x1005a6aac v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
 9: 0x1005a6b2f v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
10: 0x1005764b4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
11: 0x1007fe234 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/mydspr/.nvm/versions/node/v10.11.0/bin/node]
12: 0x3c46fcedc01d

---

[2] Deactivate Source Maps?

Works, but shouldn't be the solution, should it?

module.exports = ({ config }) => {
  config.module.rules.push(
    {
      test: /\.(js|jsx)$/,
      exclude: /node_modules/,
      use: ['babel-loader'],
    },
    {
      test: /stories\.(js|jsx)?$/,
      loaders: [require.resolve('@storybook/addon-storysource/loader')],
      enforce: 'pre',
    },
  );

  config.devtool = 'eval';

  return config;
};

Since the build succeeds with this configuration, for the sake of completeness, my bundle size:

WARN asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
WARN This can impact web performance.
WARN Assets:
WARN   main.bc006a5d8742aebd5f8c.bundle.js (1020 KiB)
WARN   vendors~main.bc006a5d8742aebd5f8c.bundle.js (4.05 MiB)
WARN entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
WARN Entrypoints:
WARN   main (5.05 MiB)
WARN       runtime~main.bc006a5d8742aebd5f8c.bundle.js
WARN       vendors~main.bc006a5d8742aebd5f8c.bundle.js
WARN       main.bc006a5d8742aebd5f8c.bundle.js
WARN
WARN webpack performance recommendations:
WARN You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
WARN For more info visit https://webpack.js.org/guides/code-splitting/

[rwieruch]

Maybe related: #6408

[shilman]

This might be an easy win: #7091

[Djiit]

Hi there ! Jumping in to say we encountered the same thing here -- and that deactivating Source Maps worked for us as a workaround.
(5.1.8, React, a bunch off official addons)

[rwieruch]

@shilman you mean finding there is one bad library which causes the issue? I don't think I added some library to my bundle in the past, I just think the bundle got too big over time. Any recommendations on how to apply code splitting here? :)

[shilman]

@rwieruch I mean that fix is going to reduce the bundle size by 5% or so. It's a small thing, but small things add up. I'm hoping we can do a perf sprint at some point soon and improve the situation.

[xcrzx]

Our team encountering the same issue. BTW Upgrading from Node v10 to v12 made it occur less often.

[shilman]

cc @ndelangen

[shilman]

@rwieruch In a separate issue we discovered the source-loader (the successor to addon-storysource/loader) was code that caused the Terser plugin to blow up. Does the problem go away when you disable/remove storysource?

[rwieruch]

@shilman I didn't run into the problem anymore, because I extracted a larger part of the UI library into a separate library (Icons). This fixed the issue for me. Unfortunately I cannot revert to an older version at the moment to verify source-loader for causing the issue.

[ndelangen]

@shilman I think we fixed this in the source-loader, right?

[nicholasbraun]

I'm having the same issue.

    "@storybook/addon-a11y": "^5.1.10",
    "@storybook/addon-actions": "^5.1.10",
    "@storybook/addon-knobs": "^5.1.10",
    "@storybook/addon-links": "^5.1.10",
    "@storybook/addon-viewport": "^5.1.10",
    "@storybook/addons": "^5.1.10",
    "@storybook/react": "^5.1.10",

92% chunk asset optimization����������������������������� 92% chunk asset optimization TerserPluginFATAL ERROR: NewSpace::Rebalance Allocation failed - process out of memory

<--- Last few GCs --->

[12192:0x5597e4ba51b0]    94804 ms: Scavenge 847.2 (902.5) -> 833.5 (904.5) MB, 43.6 / 0.2 ms  allocation failure 
[12192:0x5597e4ba51b0]    95179 ms: Scavenge 851.5 (907.4) -> 837.2 (907.9) MB, 43.3 / 1.5 ms  allocation failure 
[12192:0x5597e4ba51b0]    95537 ms: Scavenge 852.3 (907.9) -> 838.6 (908.9) MB, 83.0 / 0.9 ms  allocation failure 
[12192:0x5597e4ba51b0]    96091 ms: Scavenge 853.5 (908.9) -> 840.0 (908.9) MB, 264.6 / 1.2 ms  allocation failure 


<--- JS stacktrace --->
Cannot get stack trace in GC.
 1: node::Abort() [/usr/bin/node]
 2: 0x5597e4317011 [/usr/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/bin/node]
 5: v8::internal::MarkCompactCollector::Evacuate() [/usr/bin/node]
 6: v8::internal::MarkCompactCollector::CollectGarbage() [/usr/bin/node]
 7: v8::internal::Heap::MarkCompact() [/usr/bin/node]
 8: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/bin/node]
 9: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
10: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/bin/node]
11: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/bin/node]
12: 0x323d615840bd
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Any other solution then extracting some of my stories?

[shilman]

@nicholasbraun Can you try upgrading to 5.2.0-beta.x (using the version "next" will get you the latest) and see if you still have the problem?

[nicholasbraun]

@shilman It works on my local machine, but not on the server where I deploy it to. It has only 2GB of memory.

[shilman]

@nicholasbraun So it's the same behavior on 5.1.10 and 5.2.0-beta.28?

[nicholasbraun]

@shilman yes.

[shilman]

@nicholasbraun Are you using addon-storysource or source-loader? Kind of hard to debug over Github. If you jump onto our Discord maybe we can sort it out together. https://discordapp.com/invite/UUt2PJb

[shilman]

@nicholasbraun and I were able to work around the issue by removing react docgen. I added an issue to fix this properly: #7743

[DanielJRutledge]

Any other tricks to get this working on storybook 5.1.11? I recently updated my build size, and I'm seeing this, despite increasing

1. Increasing node heap size
2. Removing react-docgen from the default babel loader.

My webpack config:

module.exports = async ({config}) => {
  config.module.rules[0].use[0].options.plugins.pop();

  // Return the altered config
  return config;
};

Start-storybook still works fine; disabling the minification altogether would also be an option. Would that be done with
optimization: { minimize: false }?