Node crashes when building default theme - "JavaScript heap out of memory"

[zhangyx1998]

Describe the bug

I am trying to convert X.org documentation to a VitePress site.

The volume of content is very large and I ended up generating a huge sidar object, that is about 12,430 lines when formatted. The serialized sidebar object is available here: sidebar.json

The vitepress dev server can handle it pretty well. But when I try to bundle it, the build command will stuck at "rendering pages", and then crash after a few minutes. I've attached the full crash log in "additional context".

Cause of the problem

I have verified the problem is introduced by the sidebar object because as soon as I exclude the sidebar object from the config object exported by config.mjs, the problem went away.

My thoughts

Although the sidebar object is huge, I do not think it is large enough to crash Node.js. I am assuming the sidebar component of the default theme ran into circular reference when parsing the sidebar.

In addition, I have ruled out the possibility of a circular reference in my exported sidebar object by using JSON.parse(JSON.stringify(sidebar)).

Reproduction

This problem cannot be reproduced without generating actual ".md" sources.

1. Clone and initialize the project

git clone https://github.com/zhangyx-lab/x-doc.git -b bug-reproduce --depth 1
cd x-doc && npm install

Optional: Download cached resources so the next command do not need to fetch from x.org

Run following commands under x-doc/:

wget https://github.com/zhangyx-lab/x-doc/releases/download/v1.0-beta.0/caches.tar.gz && \
tar -xzf caches.tar.gz

2. Run "setup" to generate markdown source tree

Warning

This command takes a few minutes and will occupy all your CPUs

node setup

3. Now you can verify the project under dev server (should work fine):

npm run dev # You can look around and the sidebar should be rendered correctly

4. And reproduce the problem by building for distribution:

npm run build # Will crash

Expected behavior

Should build just fine, since dev server worked fine.

System Info

System:
  OS: macOS 14.0
  CPU: (10) arm64 Apple M1 Max
  Memory: 65.84 MB / 32.00 GB
  Shell: 5.9 - /bin/zsh
Binaries:
  Node: 20.8.0 - /opt/homebrew/bin/node
  npm: 10.1.0 - /opt/homebrew/bin/npm
  pnpm: 8.10.3 - ~/Library/pnpm/pnpm
Browsers:
  Chrome: 120.0.6099.109
  Safari: 17.0
npmPackages:
  vitepress: ^1.0.0-rc.32 => 1.0.0-rc.32

Additional context

Crash scene (command: npm run build)

> prebuild
> mkdir -p var


> build
> vitepress build docs


  vitepress v1.0.0-rc.32

- building client + server bundles...

(!) Some chunks are larger than 500 kB after minification. Consider:
- Using dynamic import() to code-split the application
- Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/configuration-options/#output-manualchunks
- Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.
✓ building client + server bundles...
- rendering pages...

<--- Last few GCs --->

[87992:0x130008000]   147271 ms: Scavenge 4000.3 (4126.8) -> 3995.5 (4126.8) MB, 17.46 / 0.00 ms  (average mu = 0.344, current mu = 0.336) allocation failure; 
[87992:0x130008000]   147290 ms: Scavenge 4006.3 (4126.8) -> 4001.8 (4127.0) MB, 9.62 / 0.00 ms  (average mu = 0.344, current mu = 0.336) allocation failure; 
[87992:0x130008000]   147371 ms: Scavenge 4012.7 (4127.5) -> 4008.2 (4133.0) MB, 68.54 / 0.00 ms  (average mu = 0.344, current mu = 0.336) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x102fa3a04 node::Abort() [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 2: 0x102fa4d58 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 3: 0x103111898 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 4: 0x103111848 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 5: 0x1032a4134 v8::internal::Heap::CallGCPrologueCallbacks(v8::GCType, v8::GCCallbackFlags, v8::internal::GCTracer::Scope::ScopeId) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 6: 0x1032a6b78 v8::internal::Heap::ComputeMutatorUtilization(char const*, double, double) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 7: 0x1032a4dd8 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 8: 0x1032a2c64 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
 9: 0x10329ac24 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
10: 0x10329b390 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
11: 0x10328428c v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
12: 0x10358fcc0 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
13: 0x102de0c44 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/opt/homebrew/Cellar/node/20.8.0/bin/node]
14: 0x10ad70438 
15: 0x10b7ea7e8 
16: 0x10b8f2430 
17: 0x10b3a89c8 
18: 0x10ad70150 
19: 0x10b7ea7e8 
20: 0x10b790a18 
21: 0x10acda120 
22: 0x10b6e589c 
23: 0x10b562ad4 
24: 0x10b7eb38c 
25: 0x10b70be14 
26: 0x10b7eb150 
27: 0x10b70be14 
28: 0x10b562b68 
29: 0x10b790a5c 
30: 0x10acda120 
31: 0x10b09f100 
32: 0x10b971b80 
33: 0x10b528d18 
34: 0x10b738c3c 
35: 0x10acda120 
36: 0x10b09f100 
37: 0x10b749ea8 
38: 0x10acda120 
39: 0x10b6e589c 
40: 0x10b562ad4 
41: 0x10acda5c0 
42: 0x10b6e589c 
43: 0x10b562ad4 
44: 0x10acda3d8 
45: 0x10b6e589c 
46: 0x10b6e0fb0 
47: 0x10b7555e0 
48: 0x102d8f210 Builtins_AsyncFunctionAwaitResolveClosure [/opt/homebrew/Cellar/node/20.8.0/bin/node]
49: 0x102e3cfb8 Builtins_PromiseFulfillReactionJob [/opt/homebrew/Cellar/node/20.8.0/bin/node]
50: 0x102d7eb94 Builtins_RunMicrotasks [/opt/homebrew/Cellar/node/20.8.0/bin/node]
51: 0x102d563f4 Builtins_JSRunMicrotasksEntry [/opt/homebrew/Cellar/node/20.8.0/bin/node]
52: 0x1032308e4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
53: 0x103230d74 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
54: 0x103251d80 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
55: 0x103251bb4 v8::internal::MicrotaskQueue::PerformCheckpointInternal(v8::Isolate*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
56: 0x102ed0bd4 node::InternalCallbackScope::Close() [/opt/homebrew/Cellar/node/20.8.0/bin/node]
57: 0x102fab36c node::fs::FileHandle::CloseReq::Resolve() [/opt/homebrew/Cellar/node/20.8.0/bin/node]
58: 0x102fabe14 node::fs::FileHandle::ClosePromise()::$_0::__invoke(uv_fs_s*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
59: 0x102f9ce64 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
60: 0x105f8b360 uv__work_done [/opt/homebrew/Cellar/libuv/1.46.0/lib/libuv.1.dylib]
61: 0x105f8e994 uv__async_io [/opt/homebrew/Cellar/libuv/1.46.0/lib/libuv.1.dylib]
62: 0x105f9e294 uv__io_poll [/opt/homebrew/Cellar/libuv/1.46.0/lib/libuv.1.dylib]
63: 0x105f8ee28 uv_run [/opt/homebrew/Cellar/libuv/1.46.0/lib/libuv.1.dylib]
64: 0x102ed1a40 node::SpinEventLoopInternal(node::Environment*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
65: 0x102feadfc node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
66: 0x102feab54 node::NodeMainInstance::Run() [/opt/homebrew/Cellar/node/20.8.0/bin/node]
67: 0x102f6c888 node::Start(int, char**) [/opt/homebrew/Cellar/node/20.8.0/bin/node]
68: 0x181cc5058 start [/usr/lib/dyld]

Validations

• Check if you're on the latest VitePress version.
• Follow our Code of Conduct
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

[zhangyx1998]

I've identified the problem here: build.ts:109

I have fixed it using a pooling mechanism. I will provide a PR shortly.

[brc-dd]

I guess #3285 should fix this?

[zhangyx1998]

I guess #3285 should fix this?

I've read the code. It's the same issue, but I think I can provide a solution that is more efficient.

In addition, I will try to make "thread count" available for user config.

Here is a preview: (pool.ts is added to ../utils)

/**
 * Since this pool is not a thread pool, we can give it a "thread count"
 * larger than the actual number of CPUs available.
 * ---
 * https://github.com/vuejs/vitepress/issues/3362
 * This fix will defer the memory allocation of render tasks.
 * ---
 * By limiting the max number of dispatched promises, it allows NodeJS to
 * reclaim memory associated to each promise, avoiding memory allocation
 * failures when handling large volume of pages.
 * ---
 * Fixes: #3362
 */
const renderPageTaskPool = new Pool(64);
// Pool all rendering tasks
for (const page of ['404.md', ...siteConfig.pages])
  renderPageTaskPool.add(() => renderPage(
    render,
    siteConfig,
    siteConfig.rewrites.map[page] || page,
    clientResult,
    appChunk,
    cssChunk,
    assets,
    pageToHashMap,
    metadataScript,
    additionalHeadTags
  ));
// Wait for all rendering tasks to finish
await renderPageTaskPool.drain();

Will you consider it if I file a PR?

[brc-dd]

Will you consider it if I file a PR?

Yeah sure!