Memory leak when generating many pages

[cyberalien]

Environment

• Operating System: Darwin
• Node Version: v20.5.0
• Nuxt Version: 3.9.3
• CLI Version: 3.10.0
• Nitro Version: 2.8.1
• Package Manager: pnpm@8.14.1
• Builder: -
• User Config: devtools
• Runtime Modules: -
• Build Modules: -

Reproduction

https://github.com/cyberalien/nuxt-memory-test

Describe the bug

When there are many pages, Nuxt generates all pages one after another, memory usage always grows and eventually Node crashes with error about missing memory.

Demo repository has about 65k pages, generates using basic loops:

• main page -> links for all directories
• /[dir]/ -> directories, where dir is a number from 0 to 128 (converted to string). Links to all items in that directory.
• /[dir]/[page]/ -> pages inside directories, where page is a number from 0 to 512. Links to other pages in same directory and to other directories with same pages.

So basically a simple 2 nested loops iterating number: 0 to 128 and 0 to 512, rendering all combinations of those numbers.

Additional context

When running generate, memory usage starts below 1gb, quickly goes over it, then eventually gets to 2gb and so on, until Node crashes. In my environment it took about 10 minutes for Node to crash, Nuxt generates 2390 files before crashing.

Logs

<--- Last few GCs --->

[17164:0x7fc13e100000]   306406 ms: Scavenge 4044.3 (4079.8) -> 4044.1 (4087.8) MB, 2.51 / 0.00 ms  (average mu = 0.332, current mu = 0.340) allocation failure; 
[17164:0x7fc13e100000]   306420 ms: Scavenge 4051.8 (4087.8) -> 4051.7 (4087.8) MB, 3.00 / 0.00 ms  (average mu = 0.332, current mu = 0.340) allocation failure; 
[17164:0x7fc13e100000]   306432 ms: Scavenge 4052.9 (4087.8) -> 4052.1 (4103.8) MB, 9.81 / 0.00 ms  (average mu = 0.332, current mu = 0.340) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x102acec15 node::Abort() [/usr/local/bin/node]
 2: 0x102acedf6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/local/bin/node]
 3: 0x102c7da7c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/local/bin/node]
 4: 0x102e6d837 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 5: 0x102e71e07 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
 6: 0x102e6e155 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/usr/local/bin/node]
 7: 0x102e6bbf0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x102e6b13f v8::internal::Heap::HandleGCRequest() [/usr/local/bin/node]
 9: 0x102df7a2e v8::internal::StackGuard::HandleInterrupts() [/usr/local/bin/node]
10: 0x1032c0c18 v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0x103692ef6 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]

[cyberalien]

Did more tests to pin point leak. Leak happens only when pre-rendering files.

Used same test Nuxt installation, but without modules/prerender.ts, so it won't pre-render routes. Built server, started it, used separate process to fetch pages. No memory leak. Server works flawlessly.

So leak happens only when pre-rendering routes.

[danielroe]

This is probably because we keep all payloads in memory:

nuxt/packages/nuxt/src/core/runtime/nitro/renderer.ts

Line 183 in ce7845c

const payloadCache = import.meta.prerender ? useStorage('internal:nuxt:prerender:payload') : null

We need to adopt a fs-backed cache for this.

[bootsmann1995]

@cyberalien are you using nuxt-delay-hydration ?

[cyberalien]

@cyberalien are you using nuxt-delay-hydration ?

No

[jcwieme]

Hi 👋 Any news about this bug or has this bug been prioritized in the roadmap? We're facing the same issue on our side when we use payloadExtraction for our +6k pages.

[cyberalien]

Hi 👋 Any news about this bug or has this bug been prioritized in the roadmap? We're facing the same issue on our side when we use payloadExtraction for our +6k pages.

I just switched to SSR. No point in using SSG for so many pages. I'm using prerender only for most popular pages, which is about 200 pages, the rest are rendered on demand.

Works flawlessly. No leaks. Downside is I need to run a node process, not just upload html, but a basic cheap VPS can easily handle that.

Build process is faster, much fewer files to upload to server, so deployment is many times faster than SSG. Old version of website used Svelte with SSG, it took over 30 minutes to deploy. Now it takes 5.

[andrevferreiraa]

Same thing happening here. Only happens on generate.