OOM on Webpack compilation after typedoc generation

[rbala19]

I'm managing a large doc site built with Docusaurus, and I've been encountering out-of-memory (OOM) errors during the build process, specifically during the Webpack client and server compilation phases. The build runs in a CI environment (2xlarge+ container with ample memory), and while the markdown-to-HTML conversion seems to be completing, the error occurs during the final Webpack compilation after about ~40 minutes.

What I’ve Tried:

I’ve reviewed the markdown-to-HTML generation logs, and they indicate successful rendering with no significant memory issues.
The error seems to arise after the log: [info] [webpackbar] Compiling Client, which suggests the issue may be related to Webpack's memory usage.
I’ve also tried increasing the Node.js memory limit (max_old_space_size), but I still run into an OOM when capping it to the size of the CI container.

I would appreciate any guidance or suggestions on how to optimize the Webpack client/server compilation process for large documentation sites.

Logs:

[debug] Finished conversion in 11659ms
[info] Documentation generated at ./docs/code/cmd
[debug] HTML rendering took 19ms
[info] [webpackbar] Compiling Client
[info] [webpackbar] Compiling Server

<--- Last few GCs --->

[2348:0x6f44d20]  2640714 ms: Mark-Compact 31846.9 (32710.5) -> 31423.0 (32627.3) MB, 12251.84 / 0.00 ms  (average mu = 0.913, current mu = 0.733) allocation failure; GC in old space requested
[2348:0x6f44d20]  2663474 ms: Mark-Compact 31714.5 (32915.7) -> 31713.0 (32845.0) MB, 22231.21 / 0.00 ms  (average mu = 0.768, current mu = 0.023) allocation failure; GC in old space requested


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Webpack config in docusaurus config:

webpack: {
    jsLoader: (isServer) => ({
      loader: require.resolve('swc-loader'),
      options: {
        jsc: {
          parser: {
            syntax: 'typescript',
            tsx: true,
          },
          target: 'es2017',
        },
        module: {
          type: isServer ? 'commonjs' : 'es6',
        },
      },
    }),
  },

[slorber]

Have you tried adding even more memory? export NODE_OPTIONS="--max-old-space-size=16288" or more for example?

Have you tried running it outside of a Docker container?

Can you try using canary with Docusaurus Faster, using Rspack instead of Webpack?

[slorber]

@rbala19 is your site open source?

[rbala19]

I apologize for the late response - had personal matters to attend to... Site is unfortunately not open source. I've tried allocating even more memory and running it locally but still running into OOM. I think it is worth trying docusaurus faster - will update if I run into any issues there.

[rbala19]

Upgrading to version 3.6 and turning on rspack in the experimental options did the trick - I am no nowhere close to my former limit of 40GB (I've tested by only giving the build 16 GB with no issues). Very much appreciate this optimization from the docusaurus team!

[slorber]

Awesome news, great that it worked for you!

Also curious to know how faster it is to build now