What is the page limit of a Docusaurus site?

[i-adarsh]

I've around 20k docs (pages) in my site which I'm trying to deploy, but the server is not getting started and don't even build. I've tested for 5 - 6 docs then it was working fine, but as soon as I have added all my files it didn't start and build. It is getting failed with the below error [1].

Any suggestion on how I can build 20k pages document site with docusaurus?

Just FYI: I've setup the node memory limit to 90GB

export NODE_OPTIONS="--max-old-space-size=92160"

[1]

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[slorber]

There is no specific page limit and we have seen complex sites with 20k+ pages already.

I doubt allocating 90Gb of memory to your node process is a good idea, unless your system actually have such amount of memory. This might prevent the JS runtime to GC temporary objects as it thinks it has available memory while everything is swapping. Maybe try with a more reasonable number like 8GB, 12GB, 16GB?

You might find this issue useful: #4765

Maybe you could use the DOCUSAURUS_PERF_LOGGER=true env variable and tell us at which step Docusaurus crashes? (show me the full logs, I'll figure it out)

#4765 (comment)

Creating a runnable OSS repro could help me troubleshoot it.

---

These upcoming faster options coming in v3.6 could maybe help if you are ok with using a canary version of Docusaurus until we release those: #10556

[i-adarsh]

Thank you so much for the response 🙏

I've allocated 16GB this time and the exported the env variable export DOCUSAURUS_PERF_LOGGER=true, but still facing the same issue. The build got stuck in between for ~1h.

[slorber]

Looks like it's getting stuck at the bundling phase.

I suspect it might be Webpack persistent caching system. It's possible that disabling it would fix the issue, but we don't have any config option for that atm. Try removing the cache property locally in this file and tell us please: /node_modules/@docusaurus/core/lib/webpack/base.js

It's possible that the upcoming Rspack support solves it. (see #10556)

[i-adarsh]

Thank you so much for the response, I've followed the suggestion and its still getting stuck in between:

/node_modules/@docusaurus/core/lib/webpack/base.js, please LMK if I'm commenting the exact lines, or else I've to remove or add any other lines?

npx docusaurus build

I'm waiting for the Docusaurus Faster v3.6, do we have any ETA for this? 🙏

[slorber]

You don't have to wait, use a canary version today: 0.0.0-6101

See for example this upgrade PR: react/react-native-website#4268