I tried to build a large site... and did not get very far... :-(

[ilg-ul]

I tried to build a fairly large site (+20K mdx files, 1.1 GB), and after about 30 minutes it looked like stuck:

> website-llvm@0.0.0 build
> node --max-old-space-size=16000 --stack-size=2048 ./node_modules/.bin/docusaurus build

[INFO] [en] Creating an optimized production build...

@xpack/docusaurus-plugin-doxygen: starting...
● Client ██████████████████████████████████████████████████ (41%) building /Users/ilg/tmp/website-llvm/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/ilg/tmp/website-llvm/docs/api/classes/llvm-ivusers.mdx                  
● Server ██████████████████████████████████████████████████ (27%) building /Users/ilg/tmp/website-llvm/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/ilg/tmp/website-llvm/docs/api/structs/llvm-mcpseudoprobefuncdesc.mdx    

Memory consumption was huge, the system entered swap and became less responsive (my Mac has 32 GB RAM). The system is not dead, since in the Activity Monitor I see the process still 100% active, just that the progress bar is stuck.

The site is the LLVM reference, generated as MDX from Doxygen XML output.

I enabled faster via:

  future: {
    v4: {
      removeLegacyPostBuildHeadAttribute: true
    },
    experimental_faster: true,
  },

I also increased the node process heap and stack, otherwise it crashed much earlier.

This was just a test, I admit that an extreme one, to see how the docusaurus-plugin-doxygen behaves on large sites (it performed the conversion in about 8 minutes).

But we'll probably not be able to see the result, the site seems too large for Docusaurus to render. (I killed the stuck build after 45 minutes).

[slorber]

We have another site with 20k+ docs and it builds fine but there's an algo that doesn't scale well within Rspack.
Try the trick in #11199 and see?

If it doesn't work, this would be helpful to create a repro so that I can report a bug to the Rspack team. It should work and be able to bundle such app in less than 5 minutes on a decent computer.

[ilg-ul]

I added the following to docusaurus-config.ts:

  plugins: [
    function disableExpensiveBundlerOptimizationPlugin() {
      return {
        name: "disable-expensive-bundler-optimizations",
        configureWebpack(_config, isServer) {
          return {
            optimization: {
              concatenateModules: false,
            },
          };
        },
      };
    },
  ],

After 30 minutes it went just a bit farther:

> node --max-old-space-size=16000 --stack-size=2048 ./node_modules/.bin/docusaurus build

[INFO] [en] Creating an optimized production build...
● Client ██████████████████████████████████████████████████ (45%) building /Users/ilg/tmp/website-llvm/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/ilg/tmp/website-llvm/docs/api/files/include-include-llvm-include-llvm-transforms-include-llvm-transfor
● Server ██████████████████████████████████████████████████ (29%) building /Users/ilg/tmp/website-llvm/node_modules/@docusaurus/mdx-loader/lib/index.js??ruleSet[1].rules[8].use[0]!/Users/ilg/tmp/website-llvm/docs/api/structs/llvm-mcpseudoprobefuncdesc.mdx                                  

I think that the main problem is the huge memory usage:

I'll leave it overnight, and report back tomorrow morning (the fan runs full speed, I hope it'll not catch fire...)

[ilg-ul]

The build crashed, probably due to memory:

> node --max-old-space-size=16000 --stack-size=2048 ./node_modules/.bin/docusaurus build

[INFO] [en] Creating an optimized production build...
● Client ██████████████████████████████████████████████████ (71%) sealing dependencies                                                                                                                                                                                                           
● Server ██████████████████████████████████████████████████ (71%) sealing dependencies                                                                                                                                                                                                           zsh: killed     caffeinate npm run build
caffeinate npm run build  21395.56s user 4103.03s system 148% cpu 4:46:47.45 total

There is something wrong with the build, it probably tries to start all 20k+ builds in parallel.

[slorber]

Yes, it's not normal that your system swaps like that, and kind of normal that it's super slow and crashes.

Could you please create a repro by committing the MDX files somewhere?

This would be helpful so that we can inspect with the Rspack team, try running on another computer, and run tracing/profiling tools on your build to find the bottleneck. This is likely an algorithm with a bad cyclomatic complexity that gets slower as the number of documents grows, but let's find out.

[ilg-ul]

I re-created the test and documented the process.

A large Git repo with the test is available at:

• https://github.com/xpack/web-llvm

[ilg-ul]

One more note: the purpose of this test was to check how the docusaurus-plugin-doxygen behaves on large sites, not Docusaurus itself.

I don't have any information about the LLVM team considering using Docusaurus for their site, and I doubt that the plugin will be used in such a configuration.

However, in order to improve the plugin, it would be useful for me to see how Docusaurus renders the generated MDX files.

Not to mention that fixing Docusaurus to use less memory would be beneficial for other large projects too.

[ilg-ul]

It's hard to understand why it takes so much memory to build

Please don't exclude me doing something stupid when generating the MDX files. :-(

[slorber]

We'll figure this out 😄

One thing I already notice is that your site has many MDX docs that do not compile. Since we don't finish the build, unfortunately we never print those errors anywhere but they are here (and may be the cause, let's see). I suspect we may exhaust memory by keeping many error stacktraces in memory. Some examples:

loading error Error: MDX compilation failed for file "website/docs/api/classes/anonymous-namespace-machineblockplacement-cpp--machineblockplacement.mdx"
Cause: Expected a closing tag for `<CodeBlock>` (701:141-701:152) before the end of `paragraph`
Details:
{
  "column": 1,
  "message": "Expected a closing tag for `<CodeBlock>` (701:141-701:152) before the end of `paragraph`",

loading error Error: MDX compilation failed for file "website/docs/api/classes/llvm-orc-selfexecutorprocesscontrol.mdx"
Cause: Expected the closing tag `</ProgramListing>` either after the end of `paragraph` (215:34) or another opening tag after the start of `paragraph` (215:1)
Details:
{
  "column": 1,
  "message": "Expected the closing tag `</ProgramListing>` either after the end of `paragraph` (215:34) or another opening tag after the start of `paragraph` (215:1)",
  "line": 215,

[ilg-ul]

many MDX docs that do not compile ... Expected a closing tag

Hmmm... I saw these errors before, they were usually caused by unescaped content passed to components.

Is it complicated to enable more debugging to see these errors?

[slorber]

Not complicated but most sites have a reasonable amount of bundling errors and build faster so it's better to print all errors at the end at once rather than fail fast or print them progressively. We could introduce an option or secret env variable to do so, though, as a way to troubleshoot such bundling errors.

But I'm not sure it's the problem in the end. Building your site with 20k times the same average MDX docs works fine. It looks like there are some specific docs that trigger high memory usage, possibly due to Regexp or Remark plugins (like directives or emojis, since you use : in various places)

[ilg-ul]

(like directives or emojis, since you use : in various places)

You can temporarily patch your local Docusaurus to disable the emoji library, it is definitely not needed anywhere in this website.

(one more reason to add the configuration option to do this) ;-)

[ilg-ul]

I fixed the two errors you mentioned (by adding empty lines) and pushed a new version of the mdx files.

[slorber]

So after investigating, it seems that the problem is not really the number of files, but their complexity.

You have many very large MDX files with complex content to parse with MDX.

find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 100
 29M	./index/classes/all.mdx
 21M	./index/classes/functions.mdx
 16M	./files/lib-lib-target-lib-target-x86-x86isellowering-cpp.mdx
8.1M	./files/lib-lib-target-lib-target-aarch64-aarch64isellowering-cpp.mdx
7.7M	./files/lib-lib-codegen-lib-codegen-selectiondag-dagcombiner-cpp.mdx
7.6M	./namespaces/llvm.mdx
6.1M	./files/lib-lib-target-lib-target-riscv-riscvisellowering-cpp.mdx
6.0M	./index/classes/variables.mdx
6.0M	./files/lib-lib-target-lib-target-arm-armisellowering-cpp.mdx
5.2M	./files/lib-lib-transforms-lib-transforms-vectorize-slpvectorizer-cpp.mdx
4.9M	./files/lib-lib-target-lib-target-powerpc-ppcisellowering-cpp.mdx
4.4M	./files/lib-lib-target-lib-target-amdgpu-siisellowering-cpp.mdx
4.1M	./files/lib-lib-analysis-scalarevolution-cpp.mdx
3.6M	./files/lib-lib-transforms-lib-transforms-ipo-attributorattributes-cpp.mdx
3.6M	./files/lib-lib-support-unicodenametocodepointgenerated-cpp.mdx
3.6M	./files/lib-lib-codegen-lib-codegen-selectiondag-selectiondag-cpp.mdx
3.5M	./files/lib-lib-target-lib-target-arm-lib-target-arm-asmparser-armasmparser-cpp.mdx
3.3M	./files/lib-lib-codegen-lib-codegen-selectiondag-selectiondagbuilder-cpp.mdx
3.2M	./files/lib-lib-codegen-lib-codegen-selectiondag-targetlowering-cpp.mdx
2.8M	./files/lib-lib-target-lib-target-systemz-systemzisellowering-cpp.mdx
2.7M	./files/lib-lib-transforms-lib-transforms-vectorize-loopvectorize-cpp.mdx
2.7M	./files/lib-lib-analysis-valuetracking-cpp.mdx
2.6M	./files/lib-lib-asmparser-llparser-cpp.mdx
2.6M	./classes/index.mdx
2.5M	./files/lib-lib-target-lib-target-x86-x86instrinfo-cpp.mdx
2.5M	./files/lib-lib-target-lib-target-amdgpu-siinstrinfo-cpp.mdx
2.5M	./files/lib-lib-target-lib-target-amdgpu-lib-target-amdgpu-asmparser-amdgpuasmparser-cpp.mdx
2.5M	./files/lib-lib-target-lib-target-aarch64-aarch64instrinfo-cpp.mdx
2.4M	./files/lib-lib-codegen-lib-codegen-globalisel-legalizerhelper-cpp.mdx
2.3M	./files/lib-lib-transforms-lib-transforms-instcombine-instcombinecompares-cpp.mdx
2.3M	./files/lib-lib-frontend-lib-frontend-openmp-ompirbuilder-cpp.mdx
2.2M	./files/lib-lib-target-lib-target-aarch64-lib-target-aarch64-asmparser-aarch64asmparser-cpp.mdx
2.2M	./files/lib-lib-codegen-codegenprepare-cpp.mdx
2.1M	./files/lib-lib-transforms-lib-transforms-utils-simplifycfg-cpp.mdx
2.1M	./files/lib-lib-target-lib-target-mips-lib-target-mips-asmparser-mipsasmparser-cpp.mdx
2.1M	./files/lib-lib-target-lib-target-arm-lib-target-arm-disassembler-armdisassembler-cpp.mdx
2.1M	./files/lib-lib-bitcode-lib-bitcode-reader-bitcodereader-cpp.mdx
2.1M	./files/lib-lib-analysis-instructionsimplify-cpp.mdx
2.0M	./files/lib-lib-target-lib-target-aarch64-lib-target-aarch64-gisel-aarch64instructionselector-cpp.mdx
2.0M	./files/lib-lib-codegen-lib-codegen-selectiondag-legalizevectortypes-cpp.mdx
2.0M	./files/lib-lib-codegen-lib-codegen-globalisel-combinerhelper-cpp.mdx
1.9M	./files/lib-lib-target-lib-target-x86-x86targettransforminfo-cpp.mdx
1.9M	./files/lib-lib-target-lib-target-powerpc-ppciseldagtodag-cpp.mdx
1.9M	./files/lib-lib-target-lib-target-amdgpu-amdgpulegalizerinfo-cpp.mdx
1.9M	./files/lib-lib-target-lib-target-aarch64-aarch64iseldagtodag-cpp.mdx
1.9M	./files/lib-lib-ir-verifier-cpp.mdx
1.8M	./files/lib-lib-transforms-lib-transforms-scalar-loopstrengthreduce-cpp.mdx
1.8M	./files/lib-lib-transforms-lib-transforms-instrumentation-memorysanitizer-cpp.mdx
1.8M	./files/lib-lib-target-lib-target-loongarch-loongarchisellowering-cpp.mdx
1.8M	./files/lib-lib-mc-lib-mc-mcparser-masmparser-cpp.mdx
1.7M	./index/classes/classes.mdx
1.7M	./files/lib-lib-target-lib-target-arm-armbaseinstrinfo-cpp.mdx
1.7M	./files/lib-lib-codegen-lib-codegen-selectiondag-legalizeintegertypes-cpp.mdx
1.7M	./files/include-include-llvm-include-llvm-transforms-include-llvm-transforms-ipo-attributor-h.mdx
1.6M	./files/lib-lib-target-lib-target-x86-x86iseldagtodag-cpp.mdx
1.6M	./files/lib-lib-target-lib-target-powerpc-ppcperfectshuffle-h.mdx
1.6M	./files/lib-lib-target-lib-target-aarch64-aarch64perfectshuffle-h.mdx
1.6M	./files/lib-lib-ir-core-cpp.mdx
1.6M	./files/include-include-llvm-include-llvm-demangle-itaniumdemangle-h.mdx
1.5M	./files/lib-lib-transforms-lib-transforms-ipo-openmpopt-cpp.mdx
1.5M	./files/lib-lib-transforms-lib-transforms-instcombine-instcombineandorxor-cpp.mdx
1.5M	./files/lib-lib-target-lib-target-arm-armperfectshuffle-h.mdx
1.5M	./files/lib-lib-target-lib-target-arm-armiseldagtodag-cpp.mdx
1.5M	./files/lib-lib-target-lib-target-amdgpu-amdgpuisellowering-cpp.mdx
1.5M	./files/lib-lib-target-lib-target-amdgpu-amdgpuinstructionselector-cpp.mdx
1.5M	./files/lib-lib-target-lib-target-aarch64-aarch64framelowering-cpp.mdx
1.5M	./files/lib-lib-mc-lib-mc-mcparser-asmparser-cpp.mdx
1.5M	./files/lib-lib-ir-autoupgrade-cpp.mdx
1.5M	./files/lib-lib-bitcode-lib-bitcode-writer-bitcodewriter-cpp.mdx
1.4M	./files/lib-lib-transforms-lib-transforms-scalar-sroa-cpp.mdx
1.4M	./files/lib-lib-transforms-lib-transforms-instcombine-instructioncombining-cpp.mdx
1.4M	./files/lib-lib-target-lib-target-powerpc-ppcinstrinfo-cpp.mdx
1.4M	./files/lib-lib-target-lib-target-nvptx-nvptxisellowering-cpp.mdx
1.4M	./files/lib-lib-target-lib-target-mips-mipsisellowering-cpp.mdx
1.4M	./files/lib-lib-target-lib-target-aarch64-aarch64targettransforminfo-cpp.mdx
1.4M	./files/lib-lib-object-machoobjectfile-cpp.mdx
1.4M	./files/lib-lib-ir-asmwriter-cpp.mdx
1.4M	./files/lib-lib-codegen-lib-codegen-selectiondag-legalizedag-cpp.mdx
1.4M	./files/include-include-llvm-include-llvm-codegen-targetlowering-h.mdx
1.3M	./files/lib-lib-target-lib-target-hexagon-hexagoninstrinfo-cpp.mdx
1.3M	./files/lib-lib-target-lib-target-amdgpu-amdgpuregisterbankinfo-cpp.mdx
1.3M	./files/index.mdx
1.3M	./files/include-include-llvm-include-llvm-ir-instructions-h.mdx
1.2M	./files/lib-lib-transforms-lib-transforms-ipo-memprofcontextdisambiguation-cpp.mdx
1.2M	./files/lib-lib-transforms-lib-transforms-instcombine-instcombinecalls-cpp.mdx
1.2M	./files/lib-lib-target-lib-target-x86-x86framelowering-cpp.mdx
1.2M	./files/lib-lib-target-lib-target-x86-lib-target-x86-asmparser-x86asmparser-cpp.mdx
1.2M	./files/lib-lib-target-lib-target-riscv-riscvinstrinfo-cpp.mdx
1.2M	./files/lib-lib-target-lib-target-aarch64-aarch64fastisel-cpp.mdx
1.2M	./files/lib-lib-support-apfloat-cpp.mdx
1.2M	./files/lib-lib-codegen-machinescheduler-cpp.mdx
1.2M	./files/lib-lib-codegen-lib-codegen-asmprinter-asmprinter-cpp.mdx
1.2M	./files/include-include-llvm-include-llvm-ir-debuginfometadata-h.mdx
1.2M	./files/include-include-llvm-c-core-h.mdx
1.1M	./files/lib-lib-transforms-lib-transforms-vectorize-vplan-h.mdx
1.1M	./files/lib-lib-transforms-lib-transforms-utils-simplifylibcalls-cpp.mdx
1.1M	./files/lib-lib-transforms-lib-transforms-utils-local-cpp.mdx
1.1M	./files/lib-lib-transforms-lib-transforms-scalar-newgvn-cpp.mdx
1.1M	./files/lib-lib-transforms-lib-transforms-instcombine-instcombineselect-cpp.mdx
1.1M	./files/lib-lib-target-lib-target-mips-mipsseisellowering-cpp.mdx

On the index folder, even with most MDX-specific things disabled and using a simpler CommonMark parser, a large file takes 100s to compile alone, and then apparently gets stuck (but memory usage remains low). I'm not sure it's even related to Rspack, and the content might trigger some weird performance issues in the MDX parser.

This would be interesting to try parsing those files sequentially, outside of Docusaurus, with no bundler, to see the outcome, but I suspect it wouldn't work that well either.

However, I do notice a memory spike at the end of the bundling phase when Rspack seals and emits the JS bundles.

[slorber]

From what I see, excluding some of the largest files makes it possible to bundle within ~10-20 minutes.

Here's some docs plugin config:

 exclude: [

            '**/index/classes/all.mdx',
            '**/index/classes/functions.mdx',
            '**/index/classes/variables.mdx',

            '**/files/lib-lib-target-lib-target-x86-x86isellowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64isellowering-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-dagcombiner-cpp.mdx',
            '**/files/lib-lib-target-lib-target-riscv-riscvisellowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-arm-armisellowering-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-vectorize-slpvectorizer-cpp.mdx',
            '**/files/lib-lib-target-lib-target-powerpc-ppcisellowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-siisellowering-cpp.mdx',
            '**/files/lib-lib-analysis-scalarevolution-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-ipo-attributorattributes-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-selectiondag-cpp.mdx',
            '**/files/lib-lib-support-unicodenametocodepointgenerated-cpp.mdx',
            '**/files/lib-lib-target-lib-target-arm-lib-target-arm-asmparser-armasmparser-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-selectiondagbuilder-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-targetlowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-systemz-systemzisellowering-cpp.mdx',
            '**/files/lib-lib-analysis-valuetracking-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-vectorize-loopvectorize-cpp.mdx',
            '**/files/lib-lib-asmparser-llparser-cpp.mdx',
            '**/files/lib-lib-target-lib-target-x86-x86instrinfo-cpp.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-lib-target-amdgpu-asmparser-amdgpuasmparser-cpp.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-siinstrinfo-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64instrinfo-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-globalisel-legalizerhelper-cpp.mdx',
            '**/files/lib-lib-frontend-lib-frontend-openmp-ompirbuilder-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instcombine-instcombinecompares-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-lib-target-aarch64-asmparser-aarch64asmparser-cpp.mdx',
            '**/files/lib-lib-codegen-codegenprepare-cpp.mdx',
            '**/files/lib-lib-bitcode-lib-bitcode-reader-bitcodereader-cpp.mdx',
            '**/files/lib-lib-target-lib-target-mips-lib-target-mips-asmparser-mipsasmparser-cpp.mdx',
            '**/files/lib-lib-target-lib-target-arm-lib-target-arm-disassembler-armdisassembler-cpp.mdx',
            '**/files/lib-lib-analysis-instructionsimplify-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-utils-simplifycfg-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-legalizevectortypes-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-globalisel-combinerhelper-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-lib-target-aarch64-gisel-aarch64instructionselector-cpp.mdx',
            '**/files/lib-lib-target-lib-target-powerpc-ppciseldagtodag-cpp.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-amdgpulegalizerinfo-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64iseldagtodag-cpp.mdx',
            '**/files/lib-lib-ir-verifier-cpp.mdx',
            '**/files/lib-lib-target-lib-target-x86-x86targettransforminfo-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-scalar-loopstrengthreduce-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instrumentation-memorysanitizer-cpp.mdx',
            '**/files/lib-lib-mc-lib-mc-mcparser-masmparser-cpp.mdx',
            '**/files/lib-lib-target-lib-target-loongarch-loongarchisellowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-arm-armbaseinstrinfo-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-legalizeintegertypes-cpp.mdx',
            '**/files/lib-lib-ir-core-cpp.mdx',
            '**/files/lib-lib-target-lib-target-x86-x86iseldagtodag-cpp.mdx',
            '**/files/lib-lib-target-lib-target-powerpc-ppcperfectshuffle-h.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64perfectshuffle-h.mdx',
            '**/files/lib-lib-target-lib-target-arm-armperfectshuffle-h.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-amdgpuisellowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-amdgpuinstructionselector-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-ipo-openmpopt-cpp.mdx',
            '**/files/lib-lib-ir-autoupgrade-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instcombine-instcombineandorxor-cpp.mdx',
            '**/files/lib-lib-mc-lib-mc-mcparser-asmparser-cpp.mdx',
            '**/files/lib-lib-target-lib-target-arm-armiseldagtodag-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64framelowering-cpp.mdx',
            '**/files/lib-lib-bitcode-lib-bitcode-writer-bitcodewriter-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instcombine-instructioncombining-cpp.mdx',
            '**/files/lib-lib-target-lib-target-powerpc-ppcinstrinfo-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64targettransforminfo-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-scalar-sroa-cpp.mdx',
            '**/files/lib-lib-target-lib-target-nvptx-nvptxisellowering-cpp.mdx',
            '**/files/lib-lib-target-lib-target-mips-mipsisellowering-cpp.mdx',
            '**/files/lib-lib-object-machoobjectfile-cpp.mdx',
            '**/files/lib-lib-ir-asmwriter-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-legalizedag-cpp.mdx',
            '**/files/lib-lib-target-lib-target-amdgpu-amdgpuregisterbankinfo-cpp.mdx',
            '**/files/lib-lib-target-lib-target-hexagon-hexagoninstrinfo-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instcombine-instcombinecalls-cpp.mdx',
            '**/files/lib-lib-target-lib-target-aarch64-aarch64fastisel-cpp.mdx',
            '**/files/lib-lib-support-apfloat-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-ipo-memprofcontextdisambiguation-cpp.mdx',
            '**/files/lib-lib-target-lib-target-x86-x86framelowering-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-asmprinter-asmprinter-cpp.mdx',
            '**/files/lib-lib-target-lib-target-x86-lib-target-x86-asmparser-x86asmparser-cpp.mdx',
            '**/files/lib-lib-codegen-machinescheduler-cpp.mdx',
            '**/files/lib-lib-target-lib-target-riscv-riscvinstrinfo-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instcombine-instcombineselect-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-selectiondagisel-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-utils-simplifylibcalls-cpp.mdx',
            '**/files/lib-lib-ir-instructions-cpp.mdx',
            '**/files/lib-lib-codegen-machineverifier-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-vectorize-vplan-h.mdx',
            '**/files/lib-lib-target-lib-target-mips-mipsseisellowering-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-scalar-newgvn-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-utils-local-cpp.mdx',
            '**/files/lib-lib-tablegen-tgparser-cpp.mdx',
            '**/files/lib-lib-analysis-dependenceanalysis-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-ipo-attributor-cpp.mdx',
            '**/files/lib-lib-transforms-lib-transforms-instrumentation-addresssanitizer-cpp.mdx',
            '**/files/lib-lib-target-lib-target-riscv-lib-target-riscv-asmparser-riscvasmparser-cpp.mdx',
            '**/files/lib-lib-codegen-registercoalescer-cpp.mdx',
            '**/files/lib-lib-target-lib-target-spirv-spirvinstructionselector-cpp.mdx',
            '**/files/lib-lib-target-lib-target-riscv-riscviseldagtodag-cpp.mdx',
            '**/files/lib-lib-target-lib-target-hexagon-hexagonisellowering-cpp.mdx',
            '**/files/lib-lib-support-lib-support-rpmalloc-rpmalloc-c.mdx',
            '**/files/lib-lib-codegen-lib-codegen-selectiondag-legalizefloattypes-cpp.mdx',
            '**/files/lib-lib-codegen-lib-codegen-globalisel-irtranslator-cpp.mdx',
]

But even after that, SSG is quite expensive to run on those files. Even if the system takes a lot of memory and swaps, it doesn't seem to crash and completes relatively fast. I don't know why but there are clearly files in this list that cause the bundling phase to get stuck, notably the largest all.mdx. You have 1Gb of MDX to parse, that's huge and maybe triggers some MDX edge cases.

The MDX compilation could also be faster if you used a CommonMark format, which is simpler to parse, and it's still possible to map regular HTML tags to custom components if needed (MDX is really needed only when you need things like complex JSX/React props, but not needed for simple string attributes). Even better would be to not map at all to any React component, and output standard HTML: the compiled React components would be faster.

---

I'll have to stop investigating this site. It's very time consuming and unfortunately I couldn't really find any low hanging fruit bottleneck on it. Even the Rspack team admit they are not sure their tool can handle that scale yet.

I'll keep this in my head to come back to it later, but I think v3.8 has already improved quite a lot on performance, and what remains to optimize is more complex 😅

Even if we made it bundle, from what I see the SSG phase could be an even longer bottleneck. Our SSG pipeline is not optimized for that kind of scale, and for a server bundle of that size with a very large JS entrypoint. It takes a lot of memory as soon as you use multiple workers, and is slow if you only use one.

For this kind of site, I'd recommend splitting it into many smaller ones instead.

[slorber]

Here's an example of rendering classes/* only. Using a CommonMark parser (not accurate, but simpler and doesn't produce mdx compilation errors), we can build the site.

But still, it's not a great UX because the pages are extremely long. Even if the full site builded, this would lead to a really bad UX because the HTML pages are extremely large, having to render very long documents, but also a very large non-virtualized sidebar.

We optimize the sidebar UI/markup: we don't statically render what's inside collapsed categories, but we do have to render the sidebar siblings of the current page to avoid layout shifts on load. Even with these optimizations, your typical HTML page will be 4mb, not even mentioning the JS that needs to load before the page can become interactive.

So, I think Docusaurus is definitively not good enough to support that kind of scale. Even if we did, having documents that long would not be great for performance.

The SSG is slow also due to that. The "classes" folder has 6k+ files and each HTML file has to render that slice of the sidebar. Even for smaller MDX files, generating each HTML takes ~200ms on my Mac M3 Pro, likely due to that long list that must be present in each page.

[ilg-ul]

Hmmm... That page doesn't look ok at all.

Here is how such a page should look like (on a much simpler site):

• https://micro-os-plus.github.io/micro-test-plus-xpack/docs/api/classes/

As for the sidebar, this is indeed a problem, since each page needs an entry in the sidebar, the resulting sidebar tree is huge. I really don't know how to handle this case. :-(

Here is the original LLVM reference site. The list of classes is also huge:

• https://llvm.org/doxygen/classes.html

This site doesn't look great, but it is pretty functional.

Probably I was too optimistic with my desire to move everything to Docusaurus... :-(

I'll make some optimisations, but I don't expect miracles, for now sites at the LLVM scale are too heavy for Docusaurus.

However, since the docusaurus-plugin-doxygen managed to convert it, I would say it did a fairly good job.

For smaller projects, like the one I mentioned before, I think it is ok. It can definitly be improved, but this is for future versions.

For now it cannot be used as a substitute for all original Doxygen sites, but we learned some lessons, and perhaps this test will help us improve both the plugin and Docusaurus.

---

I'll keep this project active on GitHub, in case we'll need to run extreme tests with further Docusaurus versions.

[slorber]

Hmmm... That page doesn't look ok at all.

Yes it's normal: I'm using CommonMark instead of MDX because if I don't, the bundling fail due to docs that are not valid MDX syntax. I can't fix all your bad MDX docs so it's simpler to attempt a build with the wrong parser but at least be able to bundle (because CommonMark is much less strict and won't error).

Here is how such a page should look like (on a much simpler site):

• micro-os-plus.github.io/micro-test-plus-xpack/docs/api/classes

As for the sidebar, this is indeed a problem, since each page needs an entry in the sidebar, the resulting sidebar tree is huge. I really don't know how to handle this case. :-(

Here is the original LLVM reference site. The list of classes is also huge:

• llvm.org/doxygen/classes.html

This site doesn't look great, but it is pretty functional.

This site is already taking a lot of resources for my M3 and scrolling is not really performant.

Also this site is using a much simpler HTML markup than our sidebar.

And it's using vanilla JS instead of React, so there's no hydration process like React does to make the page interactive (which for a page like this would be quite slow).

Probably I was too optimistic with my desire to move everything to Docusaurus... :-(

I'll make some optimisations, but I don't expect miracles, for now sites at the LLVM scale are too heavy for Docusaurus.

However, since the docusaurus-plugin-doxygen managed to convert it, I would say it did a fairly good job.

For smaller projects, like the one I mentioned before, I think it is ok. It can definitly be improved, but this is for future versions.

For now it cannot be used as a substitute for all original Doxygen sites, but we learned some lessons, and perhaps this test will help us improve both the plugin and Docusaurus.

I'll keep this project active on GitHub, in case we'll need to run extreme tests with further Docusaurus versions.

We'll figure out over time but to be honest I'm not sure Docusaurus is ideal for any large API ref documentation. Docusaurus and other frameworks with client-side navigation is better suited for smaller content that you author yourself.

I assume other projects such as the Javadoc may not perform well on Docusaurus, and should rather not use a React/Vue based solution but stick to the simplest possible HTML. Or they should find a way to split their site into smaller subsites.

At the end of the day, Docusaurus builds a frontend app, not a standard website. It can improve the UX, but it also has a cost, and here the cost is too big.