Lots of system errors during build: (EMFILE: too many open files)

[ilg-ul]

I have a moderately large site, with many (~1000) reference pages generated automatically outside Docusaurus from the XML output of Doxygen.

The pages are rendered ok, but most of the times I regenerate these pages while Docusaurus is started, it fails with lots of EMFILE: too many open files error, then the node instance crashes:

✔ Client
  Compiled successfully in 2.40s

client (webpack 5.99.9) compiled successfully
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (404kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (182kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (200kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (665kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)
<w> [webpack.cache.PackFileCacheStrategy] Serializing big strings (140kiB) impacts deserialization performance (consider using Buffer instead and decode when needed)

● Client █████████████████████████ building (10%) 0/3 entries 3/3 dependencies 7/3 modules 0 active 
 

[ERROR] Loading of version failed for version current
[ERROR] Site plugin reload failure - Plugin docs@default
[Error: EMFILE: too many open files, scandir '/Users/ilg/MyProjects/micro-os-plus.github/www/web-docusaurus.git/website/docs/api/classes'] {
  errno: -24,
  code: 'EMFILE',
  syscall: 'scandir',
  path: '/Users/ilg/MyProjects/micro-os-plus.github/www/web-docusaurus.git/website/docs/api/classes'
}
[ERROR] Loading of version failed for version current
[ERROR] Site plugin reload failure - Plugin docs@default
[Error: EMFILE: too many open files, scandir '/Users/ilg/MyProjects/micro-os-plus.github/www/web-docusaurus.git/website/docs/getting-started/_project'] {
  errno: -24,
  code: 'EMFILE',
  syscall: 'scandir',
  path: '/Users/ilg/MyProjects/micro-os-plus.github/www/web-docusaurus.git/website/docs/getting-started/_project'
}
... many, many

<--- Last few GCs --->

[78007:0x7f7a20008000]  5030848 ms: Mark-Compact 3864.1 (4139.8) -> 3862.8 (4138.5) MB, 469.42 / 0.00 ms  (average mu = 0.781, current mu = 0.126) allocation failure; scavenge might not succeed
[78007:0x7f7a20008000]  5031382 ms: Mark-Compact 3880.3 (4141.4) -> 3877.9 (4147.7) MB, 504.70 / 0.00 ms  (average mu = 0.617, current mu = 0.055) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x10f1c1870 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 2: 0x10f38183c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 3: 0x10f5715d7 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 4: 0x10f56fe39 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 5: 0x10f5646a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 6: 0x10f5650e5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 7: 0x10f546a3c v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 8: 0x10f53c965 v8::internal::MaybeHandle<v8::internal::SeqOneByteString> v8::internal::FactoryBase<v8::internal::Factory>::NewRawStringWithMap<v8::internal::SeqOneByteString>(int, v8::internal::Map, v8::internal::AllocationType) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 9: 0x10f548f84 v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<unsigned char const> const&, unibrow::Utf8Variant, v8::internal::AllocationType) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
10: 0x10f54912f v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<char const> const&, v8::internal::AllocationType) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
11: 0x10f3a32ce v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
12: 0x10f2af4c4 node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
13: 0x10f19625e void node::Buffer::(anonymous namespace)::StringSlice<(node::encoding)1>(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
14: 0x10fd09586 Builtins_CallApiCallback [/Users/ilg/.nvm/versions/node/v20.18.0/bin/node]
 *  Terminal will be reused by tasks, press any key to close it. 

Also a bit worrying are the webpack warnings.

My workaround is to stop Docusaurus before regenerating the reference pages, but this is quite tedious.

I'm using 3.6.3 on a macOS.

Any suggestions?

[slorber]

You should try upgrading to the latest version, and try Rspack. Also you get an OOM error so increasing the default nodejs memory could help.

Otherwise it's really hard to help without a repro

[ilg-ul]

You should try upgrading to the latest version

Ok, thank you.