Unable to start or build with too many files

[ManasMadrecha]

Environment

• Windows 11 (8gb ram)
• Node 20

Reproduction

• I have 300 files (json), each around 300kb in size in the content folder. The app fails to start itself, i.e., on npm run dev, the app fatally fails with javascript heap memory error

• So, I spilt these 300 files into 40 files each, so now I have around 12000 files (json), each much smaller with 7-8 kb in size. Now, the app starts properly (i.e., no error on npm run dev. I'm able to query properly in dev, but the app fails fatally on npm run build.

I'm using Nuxt 3 server app, i.e., Nitro Firebase preset. So, I'm using serverQueryContent in the /api/articles/index.ts file to serve my response. In this file, I'm finding all the 12000 json files, and then filtering them using normal filter function (because I want to query the files by the object keys of the same file, which is not possible with the module's query syntax). This is working properly in dev. On build, it fails.

Describe the bug

First bug (few files, but large in size)

• App fails on npm run dev itself with error of memory heap.

Second bug (many files, small in size)

• On npm run dev, several red warnings about too many files open. But the app still starts properly.

• On npm run build, there's a fatal error related to unstorage that error [unstorage] Cannot stringify value!. On inspection, I found that it is due to Promise.all syntax in the map in the content module's code. When I changed it to simple for... of loop, and added a wait time of 5 ms, this stringify error went away, and the app starts building for production. But then a new fatal error props up later related to tailwind config file that too many files open.

Additional context

Can the module contributors kindly look into it? I want to add 160000 files ultimately, but the module fails at 12000 files itself. I tried removing the files, and found the module works fine (both during dev and build) with max 6000-7000 files of 7-8 kb.

Note all these files are json (object format).

Logs

No response

[ManasMadrecha]

Can graceful-fs be a solution? If yes, where to insert it in the app to globally patch the fs?

[MrLesk]

I have the same issue. I am expected to have around 6000 pages. Right now the build fails with the same error as mentioned above.
I am looking into a way to build only the routes that are changed but in some cases I need to rebuild all routes from scratch