EMFILE: too many open files when attempting to render a large (170000) number of pages (Eleventy 2.0.0-canary.16)

[kisaragi-hiu]

Describe the bug
EMFILE: too many open files when attempting to render a site with a large number of pages from a large data set.

When it is triggered, it is triggered via EleventyTemplateError.

The number of files being opened is finite: on a system with raised limits it successfully builds.

I have one 40MB json file, containing an array of about 170000 objects, and one template that generates one page per entry. (The site is supposed to be a static dictionary.)

Eleventy version: 2.0.0-canary.16. The site only builds on 2.0 due to #2360.

This behavior has been documented before in #2226 (comment).

Personally, I think this is an issue that can be fixed, as there is no reason why the output files, which depend only on the input file and not on each other, should all be held open at the same time.

To Reproduce

Edit: An earlier version said it took an hour to see the error. That is incorrect: I misread my CI logs, and Eleventy only took 3 minutes.
Edit: This has been rewritten to be more exact and to use error messages from the site generated during the reproduction.

Steps to reproduce the behavior:

1. Create a new test folder

2. Generate the test data and put it in _data/a.json, containing an array with a large number of objects, with each object having a unique, say, "title" key. I use Emacs for this:

   mkdir -p _data
   cat > x.el <<"HERE"
   (require 'json)
   (let (lst)
     (dotimes (i 200000)
       (push (let ((tmp (make-hash-table :test #'equal)))
               (puthash \"title\" (format \"%s\" i) tmp)
               tmp)
             lst))
     (with-temp-file \"_data/a.json\"
       (insert (json-encode lst))))
   HERE
   emacs --script x.el

   The JSON looks something like this:

   [{"title": "199999"}, {"title": "199998"}, ...]

3. Create one template to generate pages for each item:

   ---
   pagination:
     data: a
     size: 1
     alias: item
   permalink: "item/{{ item.title }}/index.html"
   ---
   {{ item.title }}

4. npx @11ty/eleventy --quiet (--quiet is to silence the list of rendered files)

5. On a system with higher FD limits, this will succeed.

6. On a more constrained system, an EMFILE error will be raised (from EleventyTemplateError).

   On Linux the FD limit can be lowered for a command like this:

   prlimit --nofile=50000 npx @11ty/eleventy --quiet

$ prlimit --nofile=50000 npx @11ty/eleventy --quiet
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble writing to "_site/item/199999/index.html" from "./a.njk" (via EleventyTemplateError)
[11ty] 2. EMFILE: too many open files, open '_site/item/49141/index.html' (via Error)
[11ty] 
[11ty] Original error stack trace: Error: EMFILE: too many open files, open '_site/item/49141/index.html'
[11ty] Wrote 0 files in 93.93 seconds (v2.0.0-canary.16)

Expected behavior
Writing many files from one template should not cause an EMFILE error.

Environment:

Where the error is encountered:

• OS and Version: Ubuntu 20.04 (on GitHub Actions), with an FD limit of somewhere between 1000 and 20000
• Eleventy Version: 2.0.0-canary.16

Where the error is not encountered:

• OS and Version: Arch Linux, with an FD limit in the millions
• Eleventy Version: 2.0.0-canary.16

Additional context

I can see that in TemplateWriter.js (generateTemplates) there is essentially an async, unbuffered loop to _generateTemplate → generateMapEntry (Template.js) → _write → writeFile (from graceful-fs).

(This is the source of the EMFILE because it's the only place where EleventyTemplateError can be raised.)

This would rapidly exhaust file descriptors if graceful-fs weren't used. But graceful-fs is used, and it should have handled this, and I don't understand why it's happening.

Perhaps Eleventy should also queue up file writes?

[pdehaan]

@kisaragi-hiu How do you build that site?
I have it cloned locally on macOS, and ran npm install and then make src/_data, but I get the following error:

make src/_data

mkdir -p src/_data
cp moedict-data/dict-revised.json src/_data/dict-moe-revised.json
cp: moedict-data/dict-revised.json: No such file or directory
make: *** [src/_data/dict-moe-revised.json] Error 1

UPDATE: Ah, I missed that you were using git submodules.
I had to run the following:

git submodule update --init
brew install cask
make src/_data

But now I've graduated to the following error:

Loading /tmp/kemdict/process-data.el (source)...
Debugger entered--Lisp error: (file-missing "Cannot open load file" "No such file or directory" "ht")
  require(ht)
  eval-buffer(#<buffer  *load*-56096> nil "/tmp/kemdict/process-data.el" nil t)  ; Reading at buffer position 80
  load-with-code-conversion("/tmp/kemdict/process-data.el" "/tmp/kemdict/process-data.el" nil nil)
  load("process-data")
  eval((load "process-data"))
  (let ((load-path (cask-load-path (cask-cli--bundle)))) (add-to-list 'load-path cask-cli--path) (eval (read form)))
  cask-cli/eval("(load \"process-data\")")
  apply(cask-cli/eval "(load \"process-data\")")
  commander--handle-command(("eval" "(load \"process-data\")"))
  commander-parse(("eval" "(load \"process-data\")"))
  (if commander-parsing-done nil (commander-parse (or commander-args (cdr command-line-args-left))))
  eval-buffer(#<buffer  *load*> nil "/opt/homebrew/Cellar/cask/0.8.8/cask-cli.el" nil t)  ; Reading at buffer position 13305
  load-with-code-conversion("/opt/homebrew/Cellar/cask/0.8.8/cask-cli.el" "/opt/homebrew/Cellar/cask/0.8.8/cask-cli.el" nil t)
  load("/opt/homebrew/Cellar/cask/0.8.8/cask-cli.el" nil t t)
  command-line-1(("-scriptload" "/opt/homebrew/Cellar/cask/0.8.8/cask-cli.el" "--" "eval" "(load \"process-data\")"))
  command-line()
  normal-top-level()

make: *** [src/_data/combined.json] Error 255

[pdehaan]

Comically I got it building on my 2021 MacBook Pro M1 Max w/ 64 GB of RAM… and then it crashed my laptop.

Sadly I cloned into my /tmp folder which got reset after rebooting so I'll have to set it up again and see if we can find a solution.

ls -lash src/_data

130800 -rw-r--r--  1 pdehaan  staff    64M 26 Oct 17:18 combined.json
147272 -rw-r--r--  1 pdehaan  staff    72M 26 Oct 17:18 dict-moe-revised.json
 14704 -rw-r--r--  1 pdehaan  staff   7.2M 26 Oct 17:18 dict-moe-twblg.json

[pdehaan]

OK, the site seems to be reliably crashing my computer (but at least isn't giving me EMFILE errors, although not sure that is an improvement over crashing my entire system or just making it unusable).

First I tried removing the global data files since they will copy into all of the 170k templates' data cascade (and we're talking about 143 MB worth of JSON data in the src/_data/ filter).
Next I copied the src/_data/combined.json (64MB) to a template data file (word-pages.11tydata.json, but also had to add a root {"combined": [...]} wrapper to use for pagination):

ls -lsh src

     0 drwxr-xr-x  2 pdehaan  staff    64B 26 Oct 17:25 _data
     0 drwxr-xr-x  4 pdehaan  staff   128B 26 Oct 17:17 _includes
     8 -rw-r--r--  1 pdehaan  staff   1.2K 26 Oct 17:17 index.njk
130800 -rw-r--r--  1 pdehaan  staff    64M 26 Oct 17:21 word-pages.11tydata.json
     8 -rw-r--r--  1 pdehaan  staff   849B 26 Oct 17:17 word-pages.njk

Now my ./src/** directory structure looks roughly like:

tree src
src
├── _data/
├── _includes/
│   ├── base.njk
│   └── macros.njk
├── index.njk
├── word-pages.11tydata.json
└── word-pages.njk

2 directories, 5 files

And if I build, it seems to:

1. never finish building, and slowly crashes my machine as it dies from lack of resources.
2. writes out about 170,017 templates, per ls -l _site/word | wc -l

Per a quick Google search:

"Apple doc says that the HFS Plus file system has a theoretical limit of a 2 billion files per folder all Mac OS X versions."

Not sure about other OSes, although having 170k+ subdirectories in a single subdirectory is a bit difficult to manage.
If I can manually edit the ./src/word-pages.11tydata.json file and shrink it down to like 50k items or whatever my limit is before my super computer dies, we might be able to run benchmarking and see if there is anything else we can do, although I'm pretty doubtful this will be easy to write out 170k pages.

[kisaragi-hiu]

Right, sorry for not formatting the linked site properly. I've created a new branch for testing (e2627) that I won't modify further, and I've added build instructions (clone submodules, install Cask, cask install, npm install, make). I've also rewritten the reproduction instructions to be more clear so that it's not necessary to clone my site to reproduce it.

In regards to my site, writing out like 170000 output files is the expected result. At least on Linux it seems to be relatively doable: it builds on my PC (with 8GB of RAM), and it even builds on GitHub Actions (Ubuntu) after I raise the FD limit (sudo sysctl -w fs.file-max=500000; sudo prlimit --nofile=500000 npx @11ty/eleventy).

I tested on my PC further, using prlimit to lower the limit. The builds starts failing at a limit around 80000 --- which is weird.

prlimit --nofile=50000 npx "@11ty/eleventy" --input=./src --output=./_site --quiet # fails
prlimit --nofile=75000 npx "@11ty/eleventy" --input=./src --output=./_site --quiet #fails
prlimit --nofile=78125 npx "@11ty/eleventy" --input=./src --output=./_site --quiet #fails
prlimit --nofile=81250 npx "@11ty/eleventy" --input=./src --output=./_site --quiet # succeeds. What?
prlimit --nofile=100000 npx "@11ty/eleventy" --input=./src --output=./_site --quiet # succeeds

I also tried building it on macOS on GitHub Actions. The same error happens:

▶ Run gmake _site
npx tailwindcss --minify --postcss -i css/src.css -o _site/css/built.css

Done in 613ms.
npx "@11ty/eleventy" --input=./src --output=./_site --quiet
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble writing to "./_site/word/{[8e40]}/index.html" from "./src/word-pages.njk" (via EleventyTemplateError)
[11ty] 2. EMFILE: too many open files, open './_site/word/趑趄不前/index.html' (via Error)
[11ty] 
[11ty] Original error stack trace: Error: EMFILE: too many open files, open './_site/word/趑趄不前/index.html'
[11ty] Wrote 1 file in 172.34 seconds (v2.0.0-canary.16)
gmake: *** [Makefile:17: _site] Error 1
Error: Process completed with exit code 2.

[kaleb]

For those of you wanting a temporary fix, I solved this in my own build by patching graceful-fs timeout to be greater than 60000. It is not ideal, but it works for me currently.

[zachleat]

PR #3272 is shipping with 3.0.0-alpha.11