fd starts slowly on systems with very many cores

[tavianator]

Originally posted by @pmarreck in #1131 (comment)

I use fd in some test suites and I noticed a slowdown. I investigated and I found that simply running fd with no arguments in an empty directory on /tmp (which is in-memory!) was taking 0.2 seconds. This is ridiculously slow on my machine (Ryzen Threadripper 3990x with 64 cores/128 threads running on NVMe). Multiplied by the number of times I was calling fd in my suite, this immediately explained why multiple seconds had been added at some point.

When I add the -j1 argument, as someone else indicated, the boot runtime of running fd alone drops to 0.01 seconds, which is 20x faster and far more reasonable an expectation.

I'm on NixOS unstable running on ZFS root.

@matu3ba & @tavianator , this is NOT a filesystem problem (IMHO). This looks like a heuristics bug regarding non-specification of the number of concurrent jobs to run (per the mandocs, not specifying it "uses heuristics")

At least I now know a workaround that works!

[pmarreck]

(Just a note, I have a script at #1131 that anyone can run which shows the problem.)

[sharkdp]

Can someone please provide some benchmark results? See #1131 (comment) as a starting point. Or https://github.com/sharkdp/fd-benchmarks for even more benchmarking scripts.

[paulirish]

some benchmarks:

enormous folder (chromium repo w/ 443K files)

empty folder

fyi nproc says 48, though i had seen 96 in lscpu, thus the above command. :) oops

json results for both: https://gist.github.com/paulirish/b94888c2e10f65702d8e09210c1584ad