Improve find-packages performance

[vjpr]

Pnpm's find-packages walks the directory tree looking for package.json files and reads them.

Many commands such as pnpm exec and pnpm ls run this.

Many monorepo tooling depends on this. Anytime your tool needs to do something with pnpm workspace packages, you need to run this (usually via its caller @pnpm/find-workspace-packages).

In large monorepos this can be slow. I have a large monorepo of 300+ packages and it takes about 1s on a MBP i9 8-core. Using a native Rust package with parallelism I can get this down to 200ms which is a huge improvement.

find-packages currently uses fast-glob (which uses micromatch for glob to regex compilation)

Library Options

I just had a thorough discussion on the fdir project (which claims to be "The Fastest Directory Crawler & Globber for NodeJS") (uses picomatch).

It has comparable speeds to native Rust-based NAPI solutions.

Although, wrapping the Rust ignore package in a NAPI add-on which uses thread-based parallelism seems to be the fastest.

fdir with child_process workers could potentially compete, although the spin up of the process pool might negate benefits.

There is also tiny-glob (using globrex glob lib) which claims to be "~230% faster than fast-glob". I wouldn't recommend tiny-glob right now. Has no ignore feature. terkelg/tiny-glob#79

Interesting

• macOS readdir seems to be much slower than Linux due to some locking in the APFS file-system.
  • See: Compare to native options thecodrr/fdir#66 (comment)
• Increasing UV_THREADPOOL_SIZE (defaults to 4) can improve things on MBP i9 8-core (setting to 8 works well).
  • On Linux on less powerful machines it can help to reduce it though :/. See: Compare to native options thecodrr/fdir#66 (comment)
• Symlink resolution slows things down.

Options

Node.js

• Replace fast-glob with tiny-glob
• Replace fast-glob with fdir
• Add tip to docs to increase UV_THREADPOOL_SIZE.

Native

NAPI

• Implement find-packages as a native Rust NAPI addon using ignore or jwalk or globwalk.

WASI

• Implement WASI package.
  • Currently experimental. WASI fs impl uses libuv same as Node. Maybe globbing logic in WASI could speed things up. AssemblyScript could be used to write native code in TS syntax.

Daemon

• Using a persistent daemon. Probably too clunky.
  • See: Compare to native options thecodrr/fdir#66 (comment)
  • See: Use persistent file watcher to reduce file system reads for unchanged files #3346

[zkochan]

AssemblyScript could be used to write native code in TS syntax

this is actually very interesting, not just for this issue but overall
could run everything in AssemblyScript?

[vjpr]

I'm planning to play around with it some more.

Re: File system access

WASI is used to interact with file-system, but the Node.js impl just uses libuv under the hood. Node.js WASI is behind an experimental flag right now.

WasmTime and Wasmer are other runtimes that have their own implementation. WasmTime fs access is quite sandboxed, so it could be slower. Node might follow suit.

Re: Other stuff

It would be interesting to benchmark some hot code paths to see if there are any perf benefits.

Although adding workers could speed things up with pnpm without adding complexity of WASM compilation.

CLI

Maybe it could be used for the CLI. Although I don't think it helps with the cold-start, unless you use a runtime like Wasmer or WasmTime. With Node, you are still loading the entire Node runtime.

Might be an option instead of Rust. Although Rust is more proven at this point.

[jonschlinkert]

I just happened to see this discussion and figured I'd throw https://github.com/folder/readdir into the hat.

FWIW I'm the creator and maintainer of the picomatch/micromatch libs. It's been a while since I worked on picomatch's benchmarks, so I'm not sure how it fairs against globrex, but our main goals with picomatch were to be as accurate and safe (from ReDoSS) as possible. There are likely performance penalties for that approach, but I felt it was worth the tradeoff (returning the wrong result faster doesn't help anyone). I also know the guys that work on globrex and tiny-glob, and they do a great job too.

No worries if another approach makes more sense for you here. Either way, let me know if I can help.