Is there an API to easily collect the results of a parallel WalkBuilder?

[vjpr]

I want to build a Node.js native addon (with napi-rs) to search a directory for some files that match a pattern.

Is there an API that takes a glob pattern and collects results for the parallel builder? For example, jwalk has a nice API on top of its parallel reader.

I've seen https://github.com/BurntSushi/ripgrep/blob/master/crates/ignore/examples/walk.rs, but wondering if there is an API I can depend on instead...and I'm not sure the best way to collect results from crossbeam.

I've also found https://github.com/sharkdp/fd/blob/master/src/walk.rs#L52 too from the fd tool, but again looks too complicated, and would rather rely on a higher-level API.

Also, when should walkdir be used, and when should ignore be used? Seems like there is some overlap...

[BurntSushi]

Is there an API that takes a glob pattern and collects results for the parallel builder?

No, it doesn't exist.

For example, jwalk has a nice API on top of its parallel reader.

jwalk is built on top of ignore. So you might look at its implementation for ideas.

but again looks too complicated, and would rather rely on a higher-level API

You or someone else would need to build it. I'm not sure what it would look like.

Also, when should walkdir be used, and when should ignore be used? Seems like there is some overlap...

Of course there's overlap!

walkdir is your "classic" production ready recursive directory iterator. It's single threaded, perfectly dumb and gives options for tightly controlling resources and provides all of the options one comes to expect from a recursive directory iterator.

ignore is something entirely different. It is also a recursive directory iterator, but it provides an option to take advantage of parallelism and to allow various types of glob filters to influence the results returned to the user.

---

The problem you're facing is that the ignore crate is, approximately, my "shitty first draft" of what such a crate would look like. It works well for ripgrep's purposes, but its API is a complex disaster IMO. I don't feel too bad about it though, because what it's actually doing is also complex. Implementing the ignore rules in a way that is both approximately correct and efficient was definitely one of the harder parts of building ripgrep.

In theory, I have plans for rewriting the ignore crate and learning from my mistakes. But I've been saying that for years. I don't have a lot of time any more and I've been dedicating most of it to improving the regex crate.

[vjpr]

Thanks for the clarity.

jwalk doesn't seem to depend on ignore.

Found a way to do it in the jwalk dir:

https://github.com/jessegrosjean/jwalk/blob/9575132b76513bea64405a7cc5a98708d31d5743/benches/walk_benchmark.rs#L129-L149

Appreciate your work on these packages!

[BurntSushi]

Ah my bad, I was thinking about globwalk.