Context
bigfiles currently uses the ignore crate's parallel walker, which on macOS ultimately calls readdir per directory. macOS provides a faster bulk API: getattrlistbulk(2), which returns metadata for many directory entries in a single syscall.
For large trees (especially ~/Library, node_modules, Xcode derived data) this can meaningfully reduce syscall overhead, since each readdir + stat pair becomes one bulk call.
Goal
Evaluate whether bigfiles should add a macOS fast path using getattrlistbulk, either as a drop-in replacement for the per-directory step inside the walker or as a --fast mode.
Open questions
- Is there a maintained Rust wrapper? If not, we'd need to bind it via the
libc crate.
- How does it interact with the gitignore /
--exclude filtering layer? (Likely fine — we still get paths back, just faster.)
- Benchmarks vs
ignore walker on representative trees (~, a node_modules, Xcode derived data).
- Symlink and hidden-file behavior parity with the current walker.
References
Why this matters
macOS is bigfiles' primary platform today. Any meaningful perf win here directly improves the experience for the largest chunk of current users.
Status
Not started. Tracking for future work; contributions welcome.
Context
bigfiles currently uses the
ignorecrate's parallel walker, which on macOS ultimately callsreaddirper directory. macOS provides a faster bulk API:getattrlistbulk(2), which returns metadata for many directory entries in a single syscall.For large trees (especially
~/Library,node_modules, Xcode derived data) this can meaningfully reduce syscall overhead, since eachreaddir+statpair becomes one bulk call.Goal
Evaluate whether bigfiles should add a macOS fast path using
getattrlistbulk, either as a drop-in replacement for the per-directory step inside the walker or as a--fastmode.Open questions
libccrate.--excludefiltering layer? (Likely fine — we still get paths back, just faster.)ignorewalker on representative trees (~, anode_modules, Xcode derived data).References
walkdirissue discussing similar tradeoffs: FilterEntry::filter_entry has a suboptimal signature BurntSushi/walkdir#130Why this matters
macOS is bigfiles' primary platform today. Any meaningful perf win here directly improves the experience for the largest chunk of current users.
Status
Not started. Tracking for future work; contributions welcome.