Skip to content

Investigate getattrlistbulk on macOS / APFS #2

Description

@Par-python

Context

bigfiles currently uses the ignore crate's parallel walker, which on macOS ultimately calls readdir per directory. macOS provides a faster bulk API: getattrlistbulk(2), which returns metadata for many directory entries in a single syscall.

For large trees (especially ~/Library, node_modules, Xcode derived data) this can meaningfully reduce syscall overhead, since each readdir + stat pair becomes one bulk call.

Goal

Evaluate whether bigfiles should add a macOS fast path using getattrlistbulk, either as a drop-in replacement for the per-directory step inside the walker or as a --fast mode.

Open questions

  • Is there a maintained Rust wrapper? If not, we'd need to bind it via the libc crate.
  • How does it interact with the gitignore / --exclude filtering layer? (Likely fine — we still get paths back, just faster.)
  • Benchmarks vs ignore walker on representative trees (~, a node_modules, Xcode derived data).
  • Symlink and hidden-file behavior parity with the current walker.

References

Why this matters

macOS is bigfiles' primary platform today. Any meaningful perf win here directly improves the experience for the largest chunk of current users.

Status

Not started. Tracking for future work; contributions welcome.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions