Avoid per-entry Windows metadata queries - #359
Merged
Merged
Conversation
Byron
changed the title
Windows directory iteration already returns entry metadata without another system call, but completion-order traversal split every listing into small stat jobs and handed those entries across worker queues. A wide-directory regression demonstrated that eight workers redundantly participated in metadata conversion for one Windows directory. Convert Windows entries on the worker enumerating their parent and schedule only accepted child-directory reads for parallel work. Retain stealable stat batches on other platforms, where DirEntry::metadata may issue a separate symlink-metadata call. This follows the gix-status filesystem cache baseline, which likewise consumes metadata returned by Windows directory enumeration instead of issuing per-path metadata queries. Validated with the focused Windows scheduling regression, cargo +nightly check --all --bins --tests --examples, cargo +nightly test --all, formatting, and a Linux-target clippy build. The generic Windows clippy command still reports pre-existing warnings in crossdev.rs and inodefilter.rs; Linux clippy likewise reaches a pre-existing warning in interactive/app/common.rs.
Read FILE_ID_BOTH_DIR_INFO records directly for internal Windows walks so logical size, allocated size, modification time, file type, and file ID arrive with directory enumeration. Keep the public WalkEntry API unchanged and report enumeration failures instead of falling back to path-based stats. On this worktree, default 'dua ..' drops from about 25.9 seconds to 0.86 seconds while preserving hard-link and reparse-point handling.
Keep the Windows nightly clippy job passing with the current lint set.
Factor the non-Windows stat-reader function pointer into a type alias so the Linux Clippy job accepts the platform-specific pool abstraction.
Use one concrete entry type per target instead of carrying a generic pool, reader dispatch table, and a marker trait through the traversal engine. On Windows the native directory entry becomes the public walk entry, so both public and internal traversal use the metadata returned by directory enumeration. This removes the duplicate std::fs Windows reader and keeps the Windows implementation isolated in walk/windows.rs.
Byron
force-pushed
the
windows-performance
branch
from
August 3, 2026 09:19
f7276b7 to
f94fe6d
Compare
Byron
marked this pull request as ready for review
August 3, 2026 09:45
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d784e024a6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tasks
This section is for Byron only. Models continuing this PR must not add, remove, check, uncheck, rename, or reorder checkboxes here.
Everything below this line was generated by Codex.
Created by Codex on behalf of Byron. Byron will review before this is ready to merge.
Fixes #302
Reported issue
On Windows, the parallel walker should obtain metadata directly from directory enumeration, following the optimization used by the local
gix-status/src/fscache.rs, and avoid per-entrystat-like path queries.Behavioral summary
Both public and internal walks now consume
FILE_ID_BOTH_DIR_INFOrecords directly on Windows. Each record supplies logical size, allocation size, modification time, file type, reparse tag, and file ID without reconstructing the path or issuing a metadata call for that entry.--apparent-sizeuses the enumerated end-of-file value.WalkEntry,WalkFileType, andWalkMetadata; this intentionally changes the platform's public concrete types while retaining the commonly usedpath,is_dir,is_file,is_symlink,len, andmodifiedoperations.std::fsentry metadata path.Performance
Measured from this worktree by alternating three release runs of
dua ../for each build:mainThe branch is about 34.5× faster than
mainon this scan.Implementation
GetFileInformationByHandleEx(FileIdBothDirectoryInfo)with a 64 KiB aligned buffer.std::fsWindows reader.walk/windows.rs.filesizeonly on non-Windows targets because Windows no longer calls it.Validation
cargo +nightly check --all-targetscargo +nightly test --all(94 tests passed)cargo +nightly fmt --all -- --checkcargo +nightly clippy -- -D warningscargo +nightly doc --no-depstui-crossplatformandtrash-move.6cff20884079bdf9c0ea86f069cc48167f05675f,c9166c23080798c8158ca7208da50d8baf8088e3,352af73aa259032624c854dbf3a8ca22e1bd949c,df631a9cf3d32b8b63c9db65c959ac41fead5aed, andf7276b7b4903427d5c20db7fdaf1b72e6440a90e.