Skip to content

perf(skills): reuse walk inventory for host loading#31566

Merged
jif-oai merged 10 commits into
mainfrom
jif/reuse-skill-walk-inventory
Jul 9, 2026
Merged

perf(skills): reuse walk inventory for host loading#31566
jif-oai merged 10 commits into
mainfrom
jif/reuse-skill-walk-inventory

Conversation

@jif-oai

@jif-oai jif-oai commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Why

Thread startup loads repository skills through the primary remote environment. This still used the older host skill loader, which recursively issued directory and metadata requests over the exec-server connection. On a high-latency connection, one skill scan could turn into hundreds of round trips.

Executor-selected skill roots already avoid this pattern: they use one bounded filesystem walk, reuse the returned inventory, and overlap independent reads.

What changed

  • Extract the URI-native walk inventory into a shared private discovery module.
  • Use that discovery path for both environment-owned and host/repository skill roots.
  • Reuse the walk inventory to avoid per-skill metadata existence checks when the inventory is complete.
  • Canonicalize host skill identities and parse skill files with bounded concurrency.
  • Prune hidden host directories during traversal, preserving visible aliases and walk limits.
  • Cache namespace probes by ancestor and keep all remote namespace reads under the same concurrency bound.
  • Keep safe metadata probes for incomplete walks, file symlinks, and case aliases.

Dependency

#31570 is now merged. It provides the optional pruneHiddenDirectories walk flag used here. That flag defaults to false and is omitted when disabled.

No other exec-server API changes are introduced.

@jif-oai jif-oai force-pushed the jif/reuse-skill-walk-inventory branch from c84e3ab to c429509 Compare July 8, 2026 12:36
@jif-oai jif-oai changed the base branch from main to jif/fs-walk-hidden-pruning July 8, 2026 12:36
jif-oai added a commit that referenced this pull request Jul 8, 2026
Why

Filtering hidden directories after a walk is too late: their descendants
consume traversal limits, and canonical directory deduplication can let
a hidden path claim a target before a visible symlink reaches it.

What this changes

- Add an optional pruneHiddenDirectories walk option. It defaults to
false and is omitted from the wire when disabled.
- Return hidden directory entries, but do not traverse them or add their
canonical identities to the visited set.
- Cover the visible-symlink-to-hidden-directory case through both local
and remote filesystem implementations.

This is the small filesystem prerequisite for #31566. Skill-specific
behavior remains in that PR.
Base automatically changed from jif/fs-walk-hidden-pruning to main July 8, 2026 12:49
@jif-oai jif-oai force-pushed the jif/reuse-skill-walk-inventory branch from c429509 to 04aa595 Compare July 8, 2026 12:50
@jif-oai jif-oai marked this pull request as ready for review July 8, 2026 12:50
@jif-oai jif-oai requested a review from a team as a code owner July 8, 2026 12:57
Comment thread codex-rs/core-skills/src/loader.rs
Comment thread codex-rs/core-skills/tests/environment_loader.rs
Comment thread codex-rs/core-skills/src/loader.rs
Comment thread codex-rs/core-skills/src/loader/environment.rs
@jif-oai jif-oai merged commit e8eb600 into main Jul 9, 2026
33 checks passed
@jif-oai jif-oai deleted the jif/reuse-skill-walk-inventory branch July 9, 2026 08:54
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 9, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants