[Bug][Windows] skill-filesystem watcher can hit a high-CPU rename-event storm on a custom Codex skills root

[lyd123qw2008]

Summary

On Windows, a DSH Web process consumed significant CPU while idle (no UI interaction and no active model call). I traced it to the @deepseek-ai/dsh-skill-filesystem Chokidar watcher after configuring a custom skills root at:

C:\Users\<redacted>\.codex\skills

This is not a default DSH skills path; it was added through customSkillDirs. The direct child \.system emitted a very high rate of native Windows rename notifications, causing repeated Chokidar directory scans and lstat calls on DSH's main process.

Environment

• DeepSeek Harness: 0.1.5-alpha.2 (a second 0.1.5-alpha.1 instance was configured with the same custom root)
• OS: Windows 11 Enterprise LTSC 24H2, build 26100, x64
• Node.js: v24.13.0
• Chokidar: 5.0.0
• Web process: dsh web

Evidence captured during the spike

1. CPU

   • In a 3-second sample, the DSH process accumulated 4.52 CPU seconds, about 18.8% of an 8-logical-processor machine.
   • The hot work was in the DSH Node process, not in the Engram/MCP/browser child processes.

2. Native watcher-event rate

   • Instrumenting the affected native FSWatcher._handle.onchange callback for 3 seconds recorded 109,487 events (about 36k/s).

   • The events were repeatedly reported as rename for:

     \\?\C:\Users\<redacted>\.codex\skills\.system
     

   • During a short metadata check, the directory's LastWriteTime remained unchanged, so this looked like repeated watcher notifications rather than a sustained ordinary file update.

3. V8 CPU profile

   The hot stacks were consistent with Chokidar repeatedly rescanning that directory:

   FSWatcher._handle.onchange
     -> chokidar/handler.js handleEvent
     -> NodeFsHandler._handleRead
     -> chokidar._readdirp / readdirp
     -> readdir / lstat
   

   Other sampled hot functions were node:path normalization/join/relative and Chokidar's event throttling.

4. A/B mitigation

   • I closed only the affected in-memory Chokidar watcher for this one running DSH process; no skills/configuration/files were deleted or changed.
   • The next 5-second sample used 0.047 CPU seconds, approximately 0.12% of the same machine.

Why this seems avoidable in DSH

packages/skill/skill-filesystem/src/index.ts opens a depth-1 Chokidar watcher for each root in openRootWatcher() (currently without an ignored filter).

For this custom root, .system is an immediate child, so it is watched. However, DSH's current direct-root discovery looks for either a top-level *.md file or <top-level-directory>/SKILL.md; this .system container has no direct .system/SKILL.md (it contains nested system-skill folders). Therefore it appears to be an irrelevant subtree for the current catalog, while it can still trigger repeated scans.

Expected behavior

A custom skills root should not allow unrelated metadata/system directories or pathological native watcher notifications to consume a sustained amount of DSH CPU when the agent is otherwise idle.

Possible fixes / hardening ideas

Any of these would help:

1. Expose a safe ignore configuration for skill-filesystem, and/or ignore non-skill metadata directories such as .git; carefully handle .system only where it cannot be a valid direct skill.
2. Watch only catalog-relevant paths: the root for newly added entries, top-level *.md, and <candidate>/SKILL.md, rather than recursively watching every immediate subtree.
3. Add Windows-side debounce/backoff/coalescing for repeated raw directory events before scheduling another readdirp scan.
4. Add observability/logging for raw watcher-event rate and per-root rescans, so this kind of issue is diagnosable without attaching a profiler.

I do not suggest blindly ignoring every hidden directory, since deployments may intentionally use one. The main request is to avoid fully watching a subtree that cannot currently contribute a discoverable direct skill, or to make that behavior configurable.

Current workaround

Use a curated skills directory containing only actual DSH-compatible skill entries instead of pointing customSkillDirs at a whole externally managed skills tree, or disable live skill watching for that root until a fix is available.

I can provide a minimal reproduction / more profiler data if that would be useful.

[argszero]

Thanks for the detailed trace — the static claim holds up, and there is one more relevant fact in the tree that sharpens the scope of a fix.

Confirmed at source (packages/skill/skill-filesystem/src/index.ts):

• openRootWatcher() (:488-502) calls chokidar.watch(mode.anchor, { depth: 1, atomic: true, ... }) with no ignored matcher.
• With depth: 1, Chokidar registers a native watcher for the root and for every immediate child directory, then watches their direct children. Reproduced against the pinned chokidar@5.0.0 on a synthetic root: a write to <child>/<grandchild>/file.md (e.g. .system/nested/new.md) emits change, and creating <child>/<newdir> emits addDir. So each event for that subtree buys an lstat/rescan cycle on the DSH process, which is exactly the shape of your profile.
• Your reading of discovery is also right: discoverRoot() -> isPotentialSkillPath() admits only <root>/<name>.md and <root>/<name>/SKILL.md (:677-684), so a .system container with nested system dirs cannot contribute a catalog entry.

The missing piece: the repository already has this concept. SkillRoot carries skipSystem (:95), consulted in three places — discovery (:723), isRelevantWatchEvent() (:666) and isPotentialSkillPath() (:680). It is set for exactly one root today: the user DSH root (~/.dsh/skills, :253), documented as "The user DSH root skips its .system child" (README.md:54). Your custom root is the same shape — a Codex skills root whose .system child holds nested system skills — so the intent already exists and is simply not applied to customSkillDirs (:250). Extending the skipSystem contract to configured custom roots is a much smaller change than a new ignore feature, and it answers your own caveat: .system is a declared concept here, not "ignore every hidden directory".

For a non-skipSystem root the .system guard in isRelevantWatchEvent is unreachable by construction — the watcher has no filter, and skipSystem is precisely what gates the handler. So the filter has to be applied at watcher construction, not only in the event handler.

One trap worth avoiding while you are in there. A fix that ignores depth-1 directories that do not currently contain a SKILL.md looks like the natural "watch only catalog-relevant paths" implementation, and it silently breaks the primary hot-refresh flow. Ran against chokidar 5.0.0: mkdir <root>/late-skill produces no event; then writing <root>/late-skill/SKILL.md still produces no event (the directory was never watched, and changes under an ignored path do not trigger a parent rescan); the new skill appears only when some unrelated root-level event later causes a rescan. ignored therefore has to stay a pure name/path matcher — never a function of current catalog membership — or the very first mkdir-then-populate skill creation goes unnoticed.

ignored is still supported in Chokidar 5 (index.d.ts:29; FSWatcher._isIgnored at index.js:657 builds an anymatch over the list, and WatchHelper.filterPath/filterDir at :210/:213 use it to prune directories). Verified: ignored: [/[/\\]\.system([/\\]|$)/] keeps .system out of getWatched() and yields zero events from both probes.

Minor evidence note: rename is the generic libuv/Windows channel name for a change to a watched handle — FSWatcher._handle.onchange reports it for adds and modifications too — so 109,487 rename events does not imply renames; it is the raw channel name.

On attribution: the A/B you describe cannot separate the two hypotheses — closing the in-memory watcher removes both the subscription and the storm, so a 96x CPU drop is equally consistent with an external process generating the notifications. Worth a control before committing to a mitigation: with DSH stopped, attach a bare fs.watch (no Chokidar) to that path for a few seconds. If the rate persists, the producer is external and the DSH-side work is containment only; if it collapses, the producer is inside the process. Either way your items 3 (debounce/coalesce raw rename bursts) and 4 (raw event-rate observability) stand on their own — they are a different work item from the ignore filter, and they are the only defence if the producer turns out to be external.

Scope note: this is provider-side (config schema + watcher construction) inside the first-party package, so a downstream plugin cannot patch it — a plugin cannot inject an ignored matcher into the built-in provider's watcher, and registering a second provider over the same roots would only duplicate catalog entries.

[lyd123qw2008]

Follow-up from the reporter: I implemented and committed a local non-polling containment patch in the RC.1 source checkout as b7c3b2630c (fix(skill-filesystem): recover from Windows watcher bursts). This is a local commit, not an upstream PR.

I deliberately did not extend skipSystem / ignored to the shared Codex root. The user needs to retain the shared .system directory, and the local mitigation should not silently opt that subtree out of future direct-skill behavior. It preserves native fs.watch, watchUsePolling: false, and normal skill hot reload.

Local containment behavior

For Windows native watchers only, the provider now counts Chokidar raw rename notifications by its underlying watchedPath:

> 256 raw rename events for one directory within 1000 ms
  -> close only that skill root's Chokidar watcher
  -> wait 2000 ms
  -> reopen the normal native watcher
  -> invalidate the catalog once

The defaults are configurable as:

watchNativeBurstLimit: 256       # 0 disables this guard
watchNativeBurstWindowMs: 1000
watchNativeRecoveryMs: 2000

This addresses the expensive part after a storm starts: closing the watcher destroys active readdirp streams and prevents Chokidar from continuing its directory rescan/retry cycle. It is intentionally containment rather than a claim about who produced the notifications.

I also checked the post-update steady state: a 5-second recursive snapshot of C:\\Users\\liuyd\\.codex\\skills showed no file/timestamp/size changes, and a 5-second raw watcher sample was quiet. That still fits a finite Codex system-skill bootstrap or queued Windows notifications, amplified by Chokidar's rescan path.

Validation completed:

• focused skill-filesystem Vitest: 32 passed;
• package TypeScript build: passed;
• Oxlint: passed;
• pnpm run build:lib:host: passed.

The initial recovery delay is deliberately fixed and conservative because the observed event source is now quiet. If an immediate reopen proves to retrigger the burst repeatedly, the next focused change will be per-root exponential recovery backoff rather than switching to polling or ignoring .system.

[lyd123qw2008]

Source update: the tested patch is now pushed to my fork, not only present in the local checkout.

• Branch: https://github.com/lyd123qw2008/deepseek-harness/tree/upgrade/dsh-v0.1.5-rc.1
• Commit: lyd123qw2008@b7c3b26

The branch contains the Windows-only raw-rename burst recovery described above, plus its watcher and configuration-validation tests. It deliberately retains native watching and the shared .system tree rather than switching to polling or applying skipSystem to the custom root.

Before the push I reran the repository's full pnpm run typecheck gate successfully (via Corepack on this Windows host).