Bug: service crashes with ENOENT when the %TEMP% spill directory is removed (Windows)

[SilenceCheems]

Environment

• OS: Windows 11 Pro 23H2 (10.0.22621), x64
• Node.js: v24.16.0
• pnpm: 11.21.0
• deepseek-harness: latest main (0.1.0-rc.5), run from source (pnpm install && pnpm run build && pnpm dsh web)

Description

On Windows, the whole service process crashes with an uncaught ENOENT when
OutputCollector.spillAll() tries to open a spill log file whose parent
directory no longer exists:

node:fs:561
  return binding.open(
                 ^

Error: ENOENT: no such file or directory, open '%TEMP%\dsh-subprocess-<random>\dsh-subprocess-<pid>-1-<random>-stderr.log'
    at openSync (node:fs:561:18)
    at OutputCollector.spillAll (<repo>\packages\subprocess\subprocess-local\src\spawn.ts:169:22)
    at OutputCollector.push (<repo>\packages\subprocess\subprocess-local\src\spawn.ts:134:80)
    at Socket.<anonymous> (<repo>\packages\subprocess\subprocess-local\src\spawn.ts:366:54)
    at Socket.emit (node:events:509:28)
    ...
Node.js v24.16.0
[ELIFECYCLE] Command failed with exit code 1.

The crash happened while a task in the Web UI was running a subprocess whose
stderr output exceeded the in-memory cap, triggering the spill path.

Root cause analysis

1. privateSpillDir() (spawn.ts:90) creates a dsh-subprocess-* directory
   under os.tmpdir() via mkdtempSync. It is a module-level singleton and
   there is no code path that ever deletes it.
2. On Windows, that directory lives in %TEMP%, which system cleanup
   (Storage Sense / disk cleanup / third-party cleaners) periodically purges of
   "unowned" directories. The directory vanished while the service was running.
3. OutputCollector.spillAll() (spawn.ts:169) assumes the directory exists:
   it calls openSync(this.spillFile, 'wx', 0o600) directly, with no existence
   check and no retry.
4. The failure occurs inside the stream.on('data') handler (spawn.ts:366),
   so the exception is uncaught and takes down the entire service process.

Note: this is not just a theoretical race. We observed the crash in practice
~20-30 minutes after starting the service; after the crash, %TEMP% contained
no dsh-subprocess-* directory at all.

Steps to reproduce

1. On Windows, git clone, pnpm install, pnpm run build, pnpm dsh web.
2. Trigger at least one subprocess spawn so mkdtempSync creates
   %TEMP%\dsh-subprocess-* (any Web UI task that runs a command will do).
3. Manually delete the dsh-subprocess-* directory under %TEMP% (or wait
   for system cleanup to remove it).
4. Run a task whose subprocess emits more than the in-memory cap of stderr
   output. The service crashes with the ENOENT above.

Suggested fix

• In OutputCollector.spillAll() (spawn.ts:169), ensure the spill directory
  exists before opening the file — e.g. mkdirSync(this.spillDir, { recursive: true })
  (or catch ENOENT and recreate + retry).
• More robustly, host the spill directory outside the shared system tmpdir
  (application data dir) so external cleanup cannot remove it mid-run.
• Consider wrapping the data handler (spawn.ts:366) so an output-collection
  failure degrades gracefully (drop output) instead of killing the process.

Workaround (used to unblock)

Start the service with a dedicated temp directory:

$env:TEMP = '<repo>\.dsh-tmp'   # any app-owned dir
$env:TMP  = '<repo>\.dsh-tmp'
$env:TMPDIR = '<repo>\.dsh-tmp'
pnpm dsh web

[zoahdev]

Confirmed at source level and implemented as a cherry-pick-ready branch.

Source audit (master 47f9438)

packages/subprocess/subprocess-local/src/spawn.ts:

• privateSpillDir() mints a module-level singleton under os.tmpdir() with mkdtempSync and nothing ever deletes it - exactly the "unowned dir" that Windows Storage Sense / third-party cleaners purge.
• OutputCollector.spillAll() calls openSync(this.spillFile, 'wx', 0o600) with no parent-existence check; the throw escapes from the stream data callback at spawn.ts:366 and kills the whole service.

Patch: fix/subprocess-spill-recreate-on-enoent

https://github.com/zoahdev/deepseek-harness/tree/fix/subprocess-spill-recreate-on-enoent

• spillAll catches ENOENT, recreates the private spill directory (mkdirSync(..., { recursive: true, mode: 0o700 })), and retries the openSync once.
• If recreation still fails, the collector degrades to its in-memory tail (discardSpill) instead of crashing from the stream callback.
• All non-ENOENT open errors keep their previous behavior.

Verification

• Host typecheck (tsc -b tsconfig.host.json) - clean
• Runtime reproduction with a missing spill dir: no throw, in-memory tail intact, spill file recreated with the full stream
• New regression test added to spawn.spec.ts (this suite is excluded on Windows runners and runs in the Linux CI lane, same as the rest of spawn.spec)

Thanks for the exact stack trace - the "directory vanished mid-service" case is exactly the kind of thing that only shows up in the wild.

[SilenceCheems]

Thanks for the quick fix and for digging into the root cause - that's exactly the failure mode we hit. I'll apply the branch locally and test it against the same scenario (purged %TEMP% dir mid-service), then report back here. Appreciate you making it a cherry-pick-ready branch - hope it gets merged into upstream main.

[zoahdev]

Great - that's exactly the scenario to validate against. One note for your local test: the regression test lives in spawn.spec.ts, which the default vitest config excludes on Windows runners (it runs in the Linux CI lane), so locally the quickest check is the runtime path itself: delete %TEMP%\dsh-subprocess-* mid-service, then trigger a subprocess with enough stderr to overflow the in-memory cap. You should see the spill dir recreated and no crash.

If anything still surfaces (e.g. a race between the purge and the openSync retry), paste the stack and I'll tighten the patch.

[SilenceCheems]

Applied the fix branch locally (fast-forwarded to 6fb61d6) and restarted the service - running fine with the fix in place. Will report back after stress-testing the purge scenario. Thanks again!

[Electricitysheep]

%TEMP% spill 目录被删 → ENOENT 崩溃——#260（spill I/O 失败终止进程）同族，磁盘清理工具清了 %TEMP% 就踩到。

临时：把 dsh 的临时目录排除出清理范围（和 #758 Storage Sense 删沙箱临时目录同类处理）。第 12 章 Windows 坑位有记录：https://github.com/Electricitysheep/dsh-handbook/blob/main/docs/12-limitations.md