[Bug] Windows: boot-time corpus scan exhausts process fd limit (EMFILE) and leaves a permanent empty .lock that blocks every later boot

[loalissm-cell]

[Bug] Windows: startup session-corpus scan exhausts the process fd limit (EMFILE), and the failure leaves a permanent empty .lock that blocks every later boot

Environment

Item	Value
Package	@deepseek-ai/dsh@0.2.1-alpha.2 (npm global)
Launch	dsh web (Node v24.18.0, Windows 11 x64)
DSH_HOME	C:\Users\chkpk\.dsh
Session corpus	~2,129 files under sessions\, ~8,268 files under storages\
Antivirus	Huorong (火绒) real-time protection, paths now trusted

Summary

Two coupled problems, both reproduced deterministically today on one real machine. Neither is caused by a misbehaving third-party plugin: an empty-DSH_HOME control boot with the same full web profile plugin tree starts and stays alive, while the real corpus boot dies ~4 s in.

1. The boot-time session-corpus scan opens more file descriptors than the Node/libuv process limit (~8,189 on this machine), so a required plugin dies with EMFILE and the whole backend fails to start. The failure point observed in 4/4 boots is client-connection (required) writing the credentials file lock.
2. That EMFILE death leaves behind a 0-byte .credentials.yaml.lock that nothing ever cleans up, and because it contains no holder PID it can never be taken over — every subsequent boot times out at the 2 s writer-lock deadline ("started but won't open"), until an operator deletes the file by hand.

Reproduction

1. Grow a real $DSH_HOME to a few thousand session artifacts (mine: 2,129 log files under sessions\, 8,268 under storages\).
2. Run dsh web and watch it fail within ~4 seconds.
3. Retry booting several times: every attempt fails again, now with atomic-write: timed out waiting for the writer lock at C:\Users\chkpk\.dsh\.credentials.yaml.lock.
4. Delete .credentials.yaml.lock manually; the next boot reproduces step 2 (EMFILE) and recreates the empty lock.

Control experiment (rules out the plugin tree):

# Same profile directory copied, but DSH_HOME pointed at an empty directory:
DSH_HOME=<empty dir> dsh web   =>  alive after 45 s, no EMFILE
DSH_HOME=<real corpus> dsh web =>  exitCode=1 within ~4 s, EMFILE at the lock write, 4/4 boots

Evidence

(a) Startup diagnostics (full diagnostic file kept, credentials redacted)

StartupError: dsh: startup failed: 1 required plugin did not activate
Failed plugins (1):
  connection (required)
    Package: @deepseek-ai/dsh-client-connection
    Error: EMFILE: too many open files, open 'C:\Users\chkpk\.dsh\.credentials.yaml.lock'
        at async open (node:internal/fs/promises:640:25)
        at async writeFile (node:internal/fs/promises:1260:14)
        at async withFileLock (.../@deepseek-ai/dsh-atomic-write/lib/index.js:195:4)

Plugins waiting for services (8):
  session-log-download  connection
  open-in-app          connection
  directory-picker     connection
  session-controller   fileUploads
  schedule             sessionController
  file-upload          connection
  ui-deliverables      connection, sessionController
  dsh-pocket           connection

(b) The process fd limit on this machine (measured, not assumed)

// node -e: open() in a loop until EMFILE
opened 2000 … opened 8000
max = 8189   // EMFILE after 8,189 open fds (libuv default)

(c) The lock file never receives its holder PID

dsh-atomic-write's withFileLock creates the lock with flag: "wx" and writes `${process.pid}\n` after creation. When the process dies from EMFILE between those two steps, the lock exists but is empty:

lock created at t=3870 ms
after 30 s: len=0, content=[]

takeOverExitedLock() cannot recover it: it reads the lock record to learn the holder PID, but an empty record has no PID to probe, so the takeover path never fires. The comment in the source says "A lock records its holder's PID" — in this failure mode it demonstrably does not.

Kill-test (3 runs): every Stop-Process of a boot leaves len=0 behind; every subsequent contender waits, times out after DEFAULT_LOCK_WAIT_MS (2 s), and the cycle repeats.

run 1: killed → lock remains, len=0
run 2: killed → lock remains, len=0
run 3: killed → lock remains, len=0

(d) Timeline match

The lock appears at t≈3.9–4.0 s and the EMFILE happens on the lock write — i.e. the corpus scan has already consumed the process's entire fd budget before any plugin needs a single fd for itself.

Suggested fixes

For (2), the lock itself (small, self-contained, fixes "boot fails forever after one bad boot"):

• Write the PID via a method that cannot be split from creation — e.g. create the lock content in the same operation, or treat a wx-created-but-empty lock older than a small grace window as abandoned and removable.
• On any error path after wx succeeds, remove the lock in a finally/catch before propagating.
• Alternatively, when takeOverExitedLock() finds an empty record, treat it as a dead holder after the grace window instead of waiting forever.

For (1), the fd exhaustion (larger, but the machine-independent part is clear):

• The boot-time corpus walk should not hold an fd per artifact; stream/close or batch the scan, or bound concurrent open files during boot.
• Boot should degrade (log a warning, serve a degraded session list) rather than fail the entire web backend because one required plugin could not allocate one fd.

Notes

• This machine also hit TCP ephemeral-port exhaustion (System event 4227) at the same time as the EMFILE cascade — the unresponsive-everywhere symptom that triggers a full OS reboot. Included for completeness; the port exhaustion is a co-occurring symptom, not claimed as part of this bug.
• I can provide the full startup diagnostic JSON (with credentials redacted) and a 30 s-granularity handle/TIME_WAIT timeline captured by a local read-only sentinel on request.

[eamars]

We hit the same failure independently on 0.2.1-rc.2 (same as current master, 687f282e86), Windows 11, Node v24.18.0: EMFILE on .credentials.yaml.lock at boot, then a 0-byte lock that made every later boot time out until it was deleted by hand. Same measured limit here too: a Node process holds 8,189 open files, the next open fails with EMFILE.

Some additional findings that may help narrow it down:

1. The "corpus scan" is the session_projcache per-record table load. packages/storage/storage-json/src/per-record-unit.ts, loadTableRecords (around line 167), reads every record of a table at once:

const loaded = await Promise.all(files.map(async (file) => { ... await readRecord(join(dir, file.name), versions) ... }))

Our <DSH_HOME>/storages/session_projcache/sessions/ held 8,780 files (your storages\ count of ~8,268 is likely the same table). While it loads, the number of simultaneously open files can exceed the limit, so whatever else opens a file at that moment fails, here the credentials lock. In our case it was timing-dependent: once the table passed ~8,189 records, some boots failed and some succeeded.

Also, readRecord swallows every error (catch { return undefined }), so records that themselves hit EMFILE during the load are silently treated as absent.

2. On Windows the empty lock appears even without the process dying. An exclusive create (flag: 'wx') that fails with EMFILE has already created the file: the error comes back, but a 0-byte file is left behind. Minimal check (fill the fd table, then try a wx create):

// node check.mjs <empty temp dir>
import fs from 'node:fs/promises'; import path from 'node:path'; import os from 'node:os';
const dir = await fs.mkdtemp(path.join(process.argv[2] || os.tmpdir(), 'emfile-')); const handles = [];
try {
  for (let i = 0; ; i++) { try { handles.push(await fs.open(path.join(dir, 'f' + i), 'w')); } catch (e) { console.log('held', handles.length, e.code); break; } }
  const lock = path.join(dir, 'x.lock');
  try { await fs.writeFile(lock, process.pid + '\n', { flag: 'wx' }); } catch (e) { const s = await fs.stat(lock).catch(() => null); console.log('wx', e.code, 'left behind:', s ? s.size + ' bytes' : 'nothing'); }
} finally { for (const h of handles) await h.close().catch(() => {}); await fs.rm(dir, { recursive: true, force: true }); }

Output here: held 8189 EMFILE then wx EMFILE left behind: 0 bytes.

In withFileLock (packages/util/atomic-write/src/index.ts, around line 250), EMFILE is not a contention error, so it is rethrown; the finally that removes the lock only runs after the lock was acquired, so the empty file stays. holderExited (around line 112) deliberately never takes over a lock without a <pid>\n record, so it stays forever.

3. Why the table grows. A session_projcache record is not removed when its session leaves the session store. In our setup a plugin moves finished one-shot sessions out of the store (there is no delete-session API), so ~90% of the 8,780 records belonged to sessions no longer in the store. Removing those records before boot (with no DSH process running) brought the table to 837 files and the boots back to normal. This is only a mitigation; any per-record table that grows past ~8,000 files can trigger it.

Fixes that would cover both sides, in line with your suggestions:

• bound the concurrency in loadTableRecords (batches, or a fixed limit such as 64), and don't treat EMFILE in readRecord as "record absent";
• in withFileLock, treat a lock that has stayed empty past a short grace window as abandoned (an honest writer writes its PID right after creating it), or remove an empty lock left by a failed create before rethrowing;
• optionally, drop a session's session_projcache record when the session leaves the store.