[Bug] Memory exhaustion from high-frequency dsh-subprocess spawning (546 in 27 min, mostly git) on dsh 0.2.2-rc.1

[bwcx-jzy]

Environment

• dsh: 0.2.2-rc.1 (global install), Node v24.14.1
• OS: Ubuntu 26.04, kernel 7.0.0-43-generic
• VM: Proxmox KVM, 12 vCPU, 28 GiB RAM, 8 GiB swap (swapfile on a dedicated NVMe data disk), RTX 2080 Ti passthrough
• Concurrent workloads: several VSCode windows, llama-server, whisper-server, openclaw gateway, docker containers, GNOME desktop

Symptom

The VM freezes repeatedly and becomes unreachable (SSH down, QEMU guest agent stops responding). After one freeze, the next boot's journalctl -b -1 shows:

• 546 Started dsh-subprocess-...scope entries in 27 minutes (~20/min)
• Most are git commands:
  • git rev-parse --show-toplevel
  • git symbolic-ref --quiet --short HEAD
  • git remote -v
  • git rev-parse --path-format=absolute --git-common-dir
  • git rev-parse --short HEAD
  • git rev-parse --abbrev-ref --symbolic-full-name @{upstream}
• Each dsh-subprocess launches a separate node .../dsh-subprocess-local/lib/runner.js process (~140 MB RSS each)
• Right before the freeze, dozens of git subprocesses were spawned within a few seconds, then the journal stops.

Memory state at freeze time (QEMU monitor / guest agent before it stopped responding):

• guest memory: 26.9 GiB / 30 GiB used, free 1.9 GiB
• swap (8 GiB): 100% used
• mem_swapped_out: ~94 GB cumulative, mem_swapped_in: ~32 GB
• QEMU guest agent stops responding; SSH unreachable.

Kernel side (on 7.0.0-38 and 7.0.0-42, before upgrading to 43):

watchdog: BUG: soft lockup - CPU#0 stuck for 26s/49s/.../130s! [kcompactd0:100]
RIP: __pv_queued_spin_lock_slowpath -> pte_offset_map_lock -> remove_migration_pte -> compact_zone -> kcompactd

After upgrading to 7.0.0-43 and setting transparent_hugepage=never + vm.compaction_proactiveness=0, the soft lockup no longer appears, but memory exhaustion still recurs under load.

Questions

1. Is spawning a separate runner.js process for every short git command expected in 0.2.2-rc.1? Could git status detection reuse a persistent runner instead?
2. Are these ~20/min git scans cached or coalesced? Is there a setting to reduce their frequency or disable automatic git/workspace scanning?
3. Is this a regression in 0.2.2-rc.1 compared to 0.2.1-rc.5? Would rolling back avoid it?
4. Is there a recommended concurrency limit for dsh-subprocess on memory-constrained hosts?

Additional context

• ~28 workspaces had been opened previously (workspaceStorage), with many sessions.
• The 546 launches include user-initiated commands, but the burst immediately before the freeze is a batch of git commands that appears automated (workspace/git status refresh).

[bwcx-jzy]

Additional lead from a local investigation (2026-11-08):

• The exact git command set from the 546-spawn burst (rev-parse --path-format=absolute --git-common-dir, symbolic-ref --quiet --short HEAD, rev-parse --abbrev-ref --symbolic-full-name @{upstream}, remote -v, etc.) appears in @deepseek-ai/dsh-experimental-client-ui-forge/lib/index.js. Its inspectCheckout() runs 7 git commands per checkout.
• The Web profile had both @deepseek-ai/dsh-experimental-forge-bundle and the standalone @deepseek-ai/dsh-experimental-tool-worktree enabled together. The forge README says profiles that already selected the standalone worktree bundle should follow the upgrade guide before enabling forge; this profile has that duplicate/conflicting composition.
• With ~28 previously opened workspaces, 28 × 7 × a few refresh passes ≈ 546, which fits the observed count. This makes the forge panel a strong suspect for the subprocess storm, while the 43 kernel / memory-exhaustion behavior remains a separate layer.
• I am disabling the forge bundle for an A/B test and will report whether the git subprocess count drops and whether the SIGSEGV recurs.

[PerryLink]

每次 ctx.subprocess.spawn 都是一个独立的 runner 进程，而且（在 systemd 主机上）还会各自开一个 scope——这就是为什么 546 次会压垮一台 28 GiB 的机器。

机制

packages/subprocess/subprocess-local/src/bin.ts 是 runner 的入口：

if (import.meta.main) {
  const selection = consumeRunnerSelection()
  ...
  void runSelectedSubprocessRunner(selection)
}

也就是说，一次 spawn 不是"直接 exec git"，而是起一个 Node 进程（process.execPath + 这个入口），由它再去 exec 子进程。每个 Node 进程的基线 RSS 就是你看到的那个 ~140 MB 量级；Linux 上还会经过 linux-scope.ts，为每次 spawn 建一个 dsh-subprocess-*.scope——你 journalctl 里 546 条 Started dsh-subprocess-...scope 就是这么来的。

所以 28 × 7 不是"28 次 git"，是 196 个 Node 进程（再乘几次刷新）。即使每个只在几百毫秒内退出，峰值也够把 swap 打穿。

这笔账该记在谁头上

• 大头是那个插件：inspectCheckout() 每个 checkout 跑 7 条 git（rev-parse、symbolic-ref、remote -v…），而这些值在一次会话里几乎不变——缓存一次或按 mtime 失效就够，没必要每次刷新都重跑。
• DSH 这边能改的是"小命令不走 runner"：git 这类只读查询完全可以直连，不需要 Node 中间层。这是性能优化，不是 bug。

先做的一件事：把 @deepseek-ai/dsh-experimental-forge-bundle 和 dsh-experimental-tool-worktree 二选一（forge 的 README 说了两者同开要走升级指南），你现在的组合是两份都在跑。

只读了 master fcc0ad46 的源码；你的 journalctl 数据我没法复算。

[bwcx-jzy]

Thanks — this matches our measurements.

We removed @deepseek-ai/dsh-experimental-forge-bundle at 10:35 and left only the standalone @deepseek-ai/dsh-experimental-tool-worktree, so the "pick one" step is done. However, the dsh web process still SIGSEGV'd at 11:45:11 (PID 107189, signal 11, no core) after forge was removed. That crash is tracked separately in #12596, so forge explains the git-subprocess storm but not the 0.2.2-rc.1 userland crash.

Current state after restart (PID 283947, 0.2.2-rc.1): still running, but dsh RSS is around 2.0 GB and swap is around 1.0 GB; the subprocess count since restart is still nontrivial (though part of it is our own diagnostics). We are rolling back to 0.2.1-rc.5 for the SIGSEGV and will report the result.