[Bug] Terminal background-job records are never reclaimed in a live session: unbounded store growth, O(n) list/start scans, and a silent memory footprint that grows with session age

[yamingmou]

EN / English

Observed behavior

In the local background-job registry (dsh-jobs-local), a job record is written once on start() and never removed while its owning agent is alive. When a job reaches a terminal state (completed / failed / cancelled / stopped), settle() updates its status, detail, output, and finishedAt — but the record stays in the store indefinitely, including the full output string, until the owning agent is disposed or the jobs service is torn down.

In a long-lived session that repeatedly starts background jobs (test suites, batch processing, deployments), the store therefore grows without bound:

• every terminal job (with its output payload) is retained for the whole session lifetime,
• job_list and start() scan the entire store ([...store.values()]), so their cost grows linearly with session age,
• the memory footprint grows monotonically and silently — no warning, no retention, no LRU.

Root-cause (store lifecycle, with source references)

Verified on @deepseek-ai/dsh-jobs-local@0.1.1-rc.2 (lib/index.js):

• :165 — this.store.set(id, job) on start(): the record enters the store.
• :365-387 — settle(): only sets status / detail / output / finishedAt; never deletes the record. The terminal output string stays resident.
• :411 — this.store.delete(job.id) appears only inside disposeOwned(owner) (the owning agent's lifecycle end).
• :424 — this.store.clear() appears only inside disposeAll() (the jobs service's teardown).
• :394-405 — ensureOwnerCleanup() (added in this release): registers an owner-scoped effect that calls disposeOwned when the agent is disposed — i.e. cleanup is keyed to the owner's lifecycle boundary, not to the job's state boundary (terminal).
• :137 — every start() calls activeTaskCount() which scans the store (O(store.size)).
• :180 — list() scans the full store: [...this.store.values()].
• :102 — maxConcurrentJobsPerOwner (default 10) is checked only against active jobs (:137), so it bounds concurrency, not accumulation — a false sense of "bounded".

The cleanup trigger is misaligned: reclamation depends on a lifecycle boundary (owner disposed / service teardown) instead of a state boundary (job terminal + reported), so within a live session the store only ever grows.

Impact scope (production / progress / scheduling)

• Session layer: the job store is per-process state owned by the session's agents; a long-running automation session (batch background jobs, CI-style runs, deployments) accumulates terminal records — each carrying its output payload — for the entire session lifetime. Memory grows monotonically and silently.
• Progress layer: the accumulated terminal records are the audit trail of the session's background work — but they are never pruned, so the cost of reading them (job_list) and of starting new jobs (activeTaskCount) grows linearly with how much work the session has done. The longer and more productive the session, the slower it becomes.
• Scheduling layer: this is exactly the scenario for scheduled/queued workloads: a session that starts jobs on a schedule or in a loop accumulates records with no upper bound; nothing signals the operator until memory pressure or visible latency.
• Contrast with the design intent: maxConcurrentJobsPerOwner and the new ensureOwnerCleanup show the engine does manage job lifecycle — but only at the concurrency and owner boundaries. The terminal state, which is the earliest safe point for reclamation, is not used as a trigger.

Expected behavior

• Terminal + reported jobs should be reclaimed (or moved to a bounded, evictable history) — a retention bound on terminal records, independent of session lifetime.
• job_list / start() should not pay O(store.size) — an index by owner (or a bounded terminal-history list) keeps them constant-time.
• The store's growth should be observable: a count/bytes metric and a warning threshold, so silent monotonic growth becomes visible.

Repro

1. In one session, start a background job (e.g. via run_in_background / the jobs tool) and let it finish.
2. Repeat in a loop — the job_list result and the store grow on every iteration.
3. Observe: terminal jobs (with their output) remain in memory for the whole session; job_list and each new start() get slower as the count grows; no retention or warning exists.

Environment

• Engine: @deepseek-ai/dsh 0.1.1-rc.2 (also reproduced in 0.1.0-rc.7)
• Module: @deepseek-ai/dsh-jobs-local (lib/index.js)

Evidence (source)

:165   this.store.set(id, job)                     // written once, at start
:365   settle(job, outcome) {                     // terminal transition…
:369     job.output = outcome.output;             // …keeps the full output resident
:366     if (isTerminal(job.status)) return;      // idempotent, still no delete
:411   for (const job of owned) this.store.delete(job.id);   // only on owner disposal
:424   this.store.clear();                        // only on service teardown
:394   ensureOwnerCleanup(owner)                  // cleanup keyed to owner lifecycle
:137   if (this.activeTaskCount(spec.owner) >= …) // O(store.size) on every start()
:180   return [...this.store.values()].filter(…)  // O(store.size) on every list()

---

中文版 / ZH

现象

本地后台任务注册表（dsh-jobs-local）中，job 记录在 start() 时写入一次，在其 owner agent 存活期间从不移除。当 job 进入终态（completed / failed / cancelled / stopped）时，settle() 只更新 status、detail、output、finishedAt——记录（连同完整 output 字符串）会一直留在 store 里，直到 owner agent 被销毁或 jobs 服务被整体拆除。

因此，在一个反复启动后台任务的长会话中（测试套件、批量处理、部署），store 无界增长：

• 每个终态 job（连同其 output 载荷）在整个会话生命周期内驻留；
• job_list 与 start() 全表扫描（[...store.values()]），成本随会话时长线性上升；
• 内存占用单调、静默增长——无警告、无 retention、无 LRU。

根因（store 生命周期，附源码引用）

在 @deepseek-ai/dsh-jobs-local@0.1.1-rc.2（lib/index.js）上验证：

• :165 —— start() 时 this.store.set(id, job)：记录进入 store。
• :365-387 —— settle()：只设置 status / detail / output / finishedAt；从不删除记录。终态 output 字符串持续驻留。
• :411 —— this.store.delete(job.id) 只出现在 disposeOwned(owner)（owner agent 生命周期结束）内。
• :424 —— this.store.clear() 只出现在 disposeAll()（jobs 服务 teardown）内。
• :394-405 —— ensureOwnerCleanup()（本版本新增）：注册 owner 作用域 effect，在 agent 被销毁时调用 disposeOwned——即清理绑定在 owner 的生命周期边界，而非 job 的状态边界（终态）。
• :137 —— 每次 start() 调用 activeTaskCount()，全表扫描（O(store.size)）。
• :180 —— list() 全表扫描：[...this.store.values()]。
• :102 —— maxConcurrentJobsPerOwner（默认 10）只对活跃 job 校验（:137）——它约束并发，不约束累积，造成"有界"的错觉。

清理触发条件错位：回收依赖生命周期边界（owner 销毁 / 服务 teardown），而非状态边界（job 终态 + 已读），因此在存活的会话内 store 只会增长。

影响范围（生产 / 进度 / 调度）

• 会话层： job store 是会话的 agent 持有的进程内状态；长跑自动化会话（批量后台任务、类 CI 运行、部署）在整个会话生命周期内累积终态记录——每条都携带 output 载荷。内存单调、静默增长。
• 进度层： 累积的终态记录本是会话后台工作的审计轨迹——但从不被裁剪，于是读取它们（job_list）与启动新任务（activeTaskCount）的成本随会话完成的工作量线性上升。会话越长寿、越高效，就越慢。
• 调度层： 这正是调度/排队负载的场景：会话按计划或在循环中启动任务，记录无上限累积；在内存压力或可见延迟出现之前，没有任何信号。
• 与设计意图对照： maxConcurrentJobsPerOwner 与本版新增的 ensureOwnerCleanup 表明引擎确实在管理 job 生命周期——但只覆盖并发边界与 owner 边界。终态这个最早的、最安全的回收时机，没有被用作触发条件。

期望行为

• 终态 + 已读（reported） 的 job 应被回收（或移入有界、可驱逐的历史）——对终态记录设 retention 上限，与会话生命周期解耦。
• job_list / start() 不应付出 O(store.size)——按 owner 建索引（或有界终态历史列表）让它们保持常数时间。
• store 增长应可观测：暴露计数/字节指标与告警阈值，让静默的单调增长变得可见。

复现步骤

1. 在单个会话中启动一个后台任务（如通过 run_in_background / jobs 工具）并等它完成。
2. 循环重复——每次迭代 job_list 结果与 store 都在增长。
3. 观察：终态 job（连同 output）在整个会话期间留在内存；job_list 与每次新 start() 随数量增长而变慢；无 retention、无警告。

环境

• 引擎：@deepseek-ai/dsh 0.1.1-rc.2（0.1.0-rc.7 亦可复现）
• 模块：@deepseek-ai/dsh-jobs-local（lib/index.js）

证据（源码）

:165   this.store.set(id, job)                     // start 时写入一次
:365   settle(job, outcome) {                     // 终态迁移…
:369     job.output = outcome.output;             // …完整 output 持续驻留
:366     if (isTerminal(job.status)) return;      // 幂等，仍不删除
:411   for (const job of owned) this.store.delete(job.id);   // 仅在 owner 销毁时
:424   this.store.clear();                        // 仅在服务 teardown 时
:394   ensureOwnerCleanup(owner)                  // 清理绑定 owner 生命周期
:137   if (this.activeTaskCount(spec.owner) >= …) // 每次 start() 都 O(store.size)
:180   return [...this.store.values()].filter(…)  // 每次 list() 都 O(store.size)

---

署名 / About this report

EN: Reported by the OfferKuai (Offer快) Team — an AI startup building full-lifecycle job-application services, guided by the belief that "users need results, not repeated conversations." Founder: Zhaofeng (Yaming). We use DeepSeek Harness as part of our daily development workflow; this report is our way of contributing back to the ecosystem. Website: https://www.offerkuai.com/ | Contact: <contact@offerkuai.com>

ZH: 本报告由 OfferKuai（Offer快）团队提交 —— 一家专注 AI 全流程求职托管的创业团队，核心理念是「用户要的是结果，不是重复对话」。创始人：Zhaofeng（Yaming）。我们将 DeepSeek Harness 用于日常开发工作流，这份报告是我们对社区的回馈。官网：https://www.offerkuai.com/ | 联系：<contact@offerkuai.com>