Web host TTFT climbs to 50-170s under any sessions that has subagent fan-out / mature sessions - three measured contributors (host-side, not gateway)

[dathtd119]

TL;DR

On a self-hosted dsh web (v0.1.0-rc.7, systemd unit, one Node process), time-to-first-token as seen in the UI grows from ~10s on fresh sessions to a median 49.7s / p90 142s on mature, multi-session hosts, while the model gateway answers the same requests with a 2.1s median TTFT. I instrumented both ends (laptop session logs + the gateway router's sqlite request log) and micro-benchmarked the installed dsh-session class. Three host-side mechanisms explain the gap; all three have plausible small fixes.

Evidence chain

1. Gateway is not the bottleneck. 441 requests matched one-to-one between laptop session logs (step/start → first chunk) and the gateway router's requestDetails rows (same window, 2026-08-18 22:07 UTC →):

stage	median	p90
laptop-measured TTFT (step start → first chunk recorded)	49.7s	142.0s
request arrival at gateway (after step start)	5.3s	30.1s
gateway TTFT (router-measured, same requests)	2.1s	9.1s
gateway first token → laptop records first chunk	35.1s	128.7s

Of 220 requests with laptop TTFT >50s, 195 had gateway TTFT ≤10s. Both the outbound leg (arrival p90 30s) and the inbound leg (return median 35s) lose time inside the host process boundary.

2. It scales with concurrent in-flight steps in the one host process. TTFT by number of concurrent active model steps across all sessions (main + subagents, same host):

concurrent steps	median TTFT
1	12.8s
3	22.1s
5	71.7s
8	104.1s
10	167.5s

Fresh-spawn subagents (own session, own log) reproduce it and this is not a fork-only effect. Restarting dsh web immediately restores ~10s TTFT (sessions resume from disk), and it degrades again over hours.

3. session.events snapshot rebuild is O(log) per append. POC against the real installed Session class (rc.7), reproducing the production read pattern (token-meter _sync reads session.events on every session/event, i.e. every streamed chunk):

log length	append + events read (ms)	append alone (ms)
1,000	0.010	0.006
30,000	0.067	0.005
120,000	0.501	0.006
200,000	0.955	0.008

Append is flat O(1); the getter rebuild (Object.freeze([...this.log]), dsh-session/lib/index.js:1397-1400, invalidated per append at 1470) is O(n). One of my real sessions holds 94k reasoning-chunks + 58k assistant/chunk events, so a single streamed turn (~7k chunk appends) pays ~0.5s at 30k log length and ~6.7s at 200k. per concurrent session, on the shared event loop, on top of per-event observer fan-out (api-proxy broadcast, zstd flush).

Ask / fix directions

1. Token-meter (and other per-event observers) over an O(1) cursor instead of the snapshot array. _sync only needs "new events since my cursor". session.log.slice(consumed) (or an iterator) avoids the full-copy+freeze per chunk entirely. The snapshot getter can stay for external consumers.
2. Chunk-granular durable events are expensive by design. 150k+ events per session is normal here; even at O(1) per append, per-event observer dispatch across N concurrent sessions adds up. Would batching streamed chunks into one append per step (or per ~100ms flush) be acceptable? It would also shrink session files dramatically.
3. subagent_fork seeding does a synchronous full-log validate/deep-freeze of the parent's log (events.slice(0, lastEnd.seq+1) → Session constructor) : a ~100k-event blocking copy on the shared loop. Chunked/deferred seeding, or freezing lazily per accessed depth, would remove the stall.
4. As a mitigation knob: a host-level concurrency limit on in-flight model streams (per process), since the measured cliff starts around 4-5 concurrent steps.

Environment: dsh CLI 0.1.0-rc.7, Node v26.5.0, web profile via systemd user unit, 3-5 concurrent sessions typical, orchestrator preset with named subagents. Happy to share the POC script, the join scripts, or raw numbers if useful.

[zoahdev]

核验通过（对照 main/master HEAD 99f6f02 rc.7 TS 源码）——三个 host 侧机制全部成立，你的测量（gateway 2.1s vs host 49.7s 中位 TTFT、并发 4-5 步悬崖）与代码形状完全吻合。这是目前社区最扎实的一份性能根因报告。

机制逐条确认：

1. token-meter 每事件读 session.events：packages/llm/token-meter/src/index.ts:95-96（ctx.on('session/event') → _sync）→ :174-176（while (state.consumedEvents < session.events.length) { const event = session.events[state.consumedEvents] }）。而 packages/core/session/src/index.ts:546-551 的 get events() 在每次 append 后失效并重建 Object.freeze([...this.log])——O(n) 全量拷贝+冻结，发生在这个共享事件循环上。你的 POC 数字（120k 日志 0.5ms/200k 0.955ms per append+read）与代码一致，且流式回合里每个 chunk 都触发一次。
2. chunk 粒度持久化：assistant/chunk/reasoning-delta 逐条 append → 逐条 observer fan-out（token-meter、api-proxy broadcast、zstd flush）。这是设计选择，不是 bug，但 94k+58k 事件/会话的量级确实把它推成主成本。
3. fork 播种全量校验+深冻结：Session 构造（index.ts:495+）对 seed 逐事件 assertEnvelope + freezeRestoredObject（index.ts:216-276），10 万级事件的 fork 是共享循环上的同步 O(n) 段。

修复优先级（同意你的四项，补充一个先决动作）：
0. 先扫全量 session.events 读取点：token-meter 是第一个，但 api-proxy broadcast / session-stats / projection 等 per-event observer 很可能同模式——把所有"每事件读全量快照"一次换成 O(1) 游标，比只修 token-meter 收益大得多。

1. O(1) 游标（你的方案 1）——最高优先：_sync 只需要 state.consumedEvents 之后的新事件；用私有 log 切片/迭代器即可，快照 getter 保留给外部消费者。零语义变化、纯性能。
2. chunk 批处理（方案 2）——设计决策，需单独过：合并 append 会改变持久化粒度与回放语义（Session projection cache: one non-JSON unit state stops title caching for every session #3183 投影缓存族同源），建议先量化"每会话文件大小 + 每回合 append 数"再定批大小/窗口。
3. fork 懒冻结（方案 3）——中优先：冻结移到访问路径（只读消费者按需冻结）可消除一次性阻塞；注意与 firstLiveSeq/不可变性契约的校验。
4. 宿主级并发限制（方案 4）——立即可用的旋钮：你的数据已证明 4-5 并发是悬崖起点，作为 mitigation 先上线成本最低。

一个补充观察：eventsSnapshot 在每次 append 失效意味着流式回合中途的任何外部读都在付全量重建成本；即使 token-meter 换成游标，其它 observer 仍在付。所以方案 0（全面清点）是让方案 1 收益不打折的前提。

需要的话我可以写一个微基准回归（1k/30k/120k/200k 的 append+read 表，对比游标 vs 快照），挂到 dsh-ecosystem 或你的 PR 里做长期防退化。POC 脚本可以发我对照。

[dathtd119]

我不会中文，所以只能用英文了，对此我很抱歉。Thanks for verifying all three mechanisms against the source and for the priority list.

For the workaround field report (maps to your item 3): I disabled subagent_fork in my agent preset, so every spawned subagent now starts as a fully fresh session, and restarted the host. Same resumed session, before vs after:

	median TTFT	max	host process
before (degraded host, ~150 steps)	17.8s	205.9s	32min CPU, 5.7GB RSS peak
after restart, fork disabled (first steps)	7.7s	13.0s	455MB

A fresh process with only main session was always fast for the first hours, so this is not yet proof. The real test is a full day of my normal 3~5 concurrent sessions + their own subagents fan-out parallely.

POC cjs script url: https://gist.github.com/dathtd119/e7ac30d2ca1b6cbf6b10f15e11542287 , it resolves the installed dsh-session, uses a real source file as the delta payload with a deterministic fallback, runs in ~2s.Your microbenchmark regression offer would be great too.

On your item 0: Yeah I agreed it's the prerequisite. I only confirmed token-meter; api-proxy broadcast and session-stats were suspects in my trace but unmeasured, so a full audit of session.events readers is the right first move. A since(seq) iterator (with the snapshot getter kept for external consumers) fixes the whole class.

[denial123789]

I verified the three relevant source paths at 99f6f02 and published an independent English measurement/mitigation guide based on this report:

https://sandbaseai.github.io/deepseek-harness-handbook/slow-ttft.html

The reusable diagnostic is a four-timestamp split:

• t0: Harness step/start
• t1: request arrives at gateway
• t2: gateway first token
• t3: first streamed assistant/chunk recorded

This separates Host outbound (t1-t0), provider (t2-t1), and Host inbound (t3-t2) instead of treating UI TTFT as provider TTFT. The guide also includes a fresh/mature/concurrency/restart/fork A/B ladder and a read-only Session-export script that counts total events and chunk kinds.

Two source nuances worth preserving in any fix/benchmark:

1. TokenMeter eager _sync() on session/event is conditional on a state already existing for that Session (this.states.has(session)). Once active, the first session.events read after each append can rebuild the complete frozen snapshot because append invalidates eventsSnapshot.
2. maxConcurrentAgents is a real mitigation only for the worker-thread Workflow engine. There is no documented Host-wide model-stream ceiling at this commit; maxParallelToolCalls is not a substitute.

The guide keeps your 49.7s / 2.1s values explicitly labeled as one rc.7 deployment rather than a universal baseline. If you publish the POC/join scripts, I can link the exact artifacts and turn the measurement section into a reproducible benchmark recipe.

[argszero]

The three mechanisms are solidly verified by zoahdev + denial123789, so I won't re-litigate them. One angle on fix direction 2 (batching) that the thread hasn't covered — on-disk durability/recoverability, from the session-log corruption family (13 reports, 11 mechanisms tracked across #1333/#1452/#1497/#2167/#2839/#3198/#3450 et al.).

1. Torn-frame semantics make batch size a loss-granularity decision. The JSONL backend writes one zstd frame per append, and the loader already drops an incomplete FINAL frame (committed-prefix semantics — session-persistence-jsonl/src/index.ts:245, torn-frame recovery at :347-383). So a crash mid-write loses the whole in-flight append, never a partial record. Today that costs ≤1 chunk. With "one append per step" it costs the step's entire streamed output — for a long reasoning turn that's minutes of tokens gone, even though per-append overhead drops. A time-bounded flush (~100ms) caps the loss at ~100ms of tokens while still cutting the append count from ~7k/turn to ~tens. The file-size win survives; the loss window doesn't grow with step length.

2. Batching widens the "resume seed ≠ persisted tail" window — the family's systemic risk. The corruption family's common thread is divergence between the in-memory log and the committed file at resume: #2839 (stale prepared-source cache + write-behind revision lag), #2167 (same-process stale-view re-append), #3198 (seed boundary captured before end-seed append — firstLiveSeq = log.length at core/session/src/index.ts:539). Batching deliberately makes the in-memory log run AHEAD of the committed file by up to the batch window. That's the safe direction only if the durable append contract is already in place: durable watermark + write-side seq assertion + writer lease (in-tree since #1550) + fresh-load-on-resume when a repair would be synthesized. Without those, batching makes each of those three mechanisms easier to trigger, not harder.

3. session/end-seed and turn/end must not join a token batch. Resume logic treats them as durable turn boundaries (the end-seed append is conditional on the tail lacking one; repair synthesis keys off open-turn state). If an end-seed rides in a token batch and the crash loses the frame, the resume sees an unclosed turn and synthesizes closers — the recovery-writer-vs-live-writer collision shape (#1586/#2905-family). They should flush synchronously as their own committed frame.

4. Replay ordering (#3450) is orthogonal — no new risk. Batches are consecutive seqs; append order = seq order, so batching cannot reorder within or across frames. The turn-tail violation comes from compaction's single replace + replay emitting shadowed copies in seq order across a turn boundary — granularity doesn't touch it.

Net: batch direction is sound with three constraints — bounded flush (time/byte-cap), synchronous end-seed/turn-end, and pairing with the append-contract hardening rather than landing alone. Happy to write the micro-benchmark (append+read across 1k/30k/120k/200k with batched vs per-chunk frames) if that helps pick the flush window.