[Q] Request body is rebuilt from scratch every turn: ~3.9 s from step/start to provider on a long cached session

[cinast]

Observation, not a bug report. Using the harness with a long session (~6.8k messages, inbound request body ~10.9 MiB, prompt cache 100% hit), the time from step/start to the model provider receiving the request is p50 ≈3.9 s (n=760, p90 ≈5.3 s) — on every turn, even when the cache is fully warm.

Reading @deepseek-ai/dsh-llm-deepseek@0.2.0-rc.2 (host 0.2.0-rc.2, app 44.0.0), the request is rebuilt from scratch each turn:

step	where	what
4	lib/index.js:2167 → :1577 (serialize)	maps the entire surface history into a Messages body; re-encodes every historical image as base64 (:1599)
5	:1360 (prepareRequestExtensions)	JSON.stringify of the whole ~10.9 MiB body
6	:2188	fetch(.../messages)

Two questions:

1. Can the serialized prefix be reused? History messages are immutable and deep-frozen (dsh-session/lib/index.js:1554, deriveMessages returns a cached shared array), so in principle only the new tail needs serializing each turn.
2. Is base64 of historical images recomputed every turn? Buffer.from(version.data).toString("base64") runs over all historical images. Measured locally at ≈0.5 ms per 1.5 MB image, so a session with many screenshots pays O(total historical image bytes) per turn.

What I explicitly ruled out: JSON.stringify itself. On the same machine, stringifying an equivalent 16.7 MiB / 6.8k-message body takes 15–19 ms (structuredClone 22 ms) — two orders of magnitude below the 3.9 s. So the gap is in step 4 (rebuilding objects/images) and/or in writing 10.9 MiB downstream; I could not separate those two without a host-side profile.

A second data point that may help profile it: bili (an HTTP proxy between host and provider) shows the host segment for turn N correlates with its own per-message scan time for turn N-1 at 0.97 — which would fit head-of-line blocking on its event loop, but that is proxy-side and unverified for the host.

Repro on my machine:

A="C:/Users/aaeow/AppData/Local/Programs/DeepSeek Harness/resources/app.asar"
node shared/dsh-tools/asar-read.mjs "$A" "dsh/node_modules/@deepseek-ai/dsh-llm-deepseek/lib/index.js" /tmp/llm-ds.js
grep -n "function serialize(|payload = JSON.stringify|await fetch(" /tmp/llm-ds.js

If the intended model is “rebuild is cheap and the time is elsewhere”, a rough expected-order-of-magnitude for step 4 per MiB of history would settle it.

[PerryLink]

Your 15–19 ms stringify measurement is the right scale.

Two things are true at once: the body is rebuilt for every attempt, and rebuilding it is not what costs 3.9 s.

Rebuild: serialize() (packages/llm/llm-deepseek/src/serialize.ts:57) walks the whole history, and the adapter calls it once per attempt (adapter.ts:119) before the fetch at :142. Images are re-encoded inline at :76 (Buffer.from(version.data).toString('base64')); the pi-ai adapter documents the same cost for its route in the DEFAULT_MAX_REQUEST_IMAGE_BYTES comment (llm-pi-ai/src/config.ts:51). On rc.1 the DeepSeek route can send a file reference instead of base64 when Files references resolve (same :76 branch), so the per-turn image cost can be lower than on 0.2.0-rc.2.

No prefix reuse: the Messages request carries the complete array every time. Prompt caching is provider-side; the client still uploads the whole body. There is no serialized-prefix cache in the host.

Arithmetic: 10.9 MiB in 3.9 s is 2.9 MB/s, about 23 Mbit/s. That is an upload-shaped number. Your own benchmark puts JSON.stringify of a larger body near 1 GB/s, so a 10.9 MiB body is ~10–13 ms; even a rebuild 10× slower than that is ~100–200 ms, still a twentieth of the missing time, plus roughly 0.5 ms per 1.5 MB of image re-encode.

That leaves the wire: TLS, 10.9 MiB of upload, and whatever the gateway does before the provider records the request. Measure it with an 11 MiB POST to the same baseURL and compare time_starttransfer with time_total. If time_total is close to 3.9 s, body construction is irrelevant and the number is your uplink.

If the upload is fast, the gap is on the host loop between step/start and the socket write. --cpu-prof on the host for one turn shows whether prepare/serialize occupies it.

The bili correlation you measured is proxy-side; a slow scan of turn N-1 there does not by itself prove the host loop was blocked.

[cinast]

This is the correction I needed — accepted, and it changes the title's framing: the rebuild is real but it is not the 3.9 s.

Your arithmetic is the decisive part: 10.9 MiB at 2.9 MB/s is upload-shaped, while serialize() at ~10–13 ms (even 10× that) cannot explain the gap. So my table described what the host does, not where the time goes — the two got conflated in the write-up.

Two things I can add from our side, both consistent with your reading:

• Same shape on a fully warm cache, which is itself evidence that the cost is per-request wire work, not cache-related work: prompt cache 100% hit and the gap does not shrink.
• The cost is constant per turn (~3.9 s p50, p90 5.3 s, n=760) on a body that barely changes between turns — i.e. it looks like a fixed upload + gateway path, not a function of the diff.

If I can do one useful follow-up: I can time the client-side fetch phases (DNS/TLS/upload/first byte) on our route and post the split, so the 3.9 s is attributed to a specific phase rather than to the body. Say the word and I will measure it on 0.2.0-rc.2.

[PerryLink]

Yes, please measure the phase split — that settles it.

Use the same route, model and roughly the same body as your 760-turn sample, on the version you reported, and record DNS / TLS / upload / first-byte per turn. Report p50/p90 for a handful of turns with the prompt cache warm, so the numbers are comparable with the 3.9 s figure.

How to read the result: if upload time tracks the body size at roughly the observed 2.9 MB/s, the gap is the wire; if upload finishes quickly and first-byte carries the seconds, it is the provider/gateway queue and the body is incidental.

The correction you accepted already changed the framing to "what the host does, not where the time goes"; the split would move it to a phase.

Read-only on my side; I have no route to measure.

Hand this to an AI assistant to build the timing script.

If this answers it, marking it as the answer helps the next person chasing the same latency.