Pre-step batch places the user prompt before injected context, forfeiting cross-session prefix cache (measured: 42% vs 99.8%)

[daveycodez]

Summary

preStep() in packages/core/agent-loop assembles each step's batch as [...claimed, context] — the claimed user prompt first, then every injected context message appended behind it. Plugins that inject context (agent-instructions, tool-skill's catalog, system-prompt's runtime snapshot, third-party plugins) all hook the agent/pre-step waterfall and append with [...decision.messages, theirs].

Because prefix caching is strictly positional, this places ~2k tokens of byte-stable context behind a token that is unique to every session. That context can never participate in a cross-session cache hit.

Observed on @deepseek-ai/dsh@0.1.1-rc.2, a batch lands as:

1. user prompt              (source.kind: user)          "test 2"
2. agent-instructions       (AGENTS.md)                  ~1,816 chars
3. runtime snapshot         (dsh-system-prompt)          ~1,068 chars
4. plugin guidance          (third-party)                  ~275 chars
5. skill catalog            (skill-catalog)              ~4,293 chars

Messages 2–5 are identical on every session start in that workspace. They are re-prefilled every time.

Evidence — DeepSeek official API

Three fresh sessions, same workspace, same provider/model/config. I diffed the request headers: system (10,083 chars) and all 39 tool schemas (39,282 chars) are byte-identical across them. The only variable was the first prompt.

session	first prompt	inputTokens	cacheReadTokens	hit rate
A	test (cold)	13,088	0	0%
B	test 2	7,586	5,504	42%
C	test (repeat of A)	—	—	99.8%

Session C recovers almost the entire prefix because the prompt text matches byte-for-byte. Session B changes two characters and forfeits 7,586 tokens of prefill against an otherwise identical prompt. The delta is not the tool schemas — those are provably unchanged. It is the ordering.

Why this matters disproportionately for local models (oMLX and similar)

On a hosted API a cache miss costs money at a discount ratio — cache-hit tokens are roughly 10x cheaper, so a miss is a real but bounded expense. On local inference the currency is different: a miss costs wall-clock time to first token, on hardware the user is simultaneously using for everything else.

Measurements from the same harness pointed at a local oMLX server (DeepSeek-V4-Flash-8bit, Apple silicon), first turn of a fresh session:

cacheReadTokens:  6,144
cacheWriteTokens: 3,628

37% of the first request was re-prefilled, and the bulk of it was context that had not changed since the previous session. The four injected messages in that workspace came to ~2,080 tokens.

Several things compound here:

• Prefill throughput on Apple silicon is far below datacenter GPUs. ~2k tokens of avoidable prefill is a perceptible pause at the start of every single session, not a rounding error.
• oMLX quantizes its KV cache to 2048-token blocks — every cacheReadTokens value observed was a multiple of 2048. The context group is ~1,870–2,080 tokens, i.e. almost exactly one whole block that is reusable in principle and never reused in practice.
• The local cache is single-tenant and disk-backed. Cross-session reuse is essentially guaranteed on a prefix match, with no TTL pressure or multi-tenant eviction. Local is the environment where correct ordering pays the most — and where the current ordering wastes the most.
• Local users run many short sessions. That is the entire appeal of a fast local model. A per-session tax lands hardest precisely where the model is supposed to feel instant.

The irony is that the harness already does the hard part correctly: system and the tool schemas are stable and cache beautifully. The context messages are equally stable — they are just on the wrong side of one variable token.

Proposed change

Order the entering batch as [...context, ...prompts] instead of [...claimed, context].

Concretely, in preStep()'s innermost waterfall fallback, and ideally as a final normalization after the waterfall resolves so plugin-injected context is included:

• Partition the final batch on source.kind === "user".
• Stable-partition, preserving relative order within each group, so agent-instructions → runtime → plugins → skills → prompt is retained.
• No-op when either group is empty (turn 2 onward typically claims a lone prompt with no injections).

This is a pure reordering — no message is added, dropped, merged, or rewritten.

It does not conflict with supersede-by-append

The runtime snapshot's "This snapshot supersedes earlier runtime-context snapshots" design is deliberate and good: re-sending policy changes as an appended message, rather than mutating system, avoids invalidating the whole session's KV cache mid-run. That mechanism is untouched. Superseding works by appending later in the stream regardless of where the initial snapshot sat. Only the first-turn ordering changes.

Secondary benefit

Placing the user's actual request last also improves instruction recency — the prompt currently sits ~2k tokens upstream of the generation point, behind a wall of reference material.

Workaround

This is fixable downstream today. agent/pre-step is a waterfall, cordis documents listeners as running outermost-first, and { prepend: true } unshifts to the front of the chain. So a plugin can delegate via next(), let the built-in fallback and every other plugin assemble the complete batch, then reorder what comes back.

Two caveats make it an awkward thing to leave to userland:

1. It requires being the outermost participant. time-context and tmux-context already prepend, so correctness depends on plugin load order — last to prepend wins. That is fragile to reason about and fragile to document.
2. Patching agent-loop directly is not viable for anyone running via npx, since the checkout lives in the npm cache and is wiped on upgrade.

Every user who wants cross-session cache hits has to independently discover this and get the load-order dance right. It seems better handled in the loop.

Happy to share session exports with the raw usage events if that would help.

[denial123789]

I checked this against the rc.2 source at b150a551. The ordering claim is correct, but I would narrow the proposed repair.

What the source proves

ReactLoopAgent.preStep() claims the inbox, assembles the system prompt, projects the runtime context, then uses this fallback:

messages: context === undefined ? claimed : [...claimed, context]

The final decision.messages are then appended one by one as durable user/message events before request derivation. Several built-ins preserve the same shape:

• agent-instructions deliberately inserts its desired message immediately after the last claimed message.
• tool-skill appends the first catalog or replaces an existing catalog in place.
• time-context delegates first and appends its reading.
• tmux-context is different: it delegates first and prepends a sourced reading.

So the first-turn order you measured is consistent with the implementation, and a variable claimed prompt before a byte-stable baseline necessarily prevents that later baseline from belonging to a cross-session common prefix.

Why source.kind !== "user" is not yet a safe cache lane

A final stable partition changes the durable transcript, not only the outbound request. More importantly, provenance does not express cache stability:

• time and tmux readings are dynamic plugin messages;
• runtime snapshots can be superseding updates;
• post-compaction instruction messages intentionally represent a newer baseline;
• one proposed step can claim multiple messages, including steering input;
• session-reference rewrites direct mentions while preserving surrounding batch order;
• third-party sources have no contract saying their payload is deterministic across sessions.

Moving every non-user source ahead of every user source could therefore improve the measured first-turn cache while silently changing recency and ordering semantics elsewhere. It is not a semantic no-op because Session replay observes the new order.

Safer contract

I think the runtime needs an explicit lane, not inference from source.kind, for example:

1. stable-baseline: deterministic for an exact profile/workspace/config digest and legal before the first claimed prompt;
2. claimed-input: preserve inbox FIFO exactly;
3. dynamic-context: time, tmux, references, and other per-turn material;
4. superseding-context: appended updates whose recency is part of their meaning.

Only lane 1 should move before the first prompt. Each participant opting into it should provide or inherit a digest that changes when the rendered bytes or authority inputs change. That also makes cache measurements attributable instead of assuming every plugin injection is stable.

A smaller first patch could special-case only the core initial runtime context, then migrate agent-instructions and the first skill catalog after tests establish their exact stability conditions.

Regression matrix I would require

• fresh Session: baseline before one prompt;
• two claimed inputs: FIFO unchanged;
• mid-turn steer/inject: no reordering across the claimed batch;
• post-compaction instruction refresh: newer baseline remains later than what it supersedes;
• runtime snapshot update: replacement remains later in history;
• time/tmux context: documented relative order preserved;
• session references: direct and ordinary messages retain identity/order;
• resume/replay: durable transcript produces the same provider request;
• provider accounting: compare identical and different first prompts with stable system/tools and report cache block granularity.

The 42% versus 99.8% observation is strong evidence that order matters for this route. I would still describe the exact saved-token delta as provider/cache-granularity dependent until the same request capture is replayed through the proposed ordering change.

Pinned source points: agent.ts, agent-instructions, tool-skill, time-context, and tmux-context.

[daveycodez]

Follow-up with results — I implemented the reordering as a local plugin and measured it end to end.

Setup: local oMLX server on Apple silicon, DeepSeek-V4-Flash-8bit. Two fresh sessions in the same workspace, prompt text changed from hey to how are you? between them — i.e. exactly the case that currently forfeits the prefix.

Before ([...claimed, context], current behavior):

• cache hit ~42%
• time to first token ~10s

After ([...context, ...prompts]):

• cache hit ~94%
• time to first token ~3s

Same machine, same model, same workspace, same context on both sides — the only variable is where the injected context sits relative to the prompt.

7 seconds off every session start, from a pure reordering that adds, drops, and rewrites nothing. This is the local-inference argument from the original post in concrete terms: on a hosted API this shows up as a discounted line item, but locally it is the difference between a session that feels instant and one that visibly stalls before the first token.

Nothing regressed. The runtime snapshot's supersede-by-append still works as designed, and putting the prompt last is a mild bonus for instruction recency, since it no longer sits ~2k tokens upstream of the generation point.

Still happy to share session exports with the raw usage events if useful.