Poor prompt-cache hit rate on z.ai GLM Coding Plan: prefix is stable, hits still collapse (evidence + proposal to expose pi-ai session-affinity switches for the zai routes)

[eddiedon]

Summary

When DSH talks to glm-5.3-flash through the built-in zai provider (OpenAI-compatible endpoint https://api.z.ai/api/coding/paas/v4), prompt-cache hit rates are poor even though DSH itself constructs a perfectly stable request prefix. Measured on a real working session (DSH 0.1.2-alpha.2): 70.5% hit over the first user turn, 3.4% on a single step, and 0% on two consecutive requests 17 seconds apart.

The GLM Coding Plan bills cached tokens as normal session usage, so a collapsed cache directly burns roughly 2-3x more quota than necessary. The same workflows on clients that anchor caching (ZCode, OpenCode) drain noticeably less.

What I measured (session-log forensics)

I decoded the session JSONL (multi-frame zstd) and diffed every recorded request header plus per-step usage events:

• The system prompt is byte-stable: 18,334 chars, 0 diff across all 6 recorded request/header events.
• The message history is append-only; runtime-context snapshots are additive (the old snapshot stays in place, a new revision is appended), so there are no mid-history rewrites.
• Consecutive tool-call steps hit 95-99.9% - the prefix pipeline itself is healthy.
• All real losses are full misses with an unchanged prefix, caused by concurrent sessions sharing one API key:

time	event	usage
18:55:24	sibling session (same key) cold start	92.9k non-cached tokens (3.4% hit)
18:56:50	this session, first step	16.3% hit
19:01:03	sibling session cold start	104k non-cached tokens (7.2% hit)
19:01:25	this session, step 8	3.4% hit - 76.9k tokens of accumulated cache lost, prefix unchanged
19:05:02 / 19:05:19	this session, two steps 17s apart	0% / 0%

z.ai's implicit context cache is shared per API key, so any concurrent activity - subagents, AgentTeams members, plugin background forks (e.g. a periodic "idle checkpoint" review that replays an ~87k-token seed) - evicts everyone else's entries. With 3-4 active chains the cache pool is a tug of war.

Why it happens (source-level)

pi-ai already implements everything needed, but for the zai compat profile it is all switched off - and DSH's compat gate makes it unconfigurable:

• detectCompat() for zai sets cacheControlFormat: undefined and sendSessionAffinityHeaders: false (@earendil-works/pi-ai openai-completions.js);
• DSH's COMPLETIONS_COMPAT_GATE marks sendSessionAffinityHeaders / sessionAffinityFormat as withhold, so they cannot be enabled from settings.yaml;
• prompt_cache_key is only sent to api.openai.com (or with long retention).

So every request is an unanchored, best-effort implicit-prefix-cache request: no session affinity to pin routing, no cache breakpoints, no cache key. Upstream pi issue #6083 (already fixed in DSH: send thinking.clear_thinking: false) proved this endpoint's caching is unusually sensitive; affinity anchoring is the remaining half of the fix. ZCode and OpenCode show what anchored caching is worth on this plan.

Proposal

1. Flip sendSessionAffinityHeaders and sessionAffinityFormat from withhold to offer in the dsh-llm-pi-ai compat gate, so a settings.yaml provider profile can enable pi-ai's existing affinity implementation (session_id + x-client-request-id + x-session-affinity headers, plus prompt_cache_key) for zai / zai-coding-cn routes. Static per-provider headers config cannot substitute: the affinity value must be the per-session id, which only the host can inject.
2. Ideally default-enable affinity headers for the zai routes - worst case the endpoint ignores unknown headers, so the blast radius is zero.
3. Optional: also offer cacheControlFormat: anthropic for routes whose endpoint accepts Anthropic-style cache breakpoints (e.g. z.ai's /api/anthropic).

Happy to share the session-log decoding/analysis scripts if useful.

[eddiedon]

Update from OP, verified on 0.1.2-alpha.4 (@earendil-works/pi-ai 0.84.4): the original ask still stands, but there is now a measured, zero-code fix -- which makes the proposal stronger.

Re-verification on 0.1.2-alpha.4

• detectCompat still hardcodes sendSessionAffinityHeaders: false for the zai route, and prompt_cache_key is still only sent to api.openai.com; the DSH compat gate still marks the session-affinity fields withhold (not configurable).
• The thinking.clear_thinking: false fix (LLM cache is not working properly with z.ai GLM coding plan earendil-works/pi#6083) is still in place, so that part of the problem remains solved.

New finding: the z.ai OpenAI-compatible endpoint already honors Anthropic-style cache_control

Two independent pieces of evidence:

1. OpenCode -- which users consistently report as not having this cache problem -- talks to the same endpoint and protocol (https://api.z.ai/api/coding/paas/v4, OpenAI-compatible), and its applyCaching transform unconditionally injects cache_control: {type: "ephemeral"} breakpoints into openai-compatible requests (leading system messages + trailing messages).
2. pi-ai ships the same mechanism (applyAnthropicCacheControl: breakpoints on system + last tool + last message), and unlike the affinity fields, DSH's compat gate already exposes cacheControlFormat as offer (configurable).

So this works today with two lines of YAML per provider -- no patch, no fork:

llm-pi-ai:
  providers:
    zai:
      compat:
        cacheControlFormat: anthropic

Measured A/B (same machine, same routes; per-request usage decoded from session logs, then independently re-verified from the raw frames)

	before (implicit cache only)	after (cacheControlFormat: anthropic)
zai coding-plan route (api.z.ai), turn-level hit rate	70.5% (9 steps)	86.9% (14 steps)
open.bigmodel.cn route, turn-level hit rate	75-88% typical	90.8% (15 steps)
"prefix unchanged but full miss" crashes	one step at 3.4% (~76K cached tokens lost, 93K recomputed) 22 s after a sibling session's ~104K cold start on the same key; two requests 17 s apart both 0%; total loss across 74 s / 3 min idle gaps	zero crashes in 29 steps
recovery right after a 0% cold-start step	could stay near 0% for consecutive steps	97.4% within 7 s
steady-state tool-loop steps	95-99.9% (already fine)	93.2-99.1% (unchanged, as expected)

Notes on methodology: client-side prefix hygiene was verified byte-identical throughout (system prompt 18334 chars, diff = 0 across every request; runtime-context snapshots are append-only), so the "before" crashes were purely server-side eviction in a node-local implicit cache -- exactly what explicit breakpoints eliminate. The A/B sessions are different real-workload sessions (not synthetic replays), but with comparable multi-step tool-loop patterns. Since cached tokens still count against Coding Plan usage, this is roughly a 2x effective-quota difference on heavy multi-tool workloads.

Upgraded proposal

Given that both z.ai endpoints (international api.z.ai and CN open.bigmodel.cn) demonstrably accept Anthropic-style cache_control, and OpenCode already ships exactly this at scale:

1. Default cacheControlFormat: anthropic for the built-in zai / zai-coding-cn providers, so every GLM user gets explicit cache anchoring out of the box. Worst case is an endpoint ignoring a well-formed optional field; measured upside is +16-20 pts turn-level hit rate and the elimination of eviction-driven full misses.
2. The original ask stands unchanged: expose sendSessionAffinityHeaders / sessionAffinityFormat as offer instead of withhold -- still a useful complementary layer (notably for routes/endpoints that lack cache_control support), just no longer the only lever.

Happy to share the analysis scripts (zstd session-log frame decoder + per-step usage extractor) if anyone wants to reproduce the numbers.

[toilmonkey0-cyber]

I feel like I used to get forever use out of my ZCode sub but using it within Hermes or grok build is noticably more token heavy