Tool schemas: where they really cost you (prompt-cache order) and what happens if you drop them

[laa1991]

Two findings from measuring a local dsh deployment (2026-10-09) against the official Messages endpoint — a follow-up to our earlier startup-tax post, on the same question of which bytes cost what. Both are cheap to reproduce.

Versions and scope: @deepseek-ai/dsh-tools 0.1.0-rc.5 (a fork of deepseek-ai/deepseek-harness); 17 always-on tools on the wire after the fix below (20 before it — 3 lazily loaded tools had re-joined the field). The cache-order result is a property of the provider's prefix cache, so it applies to any client; the "constant tools array" fix is a config we added in our fork — the upstream on-demand loading path still appends a loaded tool's schema to the field.

1. tools is cached before messages, so a load costs the whole context — not the delta.

It is tempting to reason from the wire order (the request body serializes messages before system and tools). The cached prefix is not in that order. Same system, same message, one change per request:

change	cache_read	uncached
none (warm)	2,944	191
append one tool to tools	256	2,936
prepend one tool to tools	256	2,936
edit the tail of the message	2,944	192
edit the head of the message	256	2,883

The break lands at the start of the messages in every case: the cached prefix is system → tools → messages. So any change inside the tools field re-prices the entire retained history on the next request. Scaled to a session with a ~1 MB prompt: the schema being added is a few hundred tokens, while the history that gets re-priced is ~375k tokens — three orders of magnitude apart.

That matters for every on-demand / deferred tool-loading scheme: if loading a tool appends its schema to tools, the load is billed as a full context re-read. We made the array constant per session instead — a loaded tool stays off the wire, and its parameter detail arrives in the loading call's own result text. After the change: tools went from 20 entries / 13,580 B to 17 / 10,553 B, the whole tools section is byte-identical across a load, and 16 consecutive requests show cache_read climbing 179,072 -> 193,792 with 164-875 uncached — no break.

2. You can drop the JSON schema, but you cannot drop the parameter knowledge.

Four arms x 10 realistic one-call tasks, same model (deepseek-flash), same 17 tools, only the parameter presentation changed:

parameters live in	invalid first calls	correct tool picked
JSON Schema (baseline)	0 / 8	6/10
a compact text signature in system	0 / 10	6/10
the same signature appended to each tool's description	0 / 9	6/10
nowhere (names only)	3-4 / 9 (33-44%)	6-7/10

Typical failures in the last arm: read called with path instead of file_path; pwsh missing its required description. Tool choice was unaffected across all four arms (the same tasks were missed in each).

Byte math on our 17 always-on tools: JSON schemas plus structure = 10,553 B; moving the parameters into a text signature with descriptions untouched = 9,647 B (-8.6%); also collapsing the descriptions to one line = 4,838 B (-54%). We took neither: the first saves about 23 full-price-equivalent tokens per request, and the second buys ~5.7 KB by dropping the operative text in those descriptions (how to run long jobs, exit-code semantics, spill files, sandbox denials) — while one extra round caused by a mis-called tool costs thousands of those tokens. The load-path fix above is worth roughly 2,300 requests of that byte-saving.

Happy to share the probe scripts and raw numbers.

[slyyr]

A useful design principle here is to keep the tool definitions stable across requests whenever the provider's prompt-caching behavior benefits from a stable prefix.

If adding a tool changes the serialized tools section, the cache may no longer match from that point onward. Depending on the provider's caching rules and prefix structure, this can make much more of the conversation uncached than the size of the newly added schema would suggest.

I'd consider three approaches:

1. Keep a stable tool set per session: Avoid modifying the serialized tool list every time a deferred tool is loaded.
2. Separate discovery from invocation: Load detailed instructions or parameter information only when needed, while ensuring the model still has enough information to call the tool correctly.
3. Measure cache behavior directly: Compare cache-read and uncached token counts for identical requests, then repeat after adding a tool, changing a description, and modifying the conversation.

One caution: removing schemas entirely is not automatically a good optimization. The model still needs accurate parameter names, required fields, and behavioral constraints. A smaller prompt can become more expensive if it causes invalid tool calls and additional retries.

I'd evaluate the optimization using total cost per successful task, not just the number of tokens saved in the tool definitions.

Hope this helps! If this addresses the question, would you consider marking the answer as accepted?

[laa1991]

Thanks for reading it closely — the three points are, in order, the three things the post measures, so I think we're saying the same thing:

• Stable tool set per session → that is §1's fix: the tools array is byte-identical across a load (17 entries / 10,553 B), and 16 consecutive requests show no cache break.
• Discovery separate from invocation → the parameter detail rides in the loading call's own result text; §2's fourth arm is exactly the caution you raise (names only ⇒ 3–4 of 9 first calls had invalid arguments).
• Measure cache behaviour directly → §1's table is that measurement: same system and message, one change per row (append a tool / prepend a tool / edit the message tail / edit the message head).

On cost per successful task: agreed, and it is why we took neither byte-saving option — dropping the parameter text was worth ~23 full-price-equivalent tokens per request, while one extra round caused by a mis-called tool costs thousands of those.

One thing I can't do: this thread is filed under Ideas, which has no accepted-answer feature (it is also our own post), so there is nothing to mark there. If a specific number would be useful to you — the probe recipes behind either table, for instance — just ask and I will post them.

Same setup as the post: @deepseek-ai/dsh-tools 0.1.0-rc.5, measured 2026-10-09.

[laa1991]

A follow-up to §2 — what the failure profile looks like in real use, and where the parameter knowledge actually matters.

We re-ran the question against our own session corpus (939 sessions, 95,235 tool results) instead of a synthetic arm:

• 2.2% of tool results are non-success (2,110).
• 0.23% (215) are argument-validation failures (INVALID_ARGS).
• Those cluster on unfamiliar, low-frequency tools: skill 39, decide 31, next_write 30, sonar_ping 29 — while every high-frequency tool is below 7 events each, i.e. ≤0.03% (edit 17,885 calls, read 18,235, pwsh 30,368, grep 6,540).

The dominant failure modes are world-state, not argument shape: "read the file before editing it" (FS_NOT_OBSERVED, 658), edit target not found (271), file not found (199), stale file version (179).

Two adjustments to §2's framing follow from that:

1. The 33–44% invalid-argument figure came from a cold one-call arm, where the tool had never been used in that context. In a real session the parameters are usually visible in earlier calls sitting in the message history, so a schema's marginal value concentrates on the first use of an unfamiliar tool, not on the tools you call all day.
2. Per-tool byte accounting on our 17 always-on tools: only 2 are parameterless, so "call it by name" covers 2/17 of the surface; moving every parameter block into a one-line text signature saves 906 B (10,553 → 9,647) ≈ 26 full-price-equivalent tokens per request, while the description text (5,865 chars) is where the bytes actually sit.

Boundary: one deployment, one fork, one corpus — those rates are ours, not universal, and we have not yet tested removing a schema from a high-frequency tool. That A/B is the next thing we would run.

Same setup as the post: @deepseek-ai/dsh-tools 0.1.0-rc.5; corpus measured 2026-10-10.