Sglang params

[inscrut]

Hi!

Please, add --enable-cache-report flag in params default, thanks!

[hasso5703]

Thanks, good suggestion. Today none of the lanes runs it, so usage carries no cached-token count. With --enable-cache-report, SGLang adds usage.prompt_tokens_details.cached_tokens to each OpenAI response, and its Anthropic endpoint reports the same count as cache_read_input_tokens (read in the source of the SGLang build the 27B lane pins, v0.5.21). One catch on that build: a request with no cache hit reports prompt_tokens_details: null instead of 0; v0.5.22 keeps the 0 (sglang#43005).

We will try it on the reference box behind the proxy with the clients every release is tested against, and say here what we find. The Kolibri and Nemotron lanes run vLLM, whose switch for the same field is --enable-prompt-tokens-details.

Which client do you read it with?

[inscrut]

We tried it on the reference box, behind the proxy, on the 27B lane (SGLang v0.5.22), with and without the flag.

With it, each response says how much of the prompt came from the cache. On OpenAI's endpoint that is usage.prompt_tokens_details.cached_tokens (8,192 then 9,472 of a 9,539-token prompt that repeats a long system message), streaming included when you ask for usage. On the Anthropic endpoint, cache_read_input_tokens holds that part and input_tokens the rest, so the two add up to the total you got before. Without it, the cache works just as well (a 6 s prefill became 1.5 s), it simply is not reported.

Nothing else changed. The SDKs, fetch under Node and Bun, LiteLLM, Claude Code, opencode, Hermes and pi all did their task with it, opencode and Claude Code count their context the same way (only the split between cached and new tokens is new), and the throughput is the same (64.9 against 64.0 tok/s, the median of three runs each, within the runs' own spread).

It is on by default in the 27B's units since v1.40.1: after the update, a 27B lane that serves restarts once and reports it. The flash lane, on another SGLang build, does not have it yet, as it was not measured there. Thanks for asking for it.

thanks, add this flag for all lanes, this is useful info for clients and "proxy llm servers"

[hasso5703]

Thanks, agreed. The next release adds it to every lane, each measured as the 27B was before it is turned on: the flash lane with --enable-cache-report, and the vLLM lanes (Ornith, FP4-Local, Kolibri and the two Nemotron lanes) with --enable-prompt-tokens-details. We will say here what we measured.

[hasso5703]

A correction to my reply above: the cached-token count did not make it into v1.40.2. That release fixes opencode on the Ornith lane, makes claude-local ask before each command, and adds a CI check of every lane's chat template. The cached-token count on every lane comes in v1.40.3, measured lane by lane as described above, and I will post the numbers here.

[hasso5703]

v1.40.3 is out: every lane now reports what its prefix cache served. The flash lane runs SGLang with --enable-cache-report, as the 27B does, and the five vLLM lanes (Ornith, Flash FP4-Local, Kolibri, Nemotron Bolt and Super VL) run with --enable-prompt-tokens-details. Each answer carries usage.prompt_tokens_details.cached_tokens, and cache_read_input_tokens beside input_tokens on /v1/messages.

Measured on the reference box through the proxy, with two requests sharing an 8,000-token prefix and ending differently, then the same prompts again:

• 27B and flash: 8,192 of the second request's 8,248 tokens came from the cache (0.5 s, against 4.0 to 4.5 s for the first).
• Ornith: 4,224, one whole block of this hybrid model.
• Kolibri, Flash FP4-Local and Super VL: 32, 0 and 0 for the request that ends differently, since these engines reuse a prompt only from the points their cache keeps. The same prompt sent again came back whole but for a few tokens.
• Nemotron Bolt: 0. Its drafter keeps the cache's hits on 4,544-token blocks, which a 7,256-token prompt does not reach.

An agent's conversation is what those points serve. On Flash FP4-Local, a conversation that grows at every turn got 7,971 and then 8,001 of its 7,996 and 8,023 tokens from the cache, and the engine's own counters said the same.

Release notes: https://github.com/hasso5703/dgx-spark-qwen38/releases/tag/v1.40.3

Thanks again for the suggestion.

[inscrut]

thanks!