Skip to content

0.11.7

Choose a tag to compare

@github-actions github-actions released this 11 Nov 19:35
· 13 commits to main since this release
038ce1c

For anyone running Prefixwise in front of vLLM or SGLang: fixes from a review of every vLLM and
SGLang blog post that touches the prefix cache (about 60 of 310), each checked in the engines'
source. Routers now get the routing keys they route by, host_restored is claimed only
on a hit the engine join itself judged, and an SGLang with its radix cache off is no longer an
eviction on every turn. Upgrade the collector first, then the edge.

  • Routing keys reach the router. Prefixwise forwarded a fixed list of headers that left out the keys routers and engines route by: x-sgl-routing-key (sgl-router's sticky policy), x-smg-routing-key and x-smg-target-worker (SGLang Model Gateway), x-dynamo-session-id (Dynamo) and x-data-parallel-rank (vLLM and SGLang pin a rank by it). Behind a router, a session could then land on any worker and miss; across data-parallel ranks whose events Prefixwise joins, those misses were confirmed as routing_miss, with the advice to fix routing, for a rank the client had pinned. They now pass through unchanged. To a self-hosted engine or its router, x-forwarded-for and cookie pass too (the Gateway's consistent hashing falls back to them); never to a hosted API, where they could carry a login cookie or an address meant for Prefixwise's own ingress. --pass-header NAME (or PREFIXWISE_PASS_HEADERS) adds a router's own header name; Prefixwise's own headers and those urllib sets are refused there. Behind the Gateway with --engine none, add --pass-header cookie if clients rely on it.
  • host_restored only from a hit the engine join judged. With no offload tier at all, a turn that appended a large tool result could be labelled host_restored with "the tier's own stores were seen": the classifier called it a hit (half the shorter prompt read), the join, judging by this prompt, took its miss path, and the blocks it found removed became "restored". Now only the join's own hit verdict can become the detail, on vLLM and SGLang. Reproduced from a recorded event sequence. A new edge and an older collector still claim it the old way; a new collector with an older edge claims it not at all: upgrade both.
  • SGLang: a prefix another conversation recomputed is not a restore. The docs said so; the SGLang join did not check (vLLM's did). A store by another session of the previous request's blocks while this one is in flight now withholds host_restored; SGLang's load-backs carry no session, so a real restore still counts.
  • A restarted engine must show its tier again. The flag that an offload tier exists was never reset when the engine restarted or flushed its cache, so an engine restarted without a tier could still get host_restored. (After a flush, a tier that restores before its next store of its own is not claimed: conservative.)
  • SGLang with its radix cache off. --disable-radix-cache (HiSparse requires it; deterministic inference sets it on most attention backends) means nothing is ever read back, yet an unchanged prefix that missed was an inferred eviction on every turn. /server_info says so, now read with or without KV events, and such a miss is unknown, "the engine keeps no prefix cache", with nothing avoidable. Only for the provider the engine serves (a hosted API behind the same prefixwise run keeps its causes), and with no detail's advice attached. Turn facts gain no_prefix_cache, sent only when true: a 0.11.6 prefixwise collect refuses those records, so upgrade the collector first if you run SGLang with its radix cache off.
  • SGLang Mamba models ignore --full-attention. /server_info shows uses_mamba_radix_cache; SGLang frees Mamba state without an event, so the flag would let the events rule out evictions that happened.
  • More fields that change the prompt are compared: reasoning_effort (vLLM passes it to the chat template, and writes it into gpt-oss's system message), mm_processor_kwargs and media_io_kwargs, SGLang's lora_path, documents and continue_final_message. A miss where only one changed is param_rendered, not an eviction. cache_salt is not compared yet, because the E1 experiment forces its retry misses with a fresh salt; it is folded into the cross-conversation prefix keys, so tenants with different salts are no longer told they could have shared a prefix.
  • Docs: host_restored with an SGLang L3 storage backend. docs/SGLANG.md said the detail means a restore from host memory. With --hicache-storage-backend in the default host memory mode, SGLang publishes a refill from storage as a host (CPU_PINNED) store and does not yet emit storage-tier events (sglang#29709), so there host_restored also covers a restore from storage; in buffer_only mode such a restore is a plain none. The page now says so, and that SGLang's per-request return_cached_tokens_details (not read by Prefixwise yet) is what would tell the two apart. Raised by an SGLang contributor reviewing the HiCache docs recipe (sglang PR #49194), and checked in SGLang's source.