Repository navigation
0.12.0
For anyone deciding whether to trust the causes Prefixwise names: the decision procedure is now written down (docs/ALGORITHM.md), checked by invariants on generated traffic, audited twice against the code, and measured at scale on two engines. Through vLLM and SGLang, half of it replaying real Claude Code trajectories, every one of 1,772 injected changes was seen, none was named as another cause, and none of 3,622 turns that only appended got a change cause. Also new: prefixwise import gateway replays a gateway's request log through the proxy's own pipeline, so Prefixwise can be tried on yesterday's traffic without touching production; the report says how far fixing the avoidable misses would take the hit rate; prefixwise.wrap records streamed calls; and OpenRouter keeps its sticky routing through Prefixwise. 0.11.8 was prepared but never published: its changes are here.
- The algorithm, written down and checked.
docs/ALGORITHM.mdspecifies the decision procedure: what is compared, the order of the rules, the evidence each needs, the overrides, and eleven invariants.tests/test_invariants.pychecks those invariants on about 1,600 generated conversations in Anthropic's and OpenAI's request shapes (real tool schemas, timestamps in the system prompt, multi-turn tool calls and results), hosted and behind an engine: one change of each kind is named exactly, a full read is never a miss, appending is never a change, the earliest of two changes wins, avoidable tokens stay within what was reusable, engine evidence never overrides a changed prompt, and the hashes ignore exactly key order and volatile values. - Accuracy at scale, measured (E5).
experiments/e5_scalereplays conversations through Prefixwise into a live engine with its KV events joined, injects a known change at random turns (tools reordered, a tool added or removed, a timestamp, a system prompt edit, an edited earlier message, an edited tool result), and scores the verdict. On Colab L4 with Qwen3-4B: vLLM 0.33.0, generated conversations 441 of 441 named, real Claude Code trajectories (nvidia/Open-SWE-Traces) 425 of 441; SGLang 0.5.23, generated 456 of 456, real 424 of 434. The 26 not named as the primary cause were edits appended to the latest tool result, which fall in the prompt's last, uncached block: seen as a secondary cause, nothing lost. None was named as another cause, no turn that only appended (3,622) got a change cause, and none of the 5,994 requests failed. Not exercised: an edit at the start of the latest tool result, evictions under memory pressure, other models. Records:experiments/e5_scale/runs/. prefixwise import gateway: a gateway's request log, with the live proxy's verdicts. A JSON Lines log (one request per line: its body, the response with usage, when it started and ended; field names matched from the common ones) is replayed through the proxy's own pipeline on the log's clock: start and end events in time order, so requests that overlapped still overlap. Trying Prefixwise on yesterday's traffic needs no change to production. Checked by running the same traffic through the live proxy and the import: identical cause, detail, avoidable tokens and cost on every request, over 40 generated traffics with two interleaved conversations, partial and full reads and gaps past the cache's lifetime. Prompt text is read to fingerprint it and never stored.- How far a fix would take the hit rate. The report gave avoidable tokens and dollars but not the question teams ask: is there room from here? It now adds, after the list of avoidable causes, "Fixing these would take the share served from cache from 74.6% to 87.1%; the rest is new content each turn, misses with a legitimate cause and misses whose cause is not known", and the shareable block "87% from cache if fixed" (rounded down, with a decimal when whole percents would hide the gain or read as a perfect cache). It is (cached + avoidable) / prompt over the same requests: avoidable tokens are exactly the reusable prefix the cache did not serve on wasteful requests, so a fix moves them and nothing else. Checked against a real run: Claude Sonnet 4.6, 20,089 of 26,932 tokens cached (74.6%) with 3,375 avoidable predicts 87.1%, and the same agent with
--optimizeread 87.1%. prefixwise.wraprecords streamed calls.create(..., stream=True)and the SDKs'.stream()helpers (messages.streamandbeta.messages.streamon Anthropic,chat.completions.streamandresponses.streamon OpenAI), sync and async, were passed through unrecorded; agents that stream every call recorded nothing. The stream the caller iterates is still the SDK's own (isinstance(s, openai.Stream),s.response, context managers andclose()all work as before); usage is read from its events as they pass (Anthropicmessage_startandmessage_delta, the final usage chunk on Chat Completions,response.completedon Responses) and the call is recorded once, when the stream ends. A stream closed or cancelled before its end is a failure row (client_disconnected), as the proxy records a client that hangs up, the helpers' streams too; one the server or the network breaks (anerrorevent mid-stream) isupstream_dropped; one dropped without being closed or read to its end is not recorded. On Chat Completions usage is only streamed whenstream_options.include_usageis set: when the caller has not set it (instream_optionsorextra_body),wrapasks for it on a copy of the caller's options and keeps the extra usage-only chunk out of the caller's stream; a server that rejectsstream_optionsgets the call as the caller made it, and is not asked again. The session is the one current when the call was made, wherever the stream is consumed. Streams read throughwith_raw_responseorwith_streaming_responseare left untouched and unrecorded.- OpenRouter keeps its sticky routing through Prefixwise. OpenRouter sends a conversation back to the provider that holds its cache by the body
session_id(its caching guide). Prefixwise removed a client's bodysession_idon the way to every hosted API, since most reject unknown fields, so through Prefixwise an agent that sent one lost OpenRouter's stickiness, its cache could miss, and the miss was blamed on routing. A client's ownsession_idnow reaches an upstream on openrouter.ai unchanged (PREFIXWISE_FORWARD_SESSION_ID=1for another gateway that reads it); other hosted APIs still never get one, and Prefixwise never sends one of its own. - OpenAI's
usergroups turns. With no session header or bodysession_id, all traffic was one session; an end-user id now names it (Anthropicmetadata.user_id, as before, or OpenAIuser);prefixwise checknow ignores both, so its scenarios never merge. It is kept out of OpenTelemetry's conversation id, also when recorded throughprefixwise.sdk. Turns of one user's different conversations share that session, and the first turn of a later conversation with the same system prompt can read as a history edit, as it did in the single default session before: send a conversation id where you have one. Aprompt_cache_keyis deliberately not used: a key that changes mid-conversation is the misscache_key_changednames. - The earliest change in the conversation is the one named. With an earlier tool result and a later message both changed, Prefixwise named the later edit (
history_edit): it checked all messages before all tool results instead of walking the conversation in the order it was sent. Both are waste, so the avoidable count was right; the cause pointed at the wrong block. Found by the new invariant tests. - A mid-conversation developer or system message is growth, not a system prompt edit. On OpenAI's chat shape every system and developer message was taken as part of the system prompt, wherever it was sent, so an agent that appends a reminder mid-conversation got
system_editon each such turn (ornonewithsystem_editas a secondary cause). Only the messages that open the conversation are the system prompt now; a later one is a message where it was sent, as the provider renders it. Found by the specification audit. - A parallel subagent of another type is a new conversation, not an unjudged one. The in-flight rule (a request sent before its previous turn finished is
unknown) compared a new conversation with another conversation's turn, so a subagent started while the main agent's request was in flight came outunknowninstead offirst_write. And no detail is attached to a turn left unjudged: one could promote it to a cause from a negative gap. - Behind an engine, the "first changed" column follows the engine's prefix order (system before tools), as the cause does; it named the tools when both changed and the cause was the system prompt. A hosted
routing_missnow says it is inferred. - Second audit of the specification and the importer. Three independent reviews read
docs/ALGORITHM.mdagainst the code; what they found is fixed here, and the specification now states only what the code does:- Nothing is avoidable when the previous turn had no usage. Its prompt size is not known, so how much of this prompt was reusable is not known either; the whole prompt was counted, appended content included.
- The in-flight rule uses half of the shorter prompt, as the classifier's half-read test does: after a large tool result was appended, a turn that read more than half of the previous prompt could still count as unread, because that was less than half of its own, longer prompt.
- Dates without a year are masked (
Nov 12,12. November), so a system prompt carrying today's date in that form isvolatile_system, notsystem_edit. Only a capitalized month and a day of 1 to 31 count, soRule 1 may be skippedorversion 2.1 May releaseare not dates. - The importer: a status in words (
success,ok,error,failure) is read as such, not taken as success; a first-byte time is read; times in micro- and nanoseconds are read; requests without start and end times are replayed one after another, never invented as overlapping;--sincereplays the whole log and keeps the requests that ended inside the window (a request is recorded when it ends, as live), so the window's first turn is compared with its earlier turn, and the counts printed are the window's; an existing ledger is refused unless--appendis given (--ledgeralone no longer bypasses it). - The specification no longer says the hashes are never exported (they travel with records to a remote collector or record log), and scopes the invariants to well-formed requests.
- Streamed responses record the model that answered. The proxy's stream reader kept the id, usage and diagnostics but not the response's
model, so a streamed request was priced and exported by the model asked for, where a non-streamed one used the model that answered (a dated snapshot behind an alias, say). Now both use the model that answered. - The README shows the architecture, and PyPI shows it too. CI, PyPI and license badges, and the architecture diagram (light and dark) near the top. The PyPI build now rewrites image paths to the raw files on GitHub (a
blob/link is a web page, not an image; PyPI drops<source>and shows the light<img>), and the license badge'sLICENSElink, which the link rewrite missed. - Repository root tidied.
build.logis no longer tracked (*.logis ignored), andWORKLOG.mdmoved todocs/WORKLOG.md; links updated, and it is left out of the source distribution with the other project logs.