PrefixForge

Provider-agnostic prefix-cache orchestration for LLM agents: compile your prompt assembly into a byte-stable, maximally cacheable prefix, and observe the cache hit rate you actually get.

Provider-side prompt caching (Anthropic, OpenAI, Gemini, Bedrock) cuts input cost and latency sharply — but only on a byte-exact prefix match. One dynamic token (a timestamp, a reordered tool schema, a session id) at or before the cache boundary forces a full re-prefill on every request. Agents are the worst case: every tool result mutates the prompt, and tool-call gaps outlive short cache TTLs. PrefixForge fixes the part you control — how the prompt is assembled — with zero manual prefix tuning.

Install

pip install prefixforge                 # zero runtime dependencies
pip install 'prefixforge[langgraph]'    # + LangChain/LangGraph callback
pip install 'prefixforge[anthropic]'    # + live verify harness

Compile

The compiler rewrites token layout, never content:

• T1 — canonicalize tools: deterministic key order in every schema (recursively sorted; list order like enum/required preserved), tools sorted by name. Registration order stops mattering.
• T2 — dynamic-span detection: timestamps, dates, UUIDs, epoch numbers, long hex ids, and your own regex patterns, reported with exact locations.
• T3 — volatile hoisting (Anthropic): system-prompt lines containing dynamic spans move — verbatim, in order — into a second system block placed after the cache breakpoint. Stable lines stay cacheable.
• T4 — breakpoint insertion (Anthropic): cache_control on the last stable system block (caches tools + system) and on the trailing message (multi-turn reuse), only when the estimated prefix clears the model's minimum cacheable size, never exceeding the 4-breakpoint limit, and always respecting markers you placed yourself.

from prefixforge import Compiler, PromptBundle

bundle = PromptBundle(
    provider="anthropic",
    model="claude-sonnet-4-5",
    system=open("system_prompt.txt").read(),
    tools=my_tools,
    messages=[{"role": "user", "content": "hello"}],
)

result = Compiler().compile(bundle)   # advisory: your bundle is not touched
print(result.report())
send_to_api(result.proposed)          # opt in to the transformed bundle

Correctness gate

The compiler ships in advisory mode. Two invariants are enforced in code on every pass, in every mode:

• Content preservation — the multiset of content lines in the transformed bundle equals the original's; a transform that would drop or alter text raises instead of applying.
• Idempotence — compile(compile(x)) == compile(x).

Enable Compiler(mode="apply") in a deployment only after the live equivalence harness passes on your own traffic:

export ANTHROPIC_API_KEY=...
prefixforge verify --replay replay.jsonl

It sends each recorded bundle twice at temperature 0 — original and compiled — and compares the observable output (text + tool calls). Mismatches are flagged for human review.

Observe

parse_usage normalizes every provider's cache accounting (snake_case, camelCase, SDK or REST shape) into one structure; HitRateTracker turns a stream of them into token-weighted hit rate, request hit rate, and input cost vs. the uncached baseline:

from prefixforge import HitRateTracker, parse_usage
from prefixforge.providers import get_profile

tracker = HitRateTracker()
tracker.record(parse_usage("anthropic", response.usage))
print(tracker.summary(get_profile("anthropic")))

LangGraph / LangChain

from prefixforge.integrations.langgraph import PrefixForgeCallback

cb = PrefixForgeCallback()
graph.invoke(inputs, config={"callbacks": [cb]})
print(cb.summary("anthropic"))

CLI

prefixforge compile bundle.json                # advisory report
prefixforge compile bundle.json --apply -o out.json
prefixforge report usage.jsonl                 # aggregate real usage payloads
prefixforge ci-gate --replay replay.jsonl --min-hit-rate 0.6 \
    --baseline baseline.json                   # fail CI on hit-rate regression
prefixforge verify --replay replay.jsonl       # live A/B equivalence harness

ci-gate runs a deterministic offline simulation of a provider prefix cache (exact segment-hash prefix matching, TTL expiry, per-model minimum sizes, breakpoint lookback) over a recorded replay set, and exits nonzero when the compiled hit rate drops below the threshold or regresses against the stored baseline. The simulator is a documented model of provider behavior, not ground truth — live truth comes from report reading real usage fields.

Provider facts (Phase 1)

	Anthropic	OpenAI	Gemini	Bedrock (Claude)
Mechanism	explicit cache_control, max 4	automatic prefix cache	implicit (2.5+)	cachePoint (later phase)
Min prefix	1024–4096 tok (model-dep.)	~1024 tok	~1024–2048 tok	as Anthropic
Cache read	0.1× input price	~0.5×	~0.25×	0.1×
Cache write	1.25× (5m) / 2× (1h)	—	—	1.25×
Usage fields	cache_read/creation_input_tokens	prompt_tokens_details.cached_tokens	cached_content_token_count	cacheRead/WriteInputTokens

Anthropic numbers verified against provider docs (2026-06); OpenAI/Gemini from public docs, re-verified before the Phase 3 capability matrix.

Roadmap

• Phase 1 (this release): Compiler + Observer + LangGraph callback + CI gate.
• Phase 2: Warmer — cost-capped adaptive keep-alive within the cache TTL.
• Phase 3: Router — cache-aware session-affinity routing; full capability matrix.
• Phase 4: self-hosted bridge (vLLM / SGLang) + benchmark suite.

License

Apache-2.0