prompt-cache-bench

Benchmark Anthropic prompt caching: cache hit rate, latency delta between miss and hit, and exact USD cost savings. Tells you whether caching is worth it for your specific usage pattern before you commit to designing around it.

Install

pip install -e .

Quickstart

export ANTHROPIC_API_KEY=sk-ant-...
python -m promptcachebench --model claude-sonnet-4-6 --calls 10 --context path/to/context.txt

Model:           claude-sonnet-4-6
Context tokens:  12,847
Total calls:     10
Cache hit rate:  90.0%

── Latency ───────────────────────────────────────
  Cache miss avg:    3,420 ms
  Cache hit avg:       310 ms
  Speedup:            11.0×

── Cost (10 calls) ───────────────────────────────
  Without caching:   $0.0462
  With caching:      $0.0089
  Saved:             $0.0373  (80.7%)

── Per-call breakdown ────────────────────────────
  call  1  MISS  3241ms  $0.0046
  call  2  HIT    298ms  $0.0009
  call  3  HIT    305ms  $0.0009
  ...

As a library

from promptcachebench import run_bench

result = run_bench(
    context=long_system_prompt,
    queries=["Summarize.", "List key points.", "What are the risks?"],
    model="claude-sonnet-4-6",
    max_tokens=256,
)

print(f"Hit rate:  {result.hit_rate:.1%}")
print(f"Speedup:   {result.speedup:.1f}×")
print(f"Saved:     ${result.cost_saved_usd:.4f}")

What it measures

Cache hit rate — what fraction of calls read from cache vs. write. The first call always writes; subsequent calls should read.

Latency delta — real wall-clock difference between a cache miss and a hit. Typically 10–15× for long contexts.

Cost savings — exact USD using the published pricing model: cache write = 125% of base input, cache read = 10% of base input. Savings compound quickly when the same large context is reused across many queries.

Pricing model

Model	Input	Output	Cache write	Cache read
claude-opus-4-8	$15.00/MTok	$75.00/MTok	$18.75/MTok	$1.50/MTok
claude-sonnet-4-6	$3.00/MTok	$15.00/MTok	$3.75/MTok	$0.30/MTok
claude-haiku-4-5	$0.80/MTok	$4.00/MTok	$1.00/MTok	$0.08/MTok

Testing

pytest -q

...........
11 passed in 0.01s