Semantic cache for LLM APIs

A drop-in caching proxy that sits between your app and any OpenAI-compatible LLM API. It embeds each incoming prompt, looks for a stored answer to a semantically similar prompt, and serves that answer instantly when it finds one. Exact repeats and close paraphrases never hit the upstream model, so you pay for fewer tokens and your users wait less.

Point your existing client at this proxy by changing one base URL. Nothing else in your code changes.

What you get

A cache hit returns in roughly a millisecond instead of waiting on the provider. On a 2,000-request load test with a realistic mix of repeated and unique prompts, the proxy serves a meaningful share of traffic from cache and reports the exact hit rate, the latency gap between hits and misses, and the estimated spend it avoided. Those three numbers are the headline you can put on a portfolio or a cost review.

The whole stack runs offline with no API key, because it ships with a deterministic local embedder and a mock LLM provider. Flip two environment variables to switch to real OpenAI embeddings and real providers.

How it works

                   +-------------------------------------------+
   client  ---->   |  FastAPI proxy  (/v1/chat/completions)    |
 (OpenAI SDK)      |                                           |
                   |   1. resolve policy (threshold, TTL)      |
                   |   2. embed the query                      |
                   |   3. KNN search within the namespace      |
                   |        hit  -> return cached answer        |
                   |        miss -> call provider, then store   |
                   +----------+------------------+--------------+
                              |                  |
                       embeddings           provider router
                    (OpenAI | local)   (OpenAI | Anthropic | Ollama | mock)
                              |
                       vector store
                    (Redis KNN | memory)

        metrics  ->  Prometheus  ->  Grafana dashboard

Two pieces decide whether a request is a hit. The namespace isolates prompts that must never match each other, even with identical user text. The system prompt, the model, and the decoding parameters all go into the namespace hash, so a support assistant and a code reviewer asking "what do you think?" get different answers. Within a namespace, the proxy embeds the user turns and runs a nearest-neighbor search. A match counts as a hit when its cosine similarity clears the threshold for that request type.

Quick start (offline, no API key)

docker compose up --build -d

That starts four services: the proxy on port 8000, Redis with the vector search module on 6379, Prometheus on 9090, and Grafana on 3000. Send a request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mock-1","messages":[{"role":"user","content":"what is a vector database"}]}'

The first call returns with header X-Cache: MISS. Send the same request again and the header reads X-Cache: HIT, with the answer coming back far faster. Then run the load test:

python -m load_test.load_test --requests 2000 --concurrency 20

It prints the hit rate, latency percentiles for hits and misses, and the estimated cost saved. Open Grafana at http://localhost:3000 (login admin / admin) to watch the same numbers move while the test runs.

Using real providers

Set the embedder to OpenAI and supply keys. With real embeddings, paraphrases and reworded questions start hitting the cache, not just exact repeats.

export SEMCACHE_EMBEDDING_PROVIDER=openai
export SEMCACHE_OPENAI_API_KEY=sk-...
export SEMCACHE_ANTHROPIC_API_KEY=sk-ant-...   # optional
docker compose up --build -d

The router picks a provider from the model name: anything starting with gpt, o1, o3, or o4 goes to OpenAI, claude goes to Anthropic, ollama/ goes to a local Ollama server, and mock goes to the built-in mock. The map lives in app/config.py, so adding a provider is a config change.

The API

The proxy mirrors the OpenAI chat completions contract, including streaming. Streamed hits and streamed misses both come back as standard server-sent event chunks, so a client cannot tell from the wire format whether the bytes came from cache or from a live model.

Optional request headers add control without breaking the contract:

Header	Values	Effect
X-Cache-Bypass	true	Skip the cache for this one request
X-Cache-Request-Type	classification, general, creative	Override the inferred request type
X-Cache-TTL-Tier	stable, default, volatile	Set how long the answer is cached
X-Cache-Threshold	a float like 0.97	Override the similarity threshold

Every response carries X-Cache (HIT, MISS, or BYPASS), X-Cache-Similarity, X-Cache-Namespace, and X-Cache-Request-Type.

Operational endpoints sit under /admin:

• GET /admin/stats reports store size, sample count, and the active thresholds.
• GET /admin/threshold-analysis sweeps a range of thresholds and shows the hit rate at each one, computed over recent lookups.
• GET /admin/near-misses lists lookups that landed just under the threshold, which are the prompts worth a human glance before you loosen it.

Health probes are GET /health (process is up) and GET /ready (store is reachable). Prometheus scrapes GET /metrics.

Cache behavior

Different request types tolerate different amounts of fuzziness, so the threshold adapts. Classification has a tiny output space and can reuse an answer from a fairly different prompt, so its default threshold is 0.90. General questions sit at 0.95. Creative generation needs near-identical prompts before reuse makes sense, so it sits at 0.98, and you can switch caching off for it entirely with SEMCACHE_DISABLE_CACHE_FOR_CREATIVE=true.

The request type is inferred from temperature (low looks like classification, high looks like creative) and can be overridden per request with a header. The header always wins, because the heuristic is sometimes wrong.

Time to live works in tiers. A definition of quicksort is stable for years; the current weather is stale in minutes. Stable answers default to seven days, the general default is six hours, and volatile answers expire in five minutes. The caller picks a tier with a header, and the proxy falls back to the default when it does not.

Configuration

All settings come from environment variables with the SEMCACHE_ prefix, or from a .env file. The values that matter most:

Variable	Default	Notes
SEMCACHE_EMBEDDING_PROVIDER	local	local for offline, openai for real embeddings
SEMCACHE_STORE_BACKEND	memory	memory for dev, redis for production
SEMCACHE_REDIS_URL	redis://localhost:6379	Redis with the search module
SEMCACHE_DEFAULT_SIMILARITY_THRESHOLD	0.95	Used by the near-miss analyzer
SEMCACHE_OPENAI_API_KEY	empty	Required for the OpenAI embedder and provider

Testing

The suite runs anywhere with no network and no Redis, because the tests force the local embedder, the in-memory store, and the mock provider.

pip install -r requirements-dev.txt
pytest -q

The tests cover key namespacing, policy resolution, the embedder's similarity ordering, the full cache miss-then-hit cycle including namespace isolation, and the API end to end through a real TestClient.

Project layout

app/
  api/            chat route, admin endpoints, health, SSE helpers
  core/           cache orchestrator, keys, policies, pricing
  embeddings/     embedder interface, OpenAI client, local embedder
  providers/      provider interface, OpenAI/Anthropic/Ollama/mock, router
  store/          vector store interface, Redis backend, memory backend
  observability/  Prometheus metrics, structured logging
  models/         request and response schemas
  config.py       settings
  dependencies.py object graph construction
  main.py         FastAPI app and lifespan
tests/            unit and integration tests
load_test/        prompt generator and the load test runner
monitoring/       Prometheus config, Grafana datasource and dashboard

Known limitations

The local embedder is a hashing trick, not a real model. It scores exact repeats at 1.0 and shares structure between similar strings, which is enough to exercise the cache and run the load test, but only the real OpenAI embedder gives true paraphrase matching. Expect the offline demo to hit mostly on exact repeats.

The lookup samples that feed the threshold tuner and near-miss analyzer live in process memory, so they reset on restart and are per-instance. For a multi-instance deployment you would push those samples to Redis. The hooks are isolated in SemanticCache, so that change stays small.

Cost savings are an estimate from a static price table in app/core/pricing.py. Update it when provider pricing changes, and read the number as a directional figure rather than an invoice.