Semantic Cache Python v0.5.0

betterdb-semantic-cache v0.1.0

Python port of @betterdb/semantic-cache. Embeddings-based semantic cache for AI
workloads backed by Valkey vector search — similarity matching, cost tracking,
multi-modal prompts, embedding cache, and threshold tuning, with built-in
OpenTelemetry and Prometheus instrumentation.

Requires Valkey 8+ with the valkey-search module (vector index support).
Works with ElastiCache for Valkey, Memorystore for Valkey, and MemoryDB.

---

Installation

pip install betterdb-semantic-cache

Optional extras install the provider SDKs alongside the library:

pip install "betterdb-semantic-cache[openai]"
pip install "betterdb-semantic-cache[anthropic]"
pip install "betterdb-semantic-cache[langchain]"
pip install "betterdb-semantic-cache[langgraph]"
pip install "betterdb-semantic-cache[llamaindex]"
pip install "betterdb-semantic-cache[httpx]"   # voyage / cohere / ollama embed helpers
pip install "betterdb-semantic-cache[bedrock]"  # AWS Bedrock embed helper

---

What's included

SemanticCache

Method	Description
initialize()	Create or attach to the vector index
check(prompt)	Similarity lookup — returns hit/miss with confidence and optional cost saved
store(prompt, response)	Store a response with optional cost metadata
store_multipart(prompt, blocks)	Store structured content blocks
check_batch(prompts)	Pipelined batch lookup
invalidate(filter)	Delete entries matching a FT.SEARCH filter
invalidate_by_model(model)	Delete all entries for a model
invalidate_by_category(category)	Delete all entries in a category
stats()	Hit/miss counts and cumulative cost saved
index_info()	Index name, doc count, vector dimension
threshold_effectiveness()	Rolling window analysis and threshold recommendations
threshold_effectiveness_all()	Per-category analysis
flush()	Drop index and delete all cached entries

Provider adapters

Import	Provider
betterdb_semantic_cache.adapters.openai	OpenAI Chat Completions
betterdb_semantic_cache.adapters.openai_responses	OpenAI Responses API
betterdb_semantic_cache.adapters.anthropic	Anthropic Messages
betterdb_semantic_cache.adapters.llamaindex	LlamaIndex ChatMessage[]
betterdb_semantic_cache.adapters.langchain	LangChain BaseCache (async-only)
betterdb_semantic_cache.adapters.langgraph	LangGraph BetterDBSemanticStore

Embedding helpers

Import	Provider
embed.openai	OpenAI Embeddings API
embed.voyage	Voyage AI (httpx, no SDK required)
embed.cohere	Cohere Embed v3 (httpx, no SDK required)
embed.ollama	Ollama local models (httpx, no SDK required)
embed.bedrock	AWS Bedrock Titan / Cohere (boto3)

Bundled default cost table

A default cost table sourced from LiteLLM's model_prices_and_context_window.json
is bundled and refreshed on every release. Cost savings tracking works out of the
box for 1,900+ models — no cost_table configuration required.

Observability

• OpenTelemetry spans on every cache operation
• Prometheus metrics: requests_total, similarity_score, operation_duration_seconds,
  embedding_duration_seconds, cost_saved_total, embedding_cache_total,
  stale_model_evictions_total

Cluster support

Pass a ValkeyCluster client and all SCAN-based operations (flush,
invalidate_by_model, invalidate_by_category) automatically iterate all master nodes.

---

Quick start

import asyncio
import valkey.asyncio as valkey
from betterdb_semantic_cache import SemanticCache, SemanticCacheOptions
from betterdb_semantic_cache.types import CacheStoreOptions
from betterdb_semantic_cache.embed.openai import create_openai_embed

client = valkey.Valkey(host="localhost", port=6379)
cache = SemanticCache(SemanticCacheOptions(
    client=client,
    embed_fn=create_openai_embed(),
    default_threshold=0.12,
))

async def main():
    await cache.initialize()

    result = await cache.check("What is the capital of France?")
    if result.hit:
        print("Cache hit:", result.response)
    else:
        answer = "Paris"  # ... call your LLM ...
        await cache.store(
            "What is the capital of France?", answer,
            CacheStoreOptions(model="gpt-4o", input_tokens=20, output_tokens=5),
        )

asyncio.run(main())

---

Full changelog

See CHANGELOG.md for detailed history.