Time-aware semantic caching for LLM responses
Cache LLM responses by semantic meaning AND freshness. Stop serving stale answers.
- What is temporal-cache?
- The Problem
- How It Works
- Installation
- Quick Start
- Python API Reference
- Staleness Classes
- Architecture
- Configuration
- Testing
- Contributing
- Roadmap
- License
temporal-cache is a semantic cache for LLM API responses that understands when an answer was relevant, not just what it says.
Traditional semantic caches store responses and return them when a similar query arrives. But they don't know if that response is still correct. A cached answer about "current US president" from 2024 is semantically relevant in 2026 — but wrong.
temporal-cache solves this with a StalenessClass classifier that evaluates how time-sensitive a query is, and an optional LLM-based semantic classifier that detects when cached content has drifted from current facts.
Use cases:
- Knowledge bases with evolving information
- Customer support bots with changing policies
- Financial/news Q&A systems
- Any LLM application where answers expire
┌─────────────────────────────────────────────────────────────────┐
│ Traditional Semantic Cache │
│ │
│ Query: "What is the capital of France?" │
│ Cache hit → "Paris" ✓ │
│ │
│ Query: "What's the current CEO of Tesla?" │
│ Cache hit → "Elon Musk" (cached 2024) │
│ Reality: CEO changed in 2025 ✗ │
│ │
│ Problem: Cache doesn't know the answer is stale │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────┐
│ temporal-cache │
│ │
│ Query: "What's the current CEO of Tesla?" │
│ StalenessClass: DYNAMIC (changes frequently) │
│ Action: Revalidate → check if cache is still valid │
│ Result: Fresh answer or cache with metadata about freshness │
└─────────────────────────────────────────────────────────────────┘
┌──────────────┐
│ Incoming │
│ Query │
└──────┬───────┘
│
▼
┌────────────────────────┐
│ StalenessClass │
│ Classifier │
│ │
│ STATIC │ EVOLVING │ │
│ SEMI- │ DYNAMIC │ │
│ STATIC │ │ │
└────────┬───────────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ STATIC │ │ SEMI- │ │ DYNAMIC │
│ │ │ STATIC │ │ │
│ Cache │ │ Cache │ │ Reval- │
│ Forever │ │ + TTL │ │ idate │
└─────────┘ └─────────┘ └─────────┘
│ │ │
│ │ │
▼ ▼ ▼
┌─────────────────────────────────────────┐
│ Semantic Similarity │
│ (embedding) │
└──────────────────┬──────────────────────┘
│
┌─────────────┴─────────────┐
│ │
▼ ▼
Hit (≥ threshold) Miss
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│ Return │ │ Call LLM │
│ Cached │ │ Store with │
│ Response │ │ metadata │
└──────────┘ └──────────────┘
Each query is classified into one of four staleness classes:
┌──────────────────────────────────────────────────────────────────┐
│ │
│ Query Analysis │
│ ═══════════════ │
│ │
│ "What is 2+2?" → STATIC │
│ "What is Python?" → STATIC │
│ "Explain quantum computing" → SEMI_STATIC │
│ "What are Python best practices?" → SEMI_STATIC │
│ "Who is the current US president?"→ DYNAMIC │
│ "Tesla stock price today" → DYNAMIC │
│ "Latest news about OpenAI" → EVOLVING │
│ "What changed in Python 3.12?" → EVOLVING │
│ │
└──────────────────────────────────────────────────────────────────┘
pip install temporal-cacheFrom source:
git clone https://github.com/yourusername/temporal-cache.git
cd temporal-cache
pip install -e .With Redis support:
pip install temporal-cache[redis]- Python 3.9+
sentence-transformers(for embeddings)redis(optional, for Redis backend)
from temporal_cache import TemporalCache
# Initialize with default settings (in-memory backend)
cache = TemporalCache()
# Or with Redis backend
cache = TemporalCache(
backend="redis",
redis_url="redis://localhost:6379",
embedding_model="all-MiniLM-L6-v2"
)
# Store a response
cache.set(
query="What is the capital of France?",
response="Paris is the capital of France.",
metadata={"source": "wikipedia", "verified": True}
)
# Retrieve a response (semantic match)
result = cache.get("What's the capital of France?")
if result:
print(f"Cache hit: {result.response}")
print(f"Staleness: {result.staleness_class}")
print(f"Freshness: {result.freshness_score}")
else:
print("Cache miss - call your LLM here")TemporalCache(
backend: str = "memory", # "memory" or "redis"
redis_url: str = "redis://localhost:6379",
embedding_model: str = "all-MiniLM-L6-v2",
similarity_threshold: float = 0.85,
ttl_config: dict = None, # Custom TTL per staleness class
llm_classifier: bool = False, # Enable LLM-based revalidation
classifier_model: str = None # Model for LLM classifier
)set(query, response, metadata=None, staleness_class=None)
Store a query-response pair in the cache.
| Parameter | Type | Description |
|---|---|---|
query |
str |
The input query |
response |
str |
The LLM response to cache |
metadata |
dict |
Optional metadata (source, timestamp, etc.) |
staleness_class |
StalenessClass |
Override auto-detected class |
get(query, force_refresh=False)
Retrieve a cached response by semantic similarity.
| Parameter | Type | Description |
|---|---|---|
query |
str |
The query to look up |
force_refresh |
bool |
Bypass cache, always return None |
Returns CacheResult or None.
invalidate(query=None, pattern=None)
Remove entries from cache.
# Invalidate specific query
cache.invalidate(query="What is the capital of France?")
# Invalidate by pattern
cache.invalidate(pattern="*stock*")stats()
Return cache statistics.
stats = cache.stats()
# {
# "total_entries": 1523,
# "hits": 892,
# "misses": 231,
# "stale_rejections": 45,
# "avg_freshness_score": 0.87
# }@dataclass
class CacheResult:
response: str
similarity_score: float
staleness_class: StalenessClass
freshness_score: float # 0.0 (stale) to 1.0 (fresh)
cached_at: datetime
metadata: dictclass StalenessClass(Enum):
STATIC = "static" # Never changes
SEMI_STATIC = "semi_static" # Changes rarely
EVOLVING = "evolving" # Changes periodically
DYNAMIC = "dynamic" # Changes frequently| Class | Description | TTL | Revalidation | Examples |
|---|---|---|---|---|
| STATIC | Facts that don't change | ∞ | Never | "What is π?", "2+2", "What is DNA?" |
| SEMI_STATIC | Rarely changes | 30 days | On major events | "Python docs", "Physics laws", "Best practices" |
| EVOLVING | Changes periodically | 24 hours | On schedule | "Python 3.12 features", "Company policies" |
| DYNAMIC | Changes constantly | 1 hour | Every query | "Stock price", "Current president", "Live news" |
cache = TemporalCache(
ttl_config={
"static": None, # No expiration
"semi_static": 86400 * 30, # 30 days
"evolving": 86400, # 1 day
"dynamic": 3600 # 1 hour
}
)temporal_cache/
├── __init__.py
├── cache.py # TemporalCache engine
├── classifiers/
│ ├── __init__.py
│ ├── staleness.py # StalenessClass rule-based classifier
│ └── llm.py # LLM-based semantic classifier
├── backends/
│ ├── __init__.py
│ ├── memory.py # In-memory backend (dict)
│ └── redis.py # Redis backend
├── embeddings.py # Embedding generation
├── models.py # CacheResult, CacheEntry dataclasses
├── config.py # Configuration and defaults
└── utils.py # Helpers
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ User Query │
│ │ │
│ ▼ │
│ ┌─────────────┐ ┌──────────────────┐ │
│ │ Staleness │────▶│ Embedding │ │
│ │ Classifier │ │ Generator │ │
│ └──────┬──────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────┐ ┌──────────────────┐ │
│ │ TTL │ │ Backend │ │
│ │ Manager │ │ (Memory/Redis) │ │
│ └──────┬──────┘ └────────┬─────────┘ │
│ │ │ │
│ └──────────┬──────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────┐ │
│ │ Response Validator │ │
│ │ │ │
│ │ - Check similarity │ │
│ │ - Check freshness │ │
│ │ - LLM revalidation │ │
│ └──────────┬───────────┘ │
│ │ │
│ ┌──────────┴──────────┐ │
│ │ │ │
│ ▼ ▼ │
│ Cache Hit Cache Miss │
│ (return) (call LLM, store) │
│ │
└─────────────────────────────────────────────────────────────────────┘
TEMPORAL_CACHE_BACKEND=redis
TEMPORAL_CACHE_REDIS_URL=redis://localhost:6379
TEMPORAL_CACHE_SIMILARITY_THRESHOLD=0.85
TEMPORAL_CACHE_DEFAULT_TTL=3600# temporal_cache.yaml
backend: redis
redis_url: redis://localhost:6379
embedding_model: all-MiniLM-L6-v2
similarity_threshold: 0.85
llm_classifier:
enabled: true
model: gpt-4
temperature: 0.0
ttl:
static: null
semi_static: 2592000
evolving: 86400
dynamic: 3600Run the test suite:
# All tests
pytest
# With coverage
pytest --cov=temporal_cache --cov-report=html
# Verbose output
pytest -vThe test suite includes 16 tests covering:
- Cache storage and retrieval
- Semantic similarity matching
- Staleness classification accuracy
- TTL expiration behavior
- Redis backend integration
- LLM classifier revalidation
- Edge cases and error handling
Contributions are welcome! Here's how to get started:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes
- Run the tests (
pytest) - Submit a pull request
git clone https://github.com/yourusername/temporal-cache.git
cd temporal-cache
pip install -e ".[dev]"
pre-commit install- Follow PEP 8
- Use type hints
- Write docstrings for public APIs
- Add tests for new features
- v0.2.0 - Async support (
async/await) - v0.2.0 - PostgreSQL backend
- v0.3.0 - Bloom filter pre-check for fast misses
- v0.3.0 - Distributed cache invalidation
- v0.4.0 - Embedding cache (reuse embeddings across queries)
- v0.4.0 - Web dashboard for cache analytics
- v1.0.0 - Production-ready release
MIT License. See LICENSE for details.
Built with care for the LLM community