Repository navigation
Memory Quality System Evaluation
Date: December 8, 2025 System Version: v8.48.3 Data Basis: 4,762 memories
The Memory Quality System is technically successful with 75% local ONNX usage and zero costs. Performance is excellent at <50ms latency. However, challenges exist in quality assessment, primarily stemming from limitations of the ONNX model used.
| Goal | Target | Current | Status |
|---|---|---|---|
| Local SLM Usage | >95% | 75% | |
| Search Latency | <100ms | 45ms (avg) | ✅ Exceeded |
| Monthly Cost | $0 | $0 | ✅ Achieved |
| Quality Score Distribution | Balanced | Bimodal (many 1.0, many 0.0) |
- High Coverage: 95% of memories have quality scores (4,521 of 4,762)
- Local-First Success: 75% use ONNX local (3,570 memories)
- Zero Cost: No API costs, fully local processing
- Low Latency: Quality-boosted search only +17% overhead (6.5ms)
- Stability: System runs without errors or crashes
- Bimodal Distribution: Many perfect (1.0) and zero (0.0) scores
- Low Average: Average score 0.469 (expected: ~0.6-0.7)
- Self-Matching Bias: ONNX model returns 1.0 for identical query-content pairs
- 40.4% Low Quality: 1,924 memories with score <0.5
Distribution by Tier:
🟢 High Quality (≥0.7): 1,533 memories (32.2%)
🟡 Medium Quality (0.5-0.7): 1,305 memories (27.4%)
🔴 Low Quality (<0.5): 1,924 memories (40.4%)
Average Score: 0.469
Interpretation:
- Distribution is not normal, but bimodal
- Many memories cluster at 1.0 (perfect scores) or near 0.0
- This indicates the documented ONNX issue: Self-matching queries
ONNX Local: 3,570 memories (75.0%)
Implicit: 951 memories (20.0%)
No Score: 241 memories (5.0%)
Interpretation:
- 75% ONNX: Excellent! Local-first design is working
- 20% Implicit: Correct - system-generated associations (score 0.5)
- 5% No Score: Acceptable - likely very old or malformed memories
Variant Mean Median P95 Overhead
─────────────────────────────────────────────────────
Standard (100% sem) 38.2ms 38.7ms 46.4ms baseline
Quality Boost 0.3 44.7ms 45.1ms 48.1ms +17.1%
Quality Boost 0.5 45.7ms 44.8ms 53.0ms +19.7%
Interpretation:
- Very low latency increase: +6-7ms is negligible
- Sub-50ms performance: Excellent for real-time applications
- Scalable: Linear with quality_weight
Test Query: "session start hook configuration"
With Quality Boost (0.3):
Top 3 Results:
1. Score 0.882 (Sem 0.832, Qual 0.999) ← SessionEnd Hook Doc
2. Score 0.876 (Sem 0.825, Qual 0.996) ← SessionEnd Hook Guide
3. Score 0.859 (Sem 0.817, Qual 0.957) ← Session Hook Update
Without Quality Boost:
Top 3 Results:
1. Relevance 0.83 ← SessionEnd Hook Doc (same)
2. Relevance 0.82 ← SessionEnd Hook Guide (same)
3. Relevance 0.82 ← Doc Update (different memory)
Interpretation:
- Quality boost changes ranking minimally
- Effect is small because all top results are already high-quality (0.95-0.999)
- For this query, quality boost provides no significant advantage
The model used (ms-marco-MiniLM-L-6-v2) is a cross-encoder for relevance ranking, not for absolute quality assessment.
How it works:
Input: (Query, Document)
Output: Relevance Score (0-1)
The problem:
- During bulk evaluation, we generate queries from memory tags
- Example: Memory with tags "session-start, hook, config" → Query: "session start hook config"
- This query is nearly identical to memory content
- Result: Artificially high score ~1.0
Evidence from database:
-- Sample of suspiciously high scores
Score 1.000: "MCP Memory Service v6.20.0 Release..."
Score 1.000: "Enhanced Claude Branch Automation..."
Score 1.000: "Azure VM Deployment Troubleshooting..."All of these have generic tags that generate generic queries, which then match perfectly.
Examples:
Score 0.000: "CHANGELOG, README, and CLAUDE.md for v8.44.0..."
Score 0.000: "SessionEnd and other hooks work fine on Windows..."
Score 0.000: "HOOKS UPDATE INVESTIGATION - 72 Commits Pulled..."
Possible causes:
- Query generation from tags doesn't work well for these memories
- Content is too fragmented or too technical
- Tags are too generic ("documentation", "update", "investigation")
Status Quo: Quality boost is opt-in (MCP_QUALITY_BOOST_ENABLED=false)
Recommendation: Keep it! Quality boost should not be default-enabled because:
- Benefit is minimal with already high-quality memories
- 17% latency overhead for marginal gain
- Most relevant memories are already high-quality
When to use Quality Boost:
- Large memory databases (>10,000 memories)
- Explicitly searching for "best practices" or "authoritative" content
- Combined with other ranking factors (recency, tags)
Update docs/guides/memory-quality-guide.md:
⚠️ **ONNX Ranker Limitations**
The local ONNX model (`ms-marco-MiniLM-L-6-v2`) is designed for
**query-document relevance ranking**, not absolute quality assessment.
Known Issues:
- Self-matching queries produce artificially high scores (~1.0)
- Bulk evaluation with tag-generated queries causes bias
- Not suitable for absolute quality measurement
Recommended Usage:
- Use for **relative ranking** during search
- Do NOT use scores for absolute quality thresholds
- Combine with implicit signals (access count, recency)Current approach:
# Simplified example
tags = ["session-start", "hook", "troubleshooting"]
query = " ".join(tags) # "session-start hook troubleshooting"
score = ranker.predict(query, memory.content) # Often ~1.0!Better approach:
# Generate diverse queries from content
queries = [
extract_key_question(memory.content), # "How to fix hook connection?"
extract_problem_description(memory.content), # "Hook fails to connect"
extract_solution_summary(memory.content) # "Update config endpoint"
]
score = average([ranker.predict(q, memory.content) for q in queries])Combine multiple signals instead of just ONNX:
quality_score = (
0.30 * onnx_relevance_score + # Cross-encoder
0.25 * access_count_normalized + # How often retrieved?
0.20 * recency_score + # How recent?
0.15 * tag_quality_score + # Well-structured?
0.10 * content_completeness # Complete?
)Benefits:
- Less susceptible to single model bias
- Uses actual user feedback (access patterns)
- Rewards current, relevant memories
Activate the manual rating system:
# Allow users to rate memories
curl -X POST http://127.0.0.1:8000/api/quality/memories/{hash}/rate \
-d '{"rating": 1, "feedback": "Very helpful"}'Implementation:
- Show "👍 Helpful?" button in dashboard
- Track thumbs-up/down for each memory retrieval
- Weight user ratings higher than AI scores (factor 2-3x)
Run controlled experiment:
# Test setup
users_group_a = enable_quality_boost(weight=0.3)
users_group_b = standard_search()
# Metrics after 1 week
track_metrics(
retrieval_precision="How many results were useful?",
user_satisfaction="Did user click/use the memory?",
search_time="Time to find relevant memory"
)Candidates:
-
SentenceTransformers:
all-mpnet-base-v2(better semantic representation) - Specialized Models: Train on actual memory quality data
- LLM-as-Judge: Groq/Gemini with structured prompts (more expensive, but more accurate)
Evaluation Criteria:
- Correlation with user feedback
- Bias susceptibility
- Performance (latency, cost)
Implement automatic archival based on Quality + Access:
if quality_score < 0.3 and last_accessed > 90_days:
archive_memory(memory) # Move to cold storage
if quality_score > 0.8 and access_count > 10:
pin_memory(memory) # Keep in hot cache| Metric | Current | Target (3 Months) |
|---|---|---|
| Average Quality Score | 0.469 | 0.60+ |
| High Quality Ratio | 32.2% | 45%+ |
| User Rating Coverage | 0% | 20%+ |
| Quality-Boost Usage | Opt-in | 30% of queries |
| False Positive Rate (1.0 scores) | ~25% | <10% |
- Local-First Architecture: 75% ONNX usage, $0 cost
- Performance: <50ms latency, production-ready
- Coverage: 95% of memories have scores
- Stability: No crashes, errors, or data loss
- Integration: Seamless with existing search/retrieval
- ONNX for Absolute Quality: Model designed for relevance, not quality
- Tag-Based Query Generation: Creates self-matching bias
- Binary Distribution: Too many 0.0 and 1.0 scores
- No User Feedback Loop: Missing ground truth for validation
The ONNX cross-encoder is excellent for what it was designed for (ranking documents by query relevance), but not suitable for absolute quality assessment without significant prompt engineering or model fine-tuning.
The way forward: Hybrid scoring with multiple signals + user feedback.
- ✅ Document ONNX limitations clearly
- ✅ Keep quality boost as opt-in feature
- ✅ Improve query generation for bulk evaluation
- 🔄 Implement hybrid quality scoring (ONNX + implicit signals)
- 🔄 Activate manual rating system in dashboard
- 🔄 Set up A/B test for 1 week
- 🔮 Evaluate alternative quality models
- 🔮 Implement quality-driven memory lifecycle
- 🔮 Collect 20%+ user ratings for ground truth
- Total Memories: 4,762
- Memories with Quality Scores: 4,521 (95%)
- ONNX Provider: 3,570 (75%)
- Implicit Provider: 951 (20%)
- No Score: 241 (5%)
- Test Queries: 10 diverse queries
- Variants Tested: Standard, Quality Boost 0.3, Quality Boost 0.5
- Metrics: Mean, Median, P95 latency
- Environment: v8.48.3, Hybrid backend, 4,762 memories
- Query Types: Recent troubleshooting, configuration, features, problems, processes
- Comparison: Standard semantic search vs quality-boosted search
- Sample Size: 5 representative queries
- Metrics: Relevance, quality scores, ranking changes
Conclusion: The Memory Quality System is a solid first step, but not yet production-ready for quality-based ranking. The technical infrastructure is excellent, but the quality assessment methodology needs improvement.
Bottom Line: Use the system for relative ranking within search results, but not as an absolute quality threshold for memory retention/archival decisions.