discovery(rag): rag_benchmarks.py uses only mock data — never exercises real KB, provides no real retrieval quality signal

[mrveiss]

Discovery

autobot-backend/knowledge/rag_benchmarks.py (Issue #58) benchmarks RAG operations using randomly generated mock embeddings and documents. It never instantiates a real KnowledgeBase, connects to ChromaDB, or runs queries against actual indexed content. All results measure synthetic numpy array operations, not real retrieval quality.

Evidence

@pytest.fixture
def mock_embeddings(self):
    return [[random.random() for _ in range(384)] for _ in range(100)]

@pytest.fixture
def mock_documents(self):
    return [{"id": f"doc_{i}", "content": f"This is test document {i}...",
             "embedding": [random.random() for _ in range(384)]}
            for i in range(1000)]

All benchmark tests operate on these fixtures. No fixture mounts a real KB or ChromaDB collection. The benchmarks measure:

• Raw cosine similarity on random vectors (not semantic similarity)
• Top-k selection on random data (not real document relevance)
• A simulated pipeline with time.sleep() calls for "realism"

The file is also not wired into any CI pipeline, scheduler, or feedback loop.

Impact

• Benchmark "results" are meaningless for measuring actual retrieval quality changes
• Optimizations to AdvancedRAGOptimizer cannot be validated by running benchmarks
• Issue feat(rag): wire rag_benchmarks into RetrievalLearner feedback loop — evaluator contract for automated RAG quality scoring #4676 (wire benchmarks into feedback loop) needs this fixed first — you cannot feed fake scores into RetrievalLearner
• Regressions in real retrieval quality are undetectable via the benchmark suite

Fix

Add a RealKBBenchmarks test class alongside the existing mock class that:

1. Connects to a real (or test-fixture) ChromaDB instance with seeded documents
2. Runs AdvancedRAGOptimizer.advanced_search() with real queries
3. Scores results against known-good ground truth (precision@k, MRR)
4. Can run in CI with a lightweight ChromaDB fixture (in-memory mode)

The mock benchmarks can remain for pure performance microbenchmarks (vector math speed etc).

Affected File

• autobot-backend/knowledge/rag_benchmarks.py — add real-KB benchmark class

Prerequisite For

• feat(rag): wire rag_benchmarks into RetrievalLearner feedback loop — evaluator contract for automated RAG quality scoring #4676 — RAG benchmark evaluator contract (needs real scores to feed RetrievalLearner)

[mrveiss]

✅ Closed — real KB benchmark class implemented

Commit: f52eef6 feat(rag): add RealKBBenchmarks with real ChromaDB precision@k tests (#4697) — in Dev_new_gui

Acceptance Criteria Met:

• ✅ Added TestRealKBBenchmarks class using chromadb.EphemeralClient() — no external services required
• ✅ Seeded 40 domain-relevant documents across 4 topics (Python, database, networking, ML/RAG)
• ✅ Deterministic term-frequency embeddings replace random vectors — precision@k is meaningful
• ✅ 8 tests pass: corpus seeding, 5 precision@k assertions, determinism check, top-1 topic matching
• ✅ @pytest.mark.real_kb marker registered in pytest.ini so CI can run separately: pytest -m real_kb
• ✅ Mock benchmarks preserved untouched
• ✅ Prerequisite for feat(rag): wire rag_benchmarks into RetrievalLearner feedback loop — evaluator contract for automated RAG quality scoring #4676 (RAG benchmark evaluator contract) — real scores now available for RetrievalLearner