feat: add retriever providers + embedder abstraction#16
Merged
Conversation
added 2 commits
July 10, 2026 01:30
…etrics) The core pipeline was a stub that produced empty results. This wires the full RAG evaluation flow and fixes several correctness/robustness bugs. - Rewrite Pipeline._evaluate_item to retrieve, generate, and score per item - Engine builds providers via providers.factory and resolves metrics from registry - Add mock LLM + retriever for offline dry-run (no API keys required) - Executor.gather for bounded parallel evaluation (honors D006) - Fix recall_at_k to true recall (was binary, duplicated hit_rate) - Pipeline._compute_summary now reports the real error count - Hallucination metric no longer swallows all exceptions - Cache heavy models (SentenceTransformer/Ragas) per metric instance - Align config template + MetricsConfig defaults to registry metric names - Add DatasetItemModel.ground_truth_contexts for retrieval evaluation - Add mock end-to-end integration test + recall_at_k unit test - Correct .ai/03_CONTEXT.md and 05_TASKS.md false 'complete' status
…weaviate, faiss, pgvector, elasticsearch) + embedder abstraction - Add Embedder abstraction (sentence-transformers, mock) wired via RetrieverConfig.embedder - Add in-memory cosine vector retriever (NumPy, no external service) - Add BM25 lexical retriever and generic HTTP/REST retriever (httpx) - Add Qdrant, Pinecone, Weaviate, FAISS, pgvector, Elasticsearch adapters (lazy, optional-dep gated) - Extract shared score-normalization util; refactor Chroma to use it - Add generate_with_usage to Groq provider (required by the pipeline) - Register all providers in the factory; add pyproject optional-dependency extras - Add unit tests for scoring, embedders, memory/BM25/HTTP retrievers, and factory - Add docs/12_retrievers.md and update docs/08_plugin_system.md - Add scripts/run_real_eval.py (verified end-to-end with real Groq + ST embeddings)
…ization; skip HallucinationDetection tests without OPENAI_API_KEY - BM25Retriever: handle empty corpus gracefully (return []) instead of raising - BM25Retriever: convert get_scores ndarray to list before minmax_normalize - test_generation: skip DeepEval-based HallucinationDetection tests when OPENAI_API_KEY is unset (pre-existing CI failure, unrelated to retriever work)
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| Returns: | ||
| List of vectors, one per input text, each of length ``dimension``. | ||
| """ | ||
| ... |
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expands the retriever layer from a single ChromaDB adapter to a full set of pluggable providers, and adds an
Embedderabstraction so vector retrievers can embed queries/documents locally with Sentence-Transformers.New retrievers (all lazy-imported, optional-dep gated)
memory— in-memory cosine vector store (NumPy, no external service) [tested]bm25— lexical BM25 baseline (rank-bm25) [tested]http— generic REST/HTTP search endpoint (httpx) [tested]qdrant,pinecone,weaviate,faiss,pgvector,elasticsearch— server/cloud vector & lexical storesEmbedder abstraction
Embedderbase +SentenceTransformerEmbedder(all-MiniLM-L6-v2) +MockEmbedderRetrieverConfig.embedder; auto-injected into vector retrievers by the factoryOther
_scoring.py); refactored Chroma to use itgenerate_with_usageto the Groq provider (the pipeline requires it)providers/factory.py; addedpyproject.tomlextras (qdrant,pinecone,weaviate,faiss,pgvector,elasticsearch,bm25)docs/12_retrievers.md+ updateddocs/08_plugin_system.mdVerification
scripts/run_real_eval.pyruns the full pipeline end-to-end against the real Groq API (llama-3.3-70b-versatile) with Sentence-Transformers embeddings + the in-memory retriever: 3/3 items evaluated, 0 failures, ~406 tokens, ~307ms avg latency.tests/unit/test_providers/...+ config/metrics suites: 150 passed, 2 skipped.Notes
openai.pyimportstiktokenbut it is not declared in dependencies, sotest_openai.pycannot collect in this environment.Generated with OpenAgent Eval.