Skip to content

v1.2-phase2-hybrid

v1.2-phase2-hybrid: feat: Phase 2 - Hybrid retrieval (BGE + BM25)
@ereztdev ereztdev tagged this 04 Feb 23:58 
Add configurable hybrid retrieval combining BGE semantic search
with BM25 keyword search using alpha/beta weighting.

Changes:
- Add HybridRetrievalConfig to core/schema.py (alpha=0.7, beta=0.3)
- Implement Bm25Backend in pipeline/retrieval/bm25_backend.py
  - TF-IDF scoring with k1=1.2, b=0.75
  - No external dependencies (stdlib only)
  - Deterministic for same corpus + query
- Update retrieve() to accept optional hybrid_config
  - When None: BGE-only (Phase 1 behavior, backward compatible)
  - When set: Normalize BGE + BM25 scores, merge with weights
- Add _normalize_scores() with edge case handling
  - Empty corpus, zero variance, single score → [0.5, ...]
- Update baseline.md: changelog, assumptions, data flow

Acceptance:
- Backward compatible: all existing tests pass
- Deterministic: same inputs → same scores
- Functional: scripts/test_hybrid.py shows different rankings
- No new dependencies: requirements.txt unchanged
Assets 2
Loading