Skip to content

v2.1.5

Choose a tag to compare

@knowusuboaky knowusuboaky released this 26 May 06:04
· 64 commits to main since this release

What's Changed

Proper Hybrid Search with Reciprocal Rank Fusion (RRF)

Implemented industry-standard hybrid retrieval using RRF algorithm:

How it works:

  1. Stage 1: Run k-NN search (semantic similarity) and BM25 search (lexical matching) independently
  2. Stage 2: Fuse results using RRF: score = dense_weight/(k+rank_dense) + sparse_weight/(k+rank_sparse)
  3. Stage 3: Return top results sorted by fused score

Why RRF?

  • Robust to score distribution differences between retrieval methods
  • No need for score normalization (uses ranks, not raw scores)
  • Industry standard (used by Elasticsearch, Cohere, etc.)
  • Works with OpenSearch Serverless (no scripting required)

New methods:

  • text_search() - BM25 lexical search on text_content
  • hybrid_search() - RRF-fused k-NN + BM25

Parameters:

  • rrf_k=60 - Standard RRF constant from literature
  • dense_weight=0.7 - Weight for semantic results
  • sparse_weight=0.3 - Weight for lexical results

Full Changelog: v2.1.4...v2.1.5