Please cite

Algorithm: distribution-based score fusion (also called DBSF, hybrid-dbsf, distribution-based rank fusion)
Author: Michelangiolo Mazzeschi - original source
Published: 2nd November 2023

***Note: According to this arxiv paper, hybrid-dbsf is now the top semantic search algorithm in the world

hybrid-dbsf

Note: This formula was created as the first multimodal algorithm capable of combining images and text in the same search. However, it has been shown how it beats all existing benchmarks even in regular vector search, making it the #1 search algorithm (with new applications in RAG).

$$ S'_{i} = \frac{x_i - (\mu - 3\sigma)}{(\mu + 3\sigma) - (\mu - 3\sigma)} = \frac{x_i - \mu + 3\sigma}{6\sigma} $$

Limitations and Improvements

The current implementation uses a sigma value of 3 as a replacement for minmax. This implementation, however, is based on the assumption that the cosine similarity score distribution is normal, which is never the case.
To obtain even better results, quantiles (respectively [99, 1] for the tail extremes of the distribution) should be used.

$$ S'_{i} = \frac{x_i - P_{1}}{P_{99} - P_{1}} $$

implementations

• llama-index (link)
• Epsilla (link)
• Qdrant (link)
• Julep-ai (link)
• Haystack (link)
• Upstash (link)
• Redis (link)
• Deepsense.ai - Ragbits (link)
• MixPeek (link)

minor implementations

• Connapse (link)
• clarity AI chat components (link)
• Nextcloud MCP server (link)
• DocMind AI (link)
• RAG anything (link)
• agentic-rag-sdk (link)
• rankops (rust package) (link)

citations

• Stanford: VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records (link)
• AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline (link)
  In the paper, it is depicted as the #1 known algorithm for retrieval
• Multi-model fusion and re-ranking for spatial image retrieval (link)
• Verifying Facts in Patient Care Documents Generated by Large Language Models Using Electronic Health Records (link)
• KG-Orchestra: An Open-Source Multi-Agent Framework for Evidence-Based Biomedical Knowledge Graphs Enrichment (link)
• Retrieval-Augmented Generation for Sustainable Material Data Handling in Automotive Value Chain (link)
• Enriched Construction Regulation Inquiry Responses: A Hybrid Search Approach for Large Language Models (link)

books

• Rauf Aliev, Anatomy of eCommerce Search: A Practical Blueprint for Search Engineers and Architects (link)

license

As for all my algorithms, I have chosen the license that (to the extent of my knowledge) should allow for the most open usage and personalization of this algorithm.
If that is not the case, please open an issue outlining the license limitations, and I will do my best to apply the necessary changes to resolve them, as long as they align with the primary objectives of code democratization.