Hybrid Search

[davidmezzetti]

The recent sparse vector additions/improvements to txtai (#507, #508) open up the possibility of hybrid vector search (sparse and dense). This change will repurpose the scoring index already on an Embeddings instance for sparse vector search.

The overview on hybrid search is below.

• Modify how scoring works with an embeddings instance.
  • If terms is disabled, work as it does today. Term weighting for word vector based indexes. This use case isn't the most popular but it is useful for low resource languages.
  • If terms is enabled, use scoring as a sparse vector index.
• Merge sparse and dense query results using one of following:
  • Reciprocal Rank Fusion (RRF) when sparse scores aren't normalized
  • Convex combination when sparse scores are normalized