v0.27.0

PromptingTools v0.27.0

Diff since v0.26.2

Added

• Added a keyword-based search similarity to RAGTools to serve both for baseline evaluation and for advanced performance (by having a hybrid index with both embeddings and BM25). See ?RT.KeywordsIndexer and ?RT.BM25Similarity for more information, to build use build_index(KeywordsIndexer(), texts) or convert an existing embeddings-based index ChunkKeywordsIndex(index).

Updated

• For naming consistency, ChunkIndex in RAGTools has been renamed to ChunkEmbeddingsIndex (with an alias ChunkIndex for backwards compatibility). There are now two main index types: ChunkEmbeddingsIndex and ChunkKeywordsIndex (=BM25), which can be combined into a MultiIndex to serve as a hybrid index.

Commits

Merged pull requests:

• Add BM25 Index (#157) (@svilupp)
• Implement E2E Hybrid retrieval (BM25 + embeddings) (#159) (@svilupp)

Closed issues:

• [FR] RAG: Add HybridChunkIndex with BM25 support (#119)