Multi-pdf Capabilities

[dandonarahul2002]

Enhanced Multi-PDF RAG Capabilities and Optimized Reranking

Overview

This pull request significantly improves our RAG (Retrieval-Augmented Generation) system by extending single-PDF capabilities to support multiple PDFs and implementing an optimized reranking algorithm.

Key Changes

1. Multi-PDF RAG Support

• Modified rag-utils.ts to handle multiple PDF documents simultaneously
• Enhanced similarity search to work across multiple vector databases

2. Optimized Reranking Algorithm

Implemented a new bm25Rerank function with the following optimizations:

• Preprocessed query terms to filter out single-character words
• Precomputed IDF scores for improved efficiency
• Utilized a single regex for term matching, reducing string operations
• Implemented more efficient term frequency counting using a Map
• Improved BM25 score calculation for better result ranking
• Reset parameters of RecursiveCharacterTextSplitter to default values as it showed better results while manual testing

3. Type Safety Improvements

• Added a new ScoredDocument interface extending Document to include a score property
• Updated similaritySearch function to use the new bm25Rerank function, returning ScoredDocument[]

4. Text Splitting Adjustment

• Reset parameters of RecursiveCharacterTextSplitter to default values based on improved results from manual testing

Performance Impact

These changes are expected to significantly improve the accuracy of our RAG system, particularly for queries involving multiple PDFs or large document sets.

Next Steps

• Potential to improvise Reranking using Cross-Encoders (Couldn't find the funtionality yet to support Js(ONNX) models for sBert)
• Explore potential for further optimizations in vector search and embedding processes

Please review these changes, paying particular attention to the reranking algorithm and multi-PDF handling logic.

[kartikm7]

Thank you so much!

[dandonarahul2002]

God bless

…

On Sat, 23 Nov 2024 at 3:38 AM, Kartikeya Mishra ***@***.***> wrote: Thank you so much! — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AU32KXHVZ6B6KMKPSAKAN732B6TNLAVCNFSM6AAAAABPCXOACKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJUHE2DOOBVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>