Initial RAG 2.0 integration #9068
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This update includes the addition of Retrieval Augmented Generation (RAG) model and relevant utility files into the MindsDB integration. This is achieved via several newly added modules, such as `log_callback_handler.py`, `settings.py`, `vector_store_loader.py`, `rag.py`, etc. A new utility `utils.py` is added to encapsulate common functions for RAG operations. Changes also involve the update of `requirements.txt` and the addition of four types of retriever strategies based on different requirements.
The Retrieval Augmented Generation (RAG) component has been refactored with significant updates on the utility files and additions of new modules. In addition to updating the `settings.py` and `vector_store_loader.py` files, a new RAG class has been constructed in `rag.py` to handle RAG operations more efficiently. There has also been a shift from `DEFAULT_EVALUATION_PROMPT_TEMPLATE` attribute to `DEFAULT_RAG_PROMPT_TEMPLATE`, improving the prompt template for RAG. The retriever strategies have also been refined for easier pipeline creation based on distinct requirements.
# Conflicts: # requirements/requirements.txt
Refactored the code to streamline the creation and management of various retrieval pipelines, including VectorStore, AutoRetriever, and MultiVectorRetriever. The changes have centralized various parameters into the RAGPipelineModel, which supports more precise configuration of retrievers. Also added new tests to validate different retrievers. Some minor bug fixes and adjustments for better clarity and performance were incorporated.
Refactor RAG (Reading Comprehension for Question Answering) integration by removing SQL support to simplify the framework. This includes deleting SQL retrieval components and revising RAG pipeline components. In addition, a new utility for handling vector stores has been introduced to facilitate rate-limited adding of documents. Some minor alterations were made to the callback handlers as well.
The commit removes the SQL related prompt templates - `DEFAULT_TEXT_2_PGVECTOR_PROMPT_TEMPLATE`, `DEFAULT_SQL_RESULT_PROMPT_TEMPLATE` and SQL key from `DEFAULT_SQL_RETRIEVAL_PROMPT_TEMPLATE` in the settings file. The `SQL` enum from `RetrieverType` is also removed. This refactor is likely in response to changes in the app's SQL handling logic or feature set.
An extra catch block has been added to handle all types of exceptions that may occur while fetching data from the database in the vector store loader. This is to ensure that if an unexpected error occurs, it is correctly caught, logged and re-raised.
* Add support for RAG model in Langchain handler This commit introduces integrations for the RAG (Retrieval-Augmented Generation) model in the Langchain handler. It includes various retriever strategies such as SQL, Vector Store, Auto and Multi. Embeddings model support is added and the chunk size for multi-vector retrieval has been increased. A retriever builder is implemented to select the right retriever strategy based on the configuration provided. * Refactor RAG model in Langchain handler and tools The code has been refactored to better integrate the RAG (Retrieval-Augmented Generation) model in the Langchain handler. The updates include the addition of tools for creating embedding models, building retrievers and configuring RAG models. A create_retriever_tool function is now used in the handler to get the RAG query tool. * Refactor Langchain handler to streamline retrieval mode and integrate RAG model building enhancements * Refactor SkillToolController to use SkillType enum for skill type checks Refactor SkillToolController to use updated SkillType enum values and improve error messaging for unsupported skill types * Update method to retrieve RAGPipelineModel fields The method for retrieving the supported parameters of the RAGPipelineModel is updated from using '__fields__.keys()' to 'schema()['properties'].keys()'. This change ensures the correct keys are returned, aligning the functionality with the updated model schema structure. * Add support for RAG model in Langchain handler This commit introduces integrations for the RAG (Retrieval-Augmented Generation) model in the Langchain handler. It includes various retriever strategies such as SQL, Vector Store, Auto and Multi. Embeddings model support is added and the chunk size for multi-vector retrieval has been increased. A retriever builder is implemented to select the right retriever strategy based on the configuration provided. * Refactor RAG model in Langchain handler and tools The code has been refactored to better integrate the RAG (Retrieval-Augmented Generation) model in the Langchain handler. The updates include the addition of tools for creating embedding models, building retrievers and configuring RAG models. A create_retriever_tool function is now used in the handler to get the RAG query tool. * Refactor Langchain handler to streamline retrieval mode and integrate RAG model building enhancements * Refactor SkillToolController to use SkillType enum for skill type checks Refactor SkillToolController to use updated SkillType enum values and improve error messaging for unsupported skill types * Update method to retrieve RAGPipelineModel fields The method for retrieving the supported parameters of the RAGPipelineModel is updated from using '__fields__.keys()' to 'schema()['properties'].keys()'. This change ensures the correct keys are returned, aligning the functionality with the updated model schema structure. * Add retrieval model test and update retriever builder Added a new unit test for retrieval skill in the langchain handler. Made changes in Retriever Builder under integrations/utilities by removing the unsteady SQL Retriever implementation and its related code. * Enhance vector store handling and make test improvements Updated handling of vector store instantiation in 'vector_store.py', enabling vector stores to be created from a type and configuration. Made several improvements to 'test_langchain.py' including changes to how the base test class is extended and adjustments to setup and teardown processes. Also, made other minor improvements across various files for better readability and maintainability. * Update agent creation and retrieval process in Langchain The commit revises the process of agent creation and information retrieval in the Langchain system. It includes an update to the definition of a retriever skill in `test_langchain.py` and changes the default agent type to `AgentType.OPENAI_FUNCTIONS`. The retrieval process now uses a `RAG` pipeline instead of the previous method and added logging to track pipeline processing. Furthermore, it turned on verbosity in the `langchain_handler`. * Remove retriever_builder.py file This commit removes the retriever_builder.py file from the RAG integrations directory within the MindsDB project. The file contained functions for creating and building language retrievers, which are no longer in use.
remove pydantic requirement from main requirements
# Conflicts: # mindsdb/integrations/utilities/rag/loaders/vector_store_loader/vector_store_loader.py # mindsdb/integrations/utilities/rag/pipelines/rag.py # mindsdb/integrations/utilities/rag/rag_pipeline_builder.py # mindsdb/integrations/utilities/rag/retrievers/auto_retriever.py # mindsdb/integrations/utilities/rag/retrievers/multi_vector_retriever.py # mindsdb/integrations/utilities/rag/settings.py # mindsdb/integrations/utilities/rag/vector_store.py # requirements/requirements.txt
update description for retrieval
dusvyat
commented
Apr 12, 2024
| config=params.get('retriever_config', {}), | ||
| description=f'You must use this tool to get more context or information ' | ||
| f'to answer a question about {params["description"]}. ' | ||
| f'The input should be the exact question the user is asking.', |
There was a problem hiding this comment.
updated with base template for now, we can replace with llm gen if needed but this suffices for now
ea-rus
approved these changes
Apr 12, 2024
The requirements for the Langchain Handler have been updated to include the requirements of the Langchain Embedding Handler. This change will ensure all necessary dependencies are installed for successful operation.
Added Pydantic with version 1.8.2 or higher to the project's requirements.txt. This is in line with our decision to use Pydantic for data validation and settings management in the project.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This merges these #9027 into staging branch
Fixes #issue_number
Type of change
(Please delete options that are not relevant)
Verification Process
To ensure the changes are working as expected:
Additional Media:
Checklist: