Issue with RetrievalQA - Invalid Retriever Value when using Multiple Retrievals from FAISS

[murasz]

I am encountering an issue with the RetrievalQA class in the LangChain library when attempting to use multiple retrievals from FAISS.

The codebase:

import os
from langchain.vectorstores import FAISS
from langchain.chat_models import ChatOpenAI
from langchain.embeddings.openai import OpenAIEmbeddings
from dotenv import load_dotenv
from langchain.memory import ConversationBufferMemory
from langchain.chains.router import MultiRetrievalQAChain
from typing import List
from langchain.llms import OpenAI
from langchain.schema.retriever import BaseRetriever, Document

class CustomRetriever(BaseRetriever):
    def __init__(self, retriever_instances: dict):
        super().__init__()
        self.retriever_instances = retriever_instances

    def search(self, query: str) -> List[Document]:
        documents = []
        parallel_results = []
        for retriever in self.retriever_instances.values():
            parallel_results.append(retriever.search(query))
        for result in parallel_results:
            documents.extend(result)
        documents = sorted(documents, key=lambda doc: doc.metadata.get('source'))
        return documents

# Load environment variables
load_dotenv("info.env")
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

# Initialize the FAISS vector database
embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY)

# Define multiple index directories
index_directories = ["faktura", "kaufland"]

# Load indexes into a dictionary of retrievers
dbs = {directory: FAISS.load_local(directory, embeddings) for directory in index_directories}

# Initialize chat model and question answering chain
llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0, openai_api_key=OPENAI_API_KEY)

# Initialize conversation memory
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

# Set the role of the AI
role = "Expert"

# Write the role of the AI
memory.save_context({"input": f"AI: As an {role}, I can help you with your questions."}, {"output": ""})

# Initialize the MultiRetrievalQAChain with custom retriever
retriever_infos = [
    {"name": "faktura", "description": "It's just an invoice in polish", "retriever": dbs["faktura"]},
    {"name": "kaufland", "description": "It's about kaufland file", "retriever": dbs["kaufland"]},
    # Add more retrievers as needed
]

chain = MultiRetrievalQAChain.from_retrievers(
    OpenAI(),
    retriever_infos,
    default_chain=None,
    default_prompt=None,
    default_retriever=None,
    verbose=True
)

while True:
    user_input = input("User: ")

    # Add user input to the conversation history
    memory.save_context({"input": f"User: {user_input}"}, {"output": ""})

    # Check if the user wants to exit
    if user_input == "!exit":
        break

    # Run the chain on the user's query
    response = chain.run(user_input, memory)

    # Print the response with a one-line space and role indication
    print("\n" + f"{role}:\n{response}" + "\n")

The error message indicates after running it:

pydantic.v1.error_wrappers.ValidationError: 1 validation error for RetrievalQA
retriever
  value is not a valid dict (type=type_error.dict)

Expected Behavior:
The RetrievalQA class should accept and handle multiple retrievers from FAISS vector database without raising any errors.
Additional Information:
• I have ensured that the retriever information is correctly formatted as a dictionary.
• The retrievers are created using the FAISS vector database with appropriate embeddings.
• I have reviewed the documentation and examples for the RetrievalQA class but couldn't find a solution specific to using multiple retrievals from FAISS.

• I have tried different suggestions from both gpt-4 and langchain bot.

Any guidance or assistance in resolving this issue and successfully utilizing multiple retrievals from FAISS would be highly appreciated. Thank you!

[dosubot[bot]]

🤖

Hello,

Thank you for providing a detailed description of your issue. The error message you're encountering suggests that the retriever you're trying to use is not being recognized as a valid dictionary. This is likely due to the structure or data type of the retriever_infos parameter in the MultiRetrievalQAChain.from_retrievers method.

In your code, you have defined retriever_infos as follows:

retriever_infos = [
    {"name": "faktura", "description": "It's just an invoice in polish", "retriever": dbs["faktura"]},
    {"name": "kaufland", "description": "It's about kaufland file", "retriever": dbs["kaufland"]},
    # Add more retrievers as needed
]

Each dictionary in the retriever_infos list should contain the keys 'name', 'description', and 'retriever'. The 'retriever' key should correspond to an instance of a class that inherits from BaseRetriever.

Based on the error message and the information you've provided, it seems like the retriever objects you're passing in the retriever_infos list might not be instances of a class that is a subclass of BaseRetriever and that this subclass provides an implementation for the _get_relevant_documents method.

Here's an example of how your retriever_infos list might look:

retriever_infos = [
    {
        "name": "Retriever1",
        "description": "Description for Retriever1",
        "retriever": Retriever1()  # Retriever1 is a subclass of BaseRetriever and implements _get_relevant_documents
    },
    {
        "name": "Retriever2",
        "description": "Description for Retriever2",
        "retriever": Retriever2()  # Retriever2 is a subclass of BaseRetriever and implements _get_relevant_documents
    },
    # More retrievers...
]

Please ensure that the retriever objects in your retriever_infos list meet these requirements. If you're still encountering issues after making these changes, please provide more details about the retriever objects you're using.

I hope this helps! Let me know if you have any other questions.

Sources

• Issue: Dynamically select from multiple retrievers
• libs/langchain/langchain/chains/router/multi_retrieval_qa.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[murasz]

@hwchase17 Do you have any idea? Do you think I should try using Chroma for that purpose? What is missing here?

[xrandox]

I don't have experience with FAISS in particular, but I believe your issue is that you are trying to pass it an instance of your vectorstore rather than a retriever from that vectorstore. Look into the as_retriever method. Again, I'm not experienced with FAISS, but it may be as simple as doing dbs["faktura"].as_retriever()