Query Regarding Metadata-Based Filtering for Vector Search in Langchain

[mohit19906]

I have a large number of PDF documents that I would like to interact with using Langchain. My specific requirement is that I don't want all the documents to be subjected to vector search for each question. Instead, while storing the PDF files in the VectorDB, I would like to assign different sets of PDF files to different User IDs, which I believe could be achieved using Metadata.

When it comes to retrieval, my desire is for the vector search to only be performed on a specific set of PDF files associated with a particular User ID. Can Langchain filter documents based on Metadata and then perform vector search on these filtered documents?

[dosubot[bot]]

🤖

Yes, LangChain can indeed filter documents based on Metadata and then perform a vector search on these filtered documents. This can be achieved by extending the VectorStoreRetriever class and overriding the get_relevant_documents method to filter the documents based on the source path. Here's how you can do it:

class FilteredRetriever(VectorStoreRetriever):
    vectorstore: VectorStoreRetriever
    search_type: str = "similarity"
    search_kwargs: dict = Field(default_factory=dict)
    filter_prefix: str
    
    def get_relevant_documents(self, query: str) -> List[Document]:
        results = self.vectorstore.get_relevant_documents(query=query)
        return [doc for doc in results if doc.metadata['source'].startswith(self.filter_prefix)]

And use it like this:

filtered_retriever = FilteredRetriever(vectorstore=store.as_retriever(), filter_prefix=source_filter)

chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=filtered_retriever,
    memory=st.session_state.memory,
    verbose=True,
    return_source_documents=True,
)

This solution was provided in a similar issue titled Filtering retrieval with ConversationalRetrievalChain.

Additionally, the Weaviate class in LangChain provides the add_texts method which allows you to add texts to the vector store along with their associated metadata. The add_texts method takes a list of texts and an optional list of metadata dictionaries as input. The metadata can include information such as the User ID.

Once the documents are added to the vector store with their metadata, you can perform a vector search using the similarity_search, similarity_search_by_text, similarity_search_by_vector, max_marginal_relevance_search, max_marginal_relevance_search_by_vector, and similarity_search_with_score methods of the Weaviate class. These methods return a list of Document objects most similar to the query. The search results can then be filtered based on the metadata (such as User ID) to get the documents associated with a particular User ID.

Here is an example of how to use this parameter:

# Use a filter to only retrieve documents from a specific user
weaviate.similarity_search(
    query='example query',
    where_filter={'user_id':'1234'}
)

Please note that the actual implementation of the filtering based on metadata is not provided in the given context. You would need to implement this functionality yourself.

Sources

• Filtering retrieval with ConversationalRetrievalChain
• libs/langchain/langchain/vectorstores/base.py
• libs/langchain/langchain/vectorstores/weaviate.py
• libs/langchain/langchain/vectorstores/typesense.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[mohit19906]

I want to add customized metadata while trying to load the documents in the vector DB. How can I do it.
Code to read a PDF file using PDF Plumber
with pdfplumber.open(file_name) as pdf: pages = pdf.pages for page in pages: text += page.extract_text() documents.append(text)

Code to Split the documents and load in VectorDB

text_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=0) texts = text_splitter.create_documents(documents) # select which embeddings we want to use embeddings = OpenAIEmbeddings( model="text-embedding-ada-002", deployment="deployment-DummyID", chunk_size=1 ) print('texts - ', texts) db = FAISS.from_documents(texts, embeddings)

Please let me know that where in this process I can add customized meta data and how ?

[dosubot[bot]]

🤖

Hello,

To add customized metadata to your PDF files when loading them into the VectorDB using LangChain, you can use the add_documents method of the VectorStore class. This method accepts a list of Document objects, each of which contains a metadata attribute. You can include your User ID or any other metadata in this attribute.

Here is an example of how you can modify your code to include metadata:

from langchain.docstore.document import Document
from langchain.vectorstores.faiss import FAISS

# Initialize your VectorStore
db = FAISS()

# Create your documents with metadata
documents = [
    Document(page_content=text, metadata={"user_id": "user1"}),
    # Add more documents as needed
]

# Add documents to the vectorstore
db.add_documents(documents)

In the above code, replace "user1" with the actual User ID you want to assign to the PDF file. You can create a loop to assign different User IDs to different PDF files.

To perform a vector search on a specific set of PDF files associated with a particular User ID, you can use the as_retriever method of the VectorStore class. This method returns a VectorStoreRetriever object that can be used to perform searches with various parameters, including a filter for document metadata. Here is an example:

# Use a filter to only retrieve documents from a specific user
retriever = db.as_retriever(
    search_kwargs={'filter': {'user_id':'user1'}}
)

This will return documents that have a user_id metadata attribute equal to 'user1'. Please note that the actual behavior of the as_retriever method and the VectorStoreRetriever class may depend on the specific subclass of VectorStore you are using.

You can find more information about these methods in the LangChain repository.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/vectorstores/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[escano0]

could you give out more examples on how to use 'filter' in the variable 'search_kwargs' of 'as_retriever' function including loading two user_id in one retriever.

[nikitacorp]

@mohit19906 Hello, have you found the answer to your question? Because I am also creating some kind of content generator and also don't need all the data to be passed into the retriever. I would really appreciate you response!