Repository navigation
Able to query contents of a PG vector table, ever after dropping the table. #9152
Description
System Info
Hello,
Thanks for the wonderful extension on PG. I am using on DigitialOcean for my SAAS app where i have different schemas for each tenant. I had installed PG vector in public schema, so that all the tenants could make use of them.
My app is built on OpenAI api and I am using the default method from langchain to define the db
from langchain.vectorstores.pgvector import DistanceStrategy
db = PGVector.from_documents(
documents= docs,
embedding = embeddings,
collection_name= "schema.unique_value",
distance_strategy = DistanceStrategy.COSINE,
connection_string=CONNECTION_STRING)
This create two files in the public scema and i could see and query the contents. below are the tables
langchain_pg_colllection
langchain_pg_embedding
But when i wish to rewrite the contents by deleteing and even dropping the tables, i could still query the content using the below code:
db = PGVector(
collection_name=collection_name,
connection_string=CONNECTION_STRING,
embedding_function=embeddings,
)
retriever = db.as_retriever(
search_kwargs={"k": 5}
)
The deleted table gets automatically restored while query. This feature is strange and not sure what creates it. Also when i pass a non existent value, instead of failing, it creates a new record in the collection table.
I raise this issue with PG_VECTOR but was rightly directed to reach out to you.
Would seek any inputs to resolve this plase.
Who can help?
Information
- The official example notebooks/scripts
- My own modified scripts
Related Components
- LLMs/Chat Models
- Embedding Models
- Prompts / Prompt Templates / Prompt Selectors
- Output Parsers
- Document Loaders
- Vector Stores / Retrievers
- Memory
- Agents / Agent Executors
- Tools / Toolkits
- Chains
- Callbacks/Tracing
- Async
Reproduction
db = PGVector(
collection_name=collection_name,
connection_string=CONNECTION_STRING,
embedding_function=embeddings,
)
retriever = db.as_retriever(
search_kwargs={"k": 5}
)
Expected behavior
The retriever should get only the contents from the table. If not in the table (or table not available) it should fail. Thanks in anticipation
Activity
dosubot commentedon Aug 12, 2023
Answer generated by a 🤖
Hello,
Thank you for reaching out and providing a detailed explanation of your issue. The behavior you're observing is due to the design of the PGVector class in LangChain.
Querying of contents even after the associated table has been dropped: The PGVector class in LangChain uses SQLAlchemy for database operations. SQLAlchemy provides a session-based "unit of work" pattern, which means that changes made within a session are not immediately committed to the database. This allows for rollback of changes if something goes wrong. When you drop a table, the changes are not immediately reflected in the session. Therefore, you can still query the contents of the table until the session is closed or committed.
Creating a new record in the collection table when a non-existent value is passed: This is due to the
get_or_createmethod used in thecreate_collectionmethod. Theget_or_createmethod checks if a record exists with the given parameters, and if it doesn't, it creates a new record. This is a common pattern in database operations to ensure that a record exists before performing operations on it.Here is the relevant code:
In this code,
self.CollectionStore.get_or_createwill either get the existing collection or create a new one if it doesn't exist.If you want to change this behavior, you would need to modify the
create_collectionmethod to not use theget_or_createmethod, and handle the case where the collection does not exist in a different way. Similarly, to prevent querying of dropped tables, you would need to ensure that the session is committed or closed after the table is dropped.I hope this helps! Let me know if you have any other questions.
Sources
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with Dosu Bot (@dosu-bot).
rkrkrediffmail commentedon Aug 12, 2023
This is awesome reply. couldnt believe its a bot reply. quick and accurate. thanks!!