[Pinecone] Controlling index ID's

[guidorietbroek]

When testing with the Pinecone functionality within Langchain, it's not quite clear to me how indexing works. If you are inserting a couple of documents which are chunked in smaller parts, how is the indexing working? Is there a standard way of giving those records an ID? How to update 1 document (which consists of an unknown number of chunks as records inside the database) if you don't know the ID's?

I'm getting the feeling that I better use the API from Pinecone itself so I can create a list of (id, vector) tuples.

Is there a good way of defining those ID's per document consisting of multiple records? In the end you want to update a whole document if an update exists, so you need to filter on all the records from the file inside the vector database.

Hopefully somebody has some nice thoughts! Thank you!

[dosubot[bot]]

Answer generated by a 🤖

Answer

I understand that you're trying to figure out how indexing works within the Pinecone functionality of LangChain, particularly when dealing with documents that are chunked into smaller parts. You're also interested in understanding how to update a document consisting of multiple records when the IDs of these records are unknown.

In LangChain, when you use the add_texts method of the Pinecone class, it takes an iterable of texts and optionally, a list of metadata, a list of IDs, a namespace, and a batch size. If you don't provide any IDs, the method will generate a unique ID for each text using uuid.uuid4(). The method then creates embeddings for each batch of texts using the provided embedding function and upserts these embeddings to the Pinecone index along with the corresponding IDs and metadata.

If you need to update a document, you'll need to know its ID. If a document is split into multiple records, each with its own ID, you'll need to maintain a record of which IDs correspond to which document. This could be achieved by storing a mapping from document names to lists of IDs in a separate data structure.

To update a whole document, you would need to delete the old records corresponding to that document and insert the new records. You can use the delete method of the Pinecone class to delete the old records by passing it the IDs of the records to delete. To insert the new records, you can use the add_texts method.

Please note that this is one potential solution based on the current functionality of LangChain. If you find this solution helpful and believe it could benefit other users, I encourage you to make a pull request to update the documentation. This way, you can contribute to making LangChain more user-friendly for everyone.

Sources

• tests/integration_tests/vectorstores/test_pinecone.py
• langchain/vectorstores/pinecone.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

[guidorietbroek]

Love this answer by a bot 😄

Although it is helpful, I'm not quite sure if the separate mapping in a data structure is the most efficient way. I would like to add some extra possibilities:

1. What about using a unique namespace per document? Or is this slowing down queries?
2. What about using an unique metadata entry for every document and filtering on this metadata?

I did some tests and when using Pinecone extension via Langchain, the ID's are indeed random. For instance, I have one like 972ad7b1-72fe-4548-876a-09e69b1801d1.

[guidorietbroek]

So after some extra research I think I have a good solution which is not having too much of a performance decrease.

I'm know adding the external document ID from whatever service we're using to the metadata of the extracted document. This metadata will be duplicated when chunking, embedding and inserting in the vector database. When I want to delete 1 full document, I can just use the filter on metadata. Very powerful and quick.

docsearch.delete(filter={'external_id':'2'})