For sending data to Pinecone they have some hard limits:
- 40KB for metadata
- 2MB data for any request
Issues:
- Anything from langchain that outputs a Document has no such limit
- Average vectorsize for text-embedding-ada-002 is ~12.3kb, therefore if I have e.g. 30kb metadata I would send 42.3kb per entry. I cannot send more than 47 entries per batch, however, the standard is 1000.
I have personally written limits/calculations for: tokens, bytesize, batchsize around langchain but I'd expect these functionalities to be within langchain. This is actually how I did it; this might help you, and others:
// I have some logic that limits my metadata to 35KB, here is my calculateByteSize function
const encoder = new TextEncoder();
const calculateByteSize = (document: LangchainDocument): number => {
const encodedMetadata = encoder.encode(JSON.stringify(document));
return encodedMetadata.length;
};
// Max pinecone upload is 2MB, vectors are around 12KB, metadata maximum 40KB
// Therefore 2000KB / (12KB + 35KB) = 30 documents per upload
let promises: Promise<PineconeStore>[] = [];
const STEPSIZE = 30;
for (let i = 0; i < documents.length; i += STEPSIZE) {
if (promises.length > 0) {
await Promise.all(promises);
promises = [];
console.log(`Uploaded ${i}/${documents.length} documents to pinecone.`);
}
promises.push(
PineconeStore.fromDocuments(
documents.slice(i, i + STEPSIZE),
embeddingSettings,
{
pineconeIndex,
namespace: namespace,
maxConcurrency: 10
}
)
);
}Would love to see this improved 🦾
For sending data to Pinecone they have some hard limits:
Issues:
I have personally written limits/calculations for: tokens, bytesize, batchsize around langchain but I'd expect these functionalities to be within langchain. This is actually how I did it; this might help you, and others:
Would love to see this improved 🦾