[ERROR] PineconeClient: Error calling upsert

[salman0ansari]

error [Error: PineconeClient: Error calling upsert: 
Error: PineconeClient: Error calling upsertRaw: FetchError: The request failed and the interceptors did not return an alternative response]

/home/salman/Desktop/gpt4-pdf-chatbot-langchain/scripts/ingest-data.ts:44
    throw new Error('Failed to ingest your data');
          ^


[Error: Failed to ingest your data]

Node.js v19.3.0

PDF Size: 3.0M
Pages: 53

[janek]

I had the same error. I double-checked PINECONE_API_KEY, PINECONE_ENVIRONMENT and PINECONE_INDEX_NAME. I read up on PINECONE_NAME_SPACE and confirmed that you don't have to change it in code (Pinecone will create it automatically if it doesn't exist). None of that helped.

I then went back to the readme and followed the troubleshooting section's advice to start fresh. I cloned the repo again, started a new Pinecone account, and it worked. So I haven't found out what the issue was; but I'd recommend doing that.

Between this and #93, it seems like the error messages from Pinecone can be a bit uninformative

[mindgitrwx]

Same here..

[naticio]

I just ran it 6x times and still not working...!

[saccohuo]

same issue

[saccohuo]

There are something omitted on Pinecone settings in the README.
Create an index at the INDEXES part on Pinecone. It works.
@mayooear It will be grateful if the tips can be added to README, thanks.

[MoshiMaster]

Hi what is the solution @saccohuo ? Sadly I dont understand what you mean

[saccohuo]

Hi what is the solution @saccohuo ? Sadly I dont understand what you mean

Go to Pinecone--> INDEXES --> Create index

Sorry, I misspelled INDEXES to INDEXED in the above reply.

[justinkater]

I had the same issue. My pinecone index had the wrong dimension. After I created a new one with 1536 Dimensions (because thats what the OpenAI Embeddings Endpoint uses) it worked.

[salmanarshad321]

@justinkater This worked. This is a crucial step (for people not very familiar) that they did not mention.

[smiley717]

Yep, I also created another index with 1536 Dimensions, but still not working. the same issue happened.

[smiley717]

Using a short index name instead of the long one will solve this issue, it works well.
PINECONE_INDEX_NAME=gpt4-pdf-chatbot-langchain

[prophesy604]

thanks smiley!

Using a short index name instead of the long one will solve this issue, it works well. PINECONE_INDEX_NAME=gpt4-pdf-chatbot-langchain

[sagar-jani]

I have the smallest index name but still the same issue !

[smiley717]

@sagar-jani , please make sure you're using the latest node version (I'm using 18). Is there the same error message?

[theszadow]

I'm having the same issue. I've tried the ingestion in Windows and Mac OS. Same error. I'm also, on the latest node version. I've also tried other basic text-based pdfs. No difference.

[sanzgadea]

Creating a new Pinecone index with a short name and 1536 dimensions solved the error for me.

[durgendra]

I resolved it by just using only the actual index name (and not the full index address) for the PINECONE_INDEX_NAME.
I was making mistake by using the whole address (new-XXXXX.svc.XX-XXX-gcp.pinecone.io) as value for PINECONE_INDEX_NAME

[Heilemann]

I'm having this issue, but between two PDFs I'm trying to run through one works and the other will throw this error. Everything else is the same.

[Kureev]

Had the same issue. In my case, I've been creating index programmatically and even though the promise has been resolved, the index was not "ready" yet. I recommend creating a polling mechanism to check on describeIndex api for the ready state:

const checkStatus = async () => {
    const { status } = await this.client.describeIndex({
        indexName: name,
    })
    if (status?.ready) {
        return status
    } else {
        return new Promise((resolve) => {
            setTimeout(() => resolve(checkStatus()), 5000)
        })
    }
}

await checkStatus()

and use index only after that.

[Heilemann]

I resolved my issue. I had keys the value of which were null. I removed them before sending to Pinecone and all is good.

[dukejones]

I think I read somewhere it takes around 10 minutes to initialize a new index..? Anyway, I'm creating mine manually (size of 1536) & it's working

[Kureev]

@dukejones it takes me < 20 sec to init the index, probably worth a shot trying to generate it on the flight

[bengreenspan]

I had the same error. I double-checked PINECONE_API_KEY, PINECONE_ENVIRONMENT and PINECONE_INDEX_NAME. I read up on PINECONE_NAME_SPACE and confirmed that you don't have to change it in code (Pinecone will create it automatically if it doesn't exist). None of that helped.

I then went back to the readme and followed the troubleshooting section's advice to start fresh. I cloned the repo again, started a new Pinecone account, and it worked. So I haven't found out what the issue was; but I'd recommend doing that.

Between this and #93, it seems like the error messages from Pinecone can be a bit uninformative

Were you using folders to specify namespaces?

[RobbieInOz]

Thank you using the shorter version of the index name solved it for me!

[mjrussell91]

Some combination of rebuilding the project/index, changing the size to 1536, and using a shorter name worked for me (did it all at once). I suspect the root of this issue is on Pinecone's end, and rebuilding with the same parameters may have just worked too.

And your INDEX_NAME will need to be "project" not the FQDN they provide.

[DigitalSolomon]

I'm using the Free Pinecode starter plan with one index, I'm on an M1 Mac running Node v18.16.0, I'm using the short name of my index along with all the other necessary API keys. Running the sample code from LangChain's Pinecone docs fails:

import { PineconeClient } from "@pinecone-database/pinecone";
import * as dotenv from "dotenv";
import { Document } from "langchain/document";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";
import { PineconeStore } from "langchain/vectorstores/pinecone";

dotenv.config();

const client = new PineconeClient();
await client.init({
  apiKey: process.env.PINECONE_API_KEY,
  environment: process.env.PINECONE_ENVIRONMENT,
});
const pineconeIndex = client.Index(process.env.PINECONE_INDEX);

const docs = [
  new Document({
    metadata: { foo: "bar" },
    pageContent: "pinecone is a vector db",
  }),
  new Document({
    metadata: { foo: "bar" },
    pageContent: "the quick brown fox jumped over the lazy dog",
  }),
  new Document({
    metadata: { baz: "qux" },
    pageContent: "lorem ipsum dolor sit amet",
  }),
  new Document({
    metadata: { baz: "qux" },
    pageContent: "pinecones are the woody fruiting body and of a pine tree",
  }),
];

await PineconeStore.fromDocuments(docs, new OpenAIEmbeddings(), {
  pineconeIndex,
});

Outputs following error:

> node index.js
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[PineconeError: PineconeClient: Error calling upsert: PineconeError: PineconeClient: Error calling upsertRaw: FetchError: The request failed and the interceptors did not return an alternative response]

Node.js v18.16.0

I was hoping using the the index short name would do the fix, but it's not. Any other ideas?

[tdimino]

Using a short index name instead of the long one will solve this issue, it works well. PINECONE_INDEX_NAME=gpt4-pdf-chatbot-langchain

Haha! Thank you for this tip. Should absolutely be included in the README.md.

In the past, I've used repos that automatically create an index in Pinecone for you. I assumed this would be the case again. When I followed your advice, I successfully ingested my PDFs.

[smiley717]

@tdimino Great to hear you solved the issue, lol not issue. and should be in README.md for sure :)

[italojs]

update the "@pinecone-database/pinecone" dependency so you will see what's happening, i got the error message:
[PineconeError: PineconeClient: Error calling upsert: PineconeError: Your request is larger than the maximum supported size - 2MB. Please try to reduce your batch size.]

[rikwilliam70]

I resolved it by just using only the actual index name (and not the full index address) for the PINECONE_INDEX_NAME. I was making mistake by using the whole address (new-XXXXX.svc.XX-XXX-gcp.pinecone.io) as value for PINECONE_INDEX_NAME

worked for me too.

[vol3ume4]

Solution for this issue until pinecone steps in and clarified.
I was following several of the Flowise "No Code" tutorials on YouTube. Steps are clear, repeated across tutorials. Pinecone gave errors. Simple fix for that is to use in memory vector database widget.

[Salman585]

vol3ume4

You mean an index associated with a paying plan rather than the free-tier options, right?

[jjpd777]

I had this issue because the function I was using to build the list of insertion objects was async, which returned a list of promises rather than a list of objects.

[vol3ume4]

the tutorials showed using free account. i too used exact same steps.

[pkhandelwal-cdl]

This issue appears to be a transient error occurring in Pinecone's infrastructure. Without changing anything (index name, batch size, number of vectors, etc) - I have seen an upload succeed and then fail and then succeed. The only solution is to add a retry if this error is encountered.

Unfortunately LangChain doesn't provide any hook to handle this error, so when uploading a large quantity of documents, a single transient failure can terminate the whole process and since LangChain batch uploads vectors 50 at a time to Pinecone, this error can occur randomly in the middle of a given document.

[vol3ume4]

Thanks 👍 I had suspected that too after a demo suddenly stopped working after first time.

…

On Fri, Jul 14, 2023, 12:58 AM pkhandelwal-cdl ***@***.***> wrote: This issue appears to be a transient error occurring in Pinecone's infrastructure. Without changing anything (index name, batch size, number of vectors, etc) - I have seen an upload succeed and then fail and then succeed. The only solution is to add a retry if this error is encountered. Unfortunately LangChain doesn't provide any hook to handle this error, so when uploading a large quantity of documents, a single transient failure can terminate the whole process and since LangChain batch uploads vectors 50 at a time to Pinecone <https://github.com/hwchase17/langchainjs/blob/main/langchain/src/vectorstores/pinecone.ts#L111>, this error can occur randomly in the middle of a given document. — Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT5HWI7JEGCKVQKCU34FSRTXQBD5LANCNFSM6AAAAAAWOPJRR4> . You are receiving this because you commented.Message ID: ***@***.***>

[iPanchalShubham]

I tried everything. Appended api key with NEXT_PUBLIC. deleting index, reinitializing the whole process with a new new email. lastly after tiring myself up, I ran my code twice and it worked. Its seems really like a gamble, working with something like pinecone.

[vol3ume4]

😁 my guess, free tier is sharing resources in round robin fashion

…

On Sun, Jul 23, 2023 at 7:34 AM Shubham Panchal ***@***.***> wrote: I tried everything. Appended api key with NEXT_PUBLIC. deleting index, reinitializing the whole process with a new new email. lastly after tiring myself up, I ran my code twice and it worked. Its seems really like a gamble, working with something like pinecone. — Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT5HWI4ADQ63RMPH3I4JNRTXRSBCNANCNFSM6AAAAAAWOPJRR4> . You are receiving this because you commented.Message ID: ***@***.***>

[Nciso]

``hello! I was having the same problem until I removed the "namespace" attribute in the UpserRequest, apparently, this feature is not for the free tier

 const upsertRequest = {
        vectors: vectors.map((values, idx) => ({
          id: `${idx}`,
          metadata: {
            ...documents[idx].metadata,
            text: documents[idx].pageContent,
          },
          values,
        })),
        //namespace: "test", // this was the problem
      };

[iPanchalShubham]

I finally Found it after months of search and struggle, it's actually a problem related to our local DNS configuration. Just change your local WIFI network DNS Configuration to Google's DNS servers:
Preferred DNS server: 8.8.8.8
Alternate DNS server: 8.8.4.4

Man it was the toughest and the easiest bug of my life, because i was thinking that there is something wrong with my code and but it wasn't.
let me know if anyone needs more help with this.
PS: please comment before downgrading my answer in frustration.

[AliHewiagh]

Many might not be aware of a crucial point: In each free organization, only one index can be created. In my situation, I needed to have two indexes for both the staging and production environments. To solve this, I created two separate organizations. While I succeeded in setting up a free index in each of them, Pinecone's system is capable of detecting this. As a result, both indexes will not work unless you either delete one of them or opt for an upgrade.