Running Ingest from Colab

[machsci]

I'm running into the following error when trying to ingest data to Pinecone running gpt4-pdf-chatbot-langchain in a Colab notebook. I have properly set the PINECONE_INDEX_NAME, and created a new PINECONE_NAME_SPACE in the pinecone.ts file. I have set the environment variables using %env (e.g., %env PINECONE_ENVIRONMENT='us-west4-gcp').

This is the result of running ! pnpm run ingest:

> gpt4-langchain-pdf-chatbot@0.1.0 ingest /content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain
> tsx -r dotenv/config scripts/ingest-data.ts

'us-west4-gcp'
PineconeClient: Error getting project name: TypeError: fetch failed
error [Error: EISDIR: illegal operation on a directory, read] {
  errno: -21,
  code: 'EISDIR',
  syscall: 'read'
}
/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/scripts/ingest-data.ts:51
    throw new Error('Failed to ingest your data');
          ^


[Error: Failed to ingest your data]

Node.js v19.8.1
 ELIFECYCLE  Command failed with exit code 1.

[mayooear]

Hey, a few issues here:

1. EISDIR error means that you are attempting to read for a directory instead of a file. Please cross check the file path is correct in ingest-data.ts
2. Langchain requires node-fetch module, it seems it's missing on your end so pnpm add node-fetch then import fetch into the pinecone.ts file

[machsci]

After making the above changes, I get a new error:

> gpt4-langchain-pdf-chatbot@0.1.0 ingest /content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain
> tsx -r dotenv/config scripts/ingest-data.ts

/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:2288
      throw reject;
      ^

Error: The service is no longer running: spawn /content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/@esbuild+linux-x64@0.17.11/node_modules/@esbuild/linux-x64/bin/esbuild EACCES
    at /content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:816:29
    at sendRequest (/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:693:14)
    at start (/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:814:9)
    at Object.transform2 [as transform] (/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:879:5)
    at /content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:2180:77
    at new Promise (<anonymous>)
    at Object.transform (/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:2180:36)
    at /content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:2334:67
    at MessagePort.<anonymous> (/content/gdrive/MyDrive/[Path]/gpt4-pdf-chatbot-langchain/node_modules/.pnpm/esbuild@0.17.11/node_modules/esbuild/lib/main.js:2350:9)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:758:20)

Node.js v19.8.1
 ELIFECYCLE  Command failed with exit code 1.

[mayooear]

I think it's because you're using google drive. So the file permissions aren't set properly. Try copying the project to your local filesystem and testing from there.