Integrating OpenAI Embeddings with Pinecone Database

[argen666]

Hello,

I have added a feature to big-agi that integrates OpenAI embeddings with Pinecone, a vector database service. This feature leverages the power of semantic search by making it possible to store and retrieve OpenAI embeddings efficiently. This will improve the AI's ability to understand and generate responses based on the semantic meaning of the input.

Details:

Added Settings panel for Embeddings and Pinecone configuration
Integrated this service with the existing AI functionality of the app. Embeddings mode can be activated by double click on the Chat button
The new feature adheres to the current tech stack of the project, i.e., Next.js and React.

Please review my changes. I would appreciate any feedback.

Thank you!

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
big-agi	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 31, 2023 9:03pm

[argen666]

Moving forward, I propose the following enhancements:

Implement a feature to upload documents to the vector database through the existing UI.
This will make it easier for users to contribute their documents to the database and improve the AI's understanding and response generation.

Add support for other vector databases recommended by OpenAI.
This will make the application more flexible and capable of handling different types of vector databases, thus expanding its usability and effectiveness.
Recommended databases

[rossman22590]

@argen666 does this work right now?

[argen666]

@argen666 does this work right now?

yes, just set up your Pinecone API key and try

[rossman22590]

Going to try !

[peperunas]

Thank you @argen666 for the PR! As a newbie, I have a few questions though:

• Assuming the index has to be created beforehand on Pinecone, what are the dimensions needed for the index?
• Does the index need to be populated beforehand with data? If not, what is the difference between a vanilla LLM interaction and one with a vector database?

Also, I receive this error after setting up my API key and using the feature:

What am I doing wrong?

Thank you again!

[argen666]

Thank you @argen666 for the PR! As a newbie, I have a few questions though:

• Assuming the index has to be created beforehand on Pinecone, what are the dimensions needed for the index?
• Does the index need to be populated beforehand with data? If not, what is the difference between a vanilla LLM interaction and one with a vector database?

Also, I receive this error after setting up my API key and using the feature:

What am I doing wrong?

Thank you again!
Hi @peperunas , yes you need to create and populate the index first. Index dimension 1536. Thanks

[CyberRide]

is there a way this can be implemented with mogodb?

[argen666]

is there a way this can be implemented with mogodb?

@CyberRide yep, I have a plan to implement it in the next PR :)

[psociety]

Wouldn't it be better to use something like LlamaIndex (https://github.com/jerryjliu/llama_index) which is open source & selfthosted rather than pinecone? It also has LangChain integration.

[argen666]

Hi @psociety, I have a plan to implement all supported index storages in the next PR :)

[enricoros]

Hi @psociety, I have a plan to implement all supported index storages in the next PR :)

Hi @argen666 - happy to chat about this plan, you can hop on discord and we can chat about it. I'm curious on whether to do it server/client side, how to represent the new behavior in the UX too (I can help).