Skip to content

Embeds text files into vectors, stores them on Pinecone, and enables semantic search using GPT3 and Langchain in a Next.js UI

Notifications You must be signed in to change notification settings

geisera/ai-handbook-deprecated

 
 

Repository files navigation

Langchain, Pinecone, and GPT with Next.js - Full Stack Starter

This is a basic starter project for building with the following tools and APIs:

  • Next.js
  • LangchainJS
  • Pineceone Vector Database
  • GPT3
  • x

What we're building

This is an app that takes text (text files), embeds them into vectors, stores them into a vector database (Pinecone), and allows for semantic searching of the data.

Running the app

How to deploy and run this app:

Prerequisites

This app requires the following:

  1. An OpenAI API key
  2. Pinecone API Key

Up and running

To run the app locally, follow these steps:

  1. Clone this repo
git clone git@github.com:geisera/ai-handbook.git
  1. CD into the directory and install the dependencies using either NPM or Yarn
npm install
  1. Copy .example.env.local to a new file called .env.local and update with your API keys and environment.

    Be sure your environment is an actual environment given to you by Pinecone, like gcp-starter

  2. (Optional) - Add your own custom text or markdown or text files into the /documents folder. Currently, this app will search our employee handbook.

  3. Run the app:

npm run dev

Need to know

When creating the embeddings and the index, it can several minutes for the index to initialize. There is a settimeout function of 180 seconds in the utils that waits for the index to be created.

If the initialization takes longer, it will fail when you try to create the embeddings. If this happens, visit the Pinecone console to watch and wait for the status of your index being created to finish, then run the function again.

Running a query

The pre-configured app data is about the M&S Salaried Employee Handbook, so it will only understand related questions to that document unless you replace it with your own data. Here are a couple of questions you might ask it with the default data

  1. What is M&S?
  2. When was M&S founded?
  3. Are alligators allowed at work?

This project was forked from this repository.

The base of that project was guided by this Node.js tutorial, with some restructuring and ported over to Next.js. You can also follow them here on Twitter!

Getting your data

Check out GPT Repository Loader which makes it simple to turn any GitHub repo into a text format, preserving the structure of the files and file contents, making it easy to chop up and save into pinecone using my codebase.

About

Embeds text files into vectors, stores them on Pinecone, and enables semantic search using GPT3 and Langchain in a Next.js UI

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • TypeScript 88.9%
  • JavaScript 6.9%
  • CSS 4.2%