notebooks/vector_search_examples.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "355eef77",
   "metadata": {},
   "source": [
    "# Build / Use Vector Store\n",
    "Below is a simple example of first building an in memory vector store from a Notion markdown text directory, and then using the store to query for search results."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e33a7b11",
   "metadata": {},
   "source": [
    "## Env Setup\n",
    "Before we get started, install the gems we need here:\n",
    "`gem install boxcars dotenv`\n",
    "and then create / edit .env to have OPENAI_ACCESS_TOKEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bc5f28ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "require 'dotenv/load'\n",
    "require 'boxcars'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f924770a",
   "metadata": {},
   "source": [
    "## InMemory \n",
    "\n",
    "### With local markdown files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c8d81c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "root = \"./embeddings\"\n",
    "store = Boxcars::VectorStore::InMemory::BuildFromFiles.call(\n",
    "  training_data_path: \"#{root}/Notion_DB/**/*.md\",\n",
    "  split_chunk_size: 900,\n",
    "  embedding_tool: :openai\n",
    ")\n",
    "\n",
    "in_memory_search = Boxcars::VectorSearch.new(type: :in_memory, vector_documents: store)\n",
    "\n",
    "in_memory_search.call(query_vector: store[:vector_store][0][:embedding], count: 1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "aadab6b6",
   "metadata": {},
   "source": [
    "### With hash array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75d476eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_array = \n",
    "  [\n",
    "    { content: \"hello\", metadata: { a: 1 } },\n",
    "    { content: \"hi\", metadata: { a: 1 } },\n",
    "    { content: \"bye\", metadata: { a: 1 } },\n",
    "    { content: \"what's this\", metadata: { a: 1 } }\n",
    "  ]\n",
    "  \n",
    "store = Boxcars::VectorStore::InMemory::BuildFromArray.call(\n",
    "  embedding_tool: :openai,\n",
    "  input_array: input_array\n",
    ")\n",
    "\n",
    "in_memory_search = Boxcars::VectorSearch.new(type: :in_memory, vector_documents: store)\n",
    "\n",
    "search_result= in_memory_search.call(query: \"hello\", count: 1)\n",
    "\n",
    "puts search_result.first[:document].content\n",
    "# => \"hello\"\n",
    "puts search_result.first[:document].metadata\n",
    "# => {:a=>1, :dim=>1536} "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2f5e64bc",
   "metadata": {},
   "source": [
    "## Pgvector\n",
    "\n",
    "follow official [pgvector](https://github.com/pgvector/pgvector) extension for installing and setup.\n",
    "\n",
    "```sh\n",
    "cd /tmp\n",
    "git clone --branch v0.4.1 https://github.com/pgvector/pgvector.git\n",
    "cd pgvector\n",
    "make\n",
    "sudo make install\n",
    "```\n",
    "\n",
    "and then create a database and run the following sql to create the table and index.\n",
    "\n",
    "```ruby \n",
    "createdb boxcars_development\n",
    "\n",
    "conn ||= PG::Connection.new(\"postgres://postgres@localhost/boxcars_development\")\n",
    "conn.exec(\"CREATE EXTENSION IF NOT EXISTS vector\")\n",
    "\n",
    "# noteice that the number of dimensions is 1536\n",
    "# the query vector we use with openai should also be 1536\n",
    "create_table_query = <<-SQL\n",
    "      CREATE TABLE IF NOT EXISTS items (\n",
    "        id bigserial PRIMARY KEY,\n",
    "        content text,\n",
    "        embedding vector(1536),\n",
    "        metadata jsonb\n",
    "      );\n",
    "    SQL\n",
    "conn.exec(create_table_query)\n",
    "```\n",
    "\n",
    "\n",
    "### With local markdown files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f19f0f39",
   "metadata": {},
   "outputs": [],
   "source": [
    "# path is relative to the code that you are running\n",
    "training_data_path = \"./embeddings/Notion_DB/**/*.md\"\n",
    "db_url = \"postgres://postgres@localhost/boxcars_development\"\n",
    "table_name = \"items\"\n",
    "embedding_column_name = \"embedding\"\n",
    "content_column_name = \"content\"\n",
    "metadata_column_name = \"metadata\"\n",
    "\n",
    "Boxcars::VectorStore::Pgvector::BuildFromFiles.call(\n",
    "  training_data_path: training_data_path,\n",
    "  split_chunk_size: 900,\n",
    "  embedding_tool: :openai,\n",
    "  database_url: db_url,\n",
    "  table_name: table_name,\n",
    "  embedding_column_name: embedding_column_name,\n",
    "  content_column_name: content_column_name,\n",
    "  metadata_column_name: metadata_column_name\n",
    ")\n",
    "\n",
    "openai_client = Boxcars::Openai.open_ai_client(openai_access_token: ENV['OPENAI_API_KEY'])\n",
    "\n",
    "vector_documents = {\n",
    "  type: :pgvector,\n",
    "  vector_store: {\n",
    "    database_url: db_url,\n",
    "    table_name: table_name,\n",
    "    embedding_column_name: embedding_column_name,\n",
    "    content_column_name: content_column_name\n",
    "  }\n",
    "}\n",
    "\n",
    "search = Boxcars::VectorSearch.new(openai_connection: openai_client, vector_documents: vector_documents)\n",
    "\n",
    "first_query = search.call(query: \"How many holidays would I get?\", count: 1)\n",
    "puts first_query.first[:document].content\n",
    "# => \"there is money in our bank account and it’s up to us to spend it wisely. So yes, there is a budget. Just let us know if you think it’s reasonable for Blendle to pitch in.\\n- **Blendle outings:** \\nthe party agenda is pretty full. You’ll be invited when there is a party ahead.\\n- **Flexible Holidays:** \\nwe think 4-6 weeks off per year is kinda the sweet spot, with at least once 2 weeks in a row. So that’s what we put in your contract.\\n- **Flexible hours**: \\nwe want you to find out what works for you best. Just know that we don't keep track of hours. We trust you.\\n- **Laptop**: \\nwe provide you with a laptop that suits your job. Ask HR for further info.\\n- **Workplace**: \\nwe've built a pretty nice office to make sure you like being at Blendle HQ. Feel free to sit where you want. Even better: dare to switch your workplace every once in a while.\\n\\n# Work at Blendle\\n\\n---\" \n",
    "\n",
    "second_query = search.call(query: \"What should I do if someone is bullying me?\", count: 1)\n",
    "\n",
    "puts second_query.first[:document].content\n",
    "# => \"- **Talk to the offender**. If you suspect that an offender doesn’t realise they are guilty of harassment, you could talk to them directly in an effort to resolve the issue. This tactic is appropriate for cases of minor harassment (e.g. inappropriate jokes between colleagues, something you read on #overheard).\\n- **Talk to your team lead**. Your team lead will assess your situation and may contact HR if appropriate. Explain the situation in as much detail as possible. If you have any hard evidence (e.g. emails), forward it or bring it with you to the meeting.\\n- **Talk to HR**. Feel free to reach out to HR in any case of harassment no matter how minor it may seem. For your safety, contact HR as soon as possible in cases of serious harassment (e.g. sexual advances) or if your team lead is involved in your claim. Anything you disclose will remain confidential.\" "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "524725ef",
   "metadata": {},
   "source": [
    "### With hash array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46e90b8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "input_array = [\n",
    "  { content: \"hello\", metadata: { a: 1 } },\n",
    "  { content: \"hi\", metadata: { a: 1 } },\n",
    "  { content: \"bye\", metadata: { a: 1 } },\n",
    "  { content: \"what's this\", metadata: { a: 1 } }\n",
    "]\n",
    "\n",
    "Boxcars::VectorStore::Pgvector::BuildFromArray.call(\n",
    "  embedding_tool: :openai,\n",
    "  input_array: input_array,\n",
    "  database_url: db_url,\n",
    "  table_name: table_name,\n",
    "  embedding_column_name: embedding_column_name,\n",
    "  content_column_name: content_column_name,\n",
    "  metadata_column_name: metadata_column_name\n",
    ")\n",
    "\n",
    "vector_documents = {\n",
    "  type: :pgvector,\n",
    "  vector_store: {\n",
    "    database_url: db_url,\n",
    "    table_name: table_name,\n",
    "    embedding_column_name: embedding_column_name,\n",
    "    content_column_name: content_column_name\n",
    "  }\n",
    "}\n",
    "search = Boxcars::VectorSearch.new(openai_connection: openai_client, vector_documents: vector_documents)\n",
    "\n",
    "query = search.call(query: \"bye\", count: 1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dde3e96d",
   "metadata": {},
   "source": [
    "## Hnswlib \n",
    "\n",
    "### Build the Vector Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "562fc854",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building Hnswlib vector store...\n",
      "Added 50 files to data. Splitting text into chunks...\n",
      "Loaded 50 files from /Users/francis/src/notebooks/boxcars/embeddings/Notion_DB/**/*.md\n",
      "Split 50 files into 140 chunks\n",
      "Generated 140 vectors\n",
      "Added 140 vectors to vector store\n",
      "built\n"
     ]
    }
   ],
   "source": [
    "root = \"./embeddings\"\n",
    "store = Boxcars::VectorStore::Hnswlib::BuildFromFiles.call(\n",
    "  training_data_path: \"#{root}/Notion_DB/**/*.md\",\n",
    "  index_file_path: \"#{root}/hnswlib_notion_db_index.bin\",\n",
    "  force_rebuild: true\n",
    ")\n",
    "puts :built"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1e9fadf8",
   "metadata": {},
   "source": [
    "### Query the Store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "342ebacb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"we provide you with a laptop that suits your job. Ask HR for further info.\\n- **Workplace**: \\nwe've built a pretty nice office to make sure you like being at Blendle HQ. Feel free to sit where you want. Even better: dare to switch your workplace every once in a while.\\n\\n# Work at Blendle\\n\\n---\\n\\nIf you want to work at Blendle you can check our [job ads here](https://blendle.homerun.co/). If you want to be kept in the loop about Blendle, you can sign up for [our behind the scenes newsletter](https://blendle.homerun.co/yes-keep-me-posted/tr/apply?token=8092d4128c306003d97dd3821bad06f2).\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "openai_client = Boxcars::Openai.open_ai_client(openai_access_token: ENV['OPENAI_API_KEY'])\n",
    "\n",
    "similarity_search = Boxcars::VectorSearch.new(\n",
    "  openai_connection: openai_client,\n",
    "  vector_documents: store)\n",
    "ss = similarity_search.call query: \"Do I get a laptop?\", count: 1\n",
    "ss.first[:document].content"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "32d453f2",
   "metadata": {},
   "source": [
    "## Answer a Question from Search Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "28ba2e7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;30m> Entering VectorAnswer#run\u001b[0m\n",
      "\u001b[0;34mDo I get a laptop?\u001b[0m\n",
      "\u001b[0;35m{\"status\":\"ok\",\"answer\":\"Yes, you will be provided with a laptop that suits your job. You can ask HR for further information.\",\"explanation\":\"Answer: Yes, you will be provided with a laptop that suits your job. You can ask HR for further information.\"}\u001b[0m\n",
      "\u001b[1;30m< Exiting VectorAnswer#run\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Yes, you will be provided with a laptop that suits your job. You can ask HR for further information.\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "va = Boxcars::VectorAnswer.new(embeddings: \"#{root}/hnswlib_notion_db_index.json\", vector_documents: store)\n",
    "va.conduct(\"Do I get a laptop?\").to_answer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c415befc",
   "metadata": {},
   "source": [
    "## More\n",
    "You could of course use text files and get similar results. Other libraries can be brought it to handle PDFs and other binary formats. Add Issues and/or PRs for other types that you want supported."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Ruby 3.2.2",
   "language": "ruby",
   "name": "ruby"
  },
  "language_info": {
   "file_extension": ".rb",
   "mimetype": "application/x-ruby",
   "name": "ruby",
   "version": "3.2.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}