docs/docs/integrations/vectorstores/pgvector.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7679dd7b-7ed4-4755-a499-824deadba708",
   "metadata": {},
   "source": [
    "# PGVector\n",
    "\n",
    "> An implementation of LangChain vectorstore abstraction using `postgres` as the backend and utilizing the `pgvector` extension.\n",
    "\n",
    "The code lives in an integration package called: [langchain_postgres](https://github.com/langchain-ai/langchain-postgres/).\n",
    "\n",
    "You can run the following command to spin up a a postgres container with the `pgvector` extension:\n",
    "\n",
    "```shell\n",
    "docker run --name pgvector-container -e POSTGRES_USER=langchain -e POSTGRES_PASSWORD=langchain -e POSTGRES_DB=langchain -p 6024:5432 -d pgvector/pgvector:pg16\n",
    "```\n",
    "\n",
    "## Status\n",
    "\n",
    "This code has been ported over from `langchain_community` into a dedicated package called `langchain-postgres`. The following changes have been made:\n",
    "\n",
    "* langchain_postgres works only with psycopg3. Please update your connnecion strings from `postgresql+psycopg2://...` to `postgresql+psycopg://langchain:langchain@...` (yes, it's the driver name is `psycopg` not `psycopg3`, but it'll use `psycopg3`.\n",
    "* The schema of the embedding store and collection have been changed to make add_documents work correctly with user specified ids.\n",
    "* One has to pass an explicit connection object now.\n",
    "\n",
    "\n",
    "Currently, there is **no mechanism** that supports easy data migration on schema changes. So any schema changes in the vectorstore will require the user to recreate the tables and re-add the documents.\n",
    "If this is a concern, please use a different vectorstore. If not, this implementation should be fine for your use case."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "342cd5e9-f349-42b4-9713-12e63779835b",
   "metadata": {},
   "source": [
    "## Install dependencies\n",
    "\n",
    "Here, we're using `langchain_cohere` for embeddings, but you can use other embeddings providers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "42d42297-11b8-44e3-bf21-7c3d1bce8277",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --quiet -U langchain_cohere\n",
    "!pip install --quiet -U langchain_postgres"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eee31ce1-2c28-484d-82be-d22d9f9a31fd",
   "metadata": {},
   "source": [
    "## Initialize the vectorstore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "979a65bd-742f-4b0d-be1e-c0baae245ec6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_cohere import CohereEmbeddings\n",
    "from langchain_core.documents import Document\n",
    "from langchain_postgres import PGVector\n",
    "from langchain_postgres.vectorstores import PGVector\n",
    "\n",
    "# See docker command above to launch a postgres instance with pgvector enabled.\n",
    "connection = \"postgresql+psycopg://langchain:langchain@localhost:6024/langchain\"  # Uses psycopg3!\n",
    "collection_name = \"my_docs\"\n",
    "embeddings = CohereEmbeddings()\n",
    "\n",
    "vectorstore = PGVector(\n",
    "    embeddings=embeddings,\n",
    "    collection_name=collection_name,\n",
    "    connection=connection,\n",
    "    use_jsonb=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fc32168-5a82-4629-a78d-158fe2615086",
   "metadata": {},
   "source": [
    "## Drop tables\n",
    "\n",
    "If you need to drop tables (e.g., updating the embedding to a different dimension or just updating the embedding provider): "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de5ef98-7dbb-4892-853f-47c7dc87b70e",
   "metadata": {
    "tags": []
   },
   "source": [
    "```python\n",
    "vectorstore.drop_tables()\n",
    "````"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a224a1-d70b-4daf-86ba-ab6e43c08b50",
   "metadata": {},
   "source": [
    "## Add documents\n",
    "\n",
    "Add documents to the vectorstore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "88a288cc-ffd4-4800-b011-750c72b9fd10",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs = [\n",
    "    Document(\n",
    "        page_content=\"there are cats in the pond\",\n",
    "        metadata={\"id\": 1, \"location\": \"pond\", \"topic\": \"animals\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"ducks are also found in the pond\",\n",
    "        metadata={\"id\": 2, \"location\": \"pond\", \"topic\": \"animals\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"fresh apples are available at the market\",\n",
    "        metadata={\"id\": 3, \"location\": \"market\", \"topic\": \"food\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the market also sells fresh oranges\",\n",
    "        metadata={\"id\": 4, \"location\": \"market\", \"topic\": \"food\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the new art exhibit is fascinating\",\n",
    "        metadata={\"id\": 5, \"location\": \"museum\", \"topic\": \"art\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"a sculpture exhibit is also at the museum\",\n",
    "        metadata={\"id\": 6, \"location\": \"museum\", \"topic\": \"art\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"a new coffee shop opened on Main Street\",\n",
    "        metadata={\"id\": 7, \"location\": \"Main Street\", \"topic\": \"food\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the book club meets at the library\",\n",
    "        metadata={\"id\": 8, \"location\": \"library\", \"topic\": \"reading\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the library hosts a weekly story time for kids\",\n",
    "        metadata={\"id\": 9, \"location\": \"library\", \"topic\": \"reading\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"a cooking class for beginners is offered at the community center\",\n",
    "        metadata={\"id\": 10, \"location\": \"community center\", \"topic\": \"classes\"},\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "73aa9124-9d49-4e10-8ed3-82255e7a4106",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorstore.add_documents(docs, ids=[doc.metadata[\"id\"] for doc in docs])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a5b2b71f-49eb-407d-b03a-dea4c0a517d6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='there are cats in the pond', metadata={'id': 1, 'topic': 'animals', 'location': 'pond'}),\n",
       " Document(page_content='the book club meets at the library', metadata={'id': 8, 'topic': 'reading', 'location': 'library'}),\n",
       " Document(page_content='the library hosts a weekly story time for kids', metadata={'id': 9, 'topic': 'reading', 'location': 'library'}),\n",
       " Document(page_content='the new art exhibit is fascinating', metadata={'id': 5, 'topic': 'art', 'location': 'museum'}),\n",
       " Document(page_content='ducks are also found in the pond', metadata={'id': 2, 'topic': 'animals', 'location': 'pond'}),\n",
       " Document(page_content='the market also sells fresh oranges', metadata={'id': 4, 'topic': 'food', 'location': 'market'}),\n",
       " Document(page_content='a cooking class for beginners is offered at the community center', metadata={'id': 10, 'topic': 'classes', 'location': 'community center'}),\n",
       " Document(page_content='fresh apples are available at the market', metadata={'id': 3, 'topic': 'food', 'location': 'market'}),\n",
       " Document(page_content='a sculpture exhibit is also at the museum', metadata={'id': 6, 'topic': 'art', 'location': 'museum'}),\n",
       " Document(page_content='a new coffee shop opened on Main Street', metadata={'id': 7, 'topic': 'food', 'location': 'Main Street'})]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorstore.similarity_search(\"kitty\", k=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d87a413-015a-4b46-a64e-332f30806524",
   "metadata": {},
   "source": [
    "Adding documents by ID will over-write any existing documents that match that ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "13c69357-aaee-4de0-bcc2-7ab4419c920e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs = [\n",
    "    Document(\n",
    "        page_content=\"there are cats in the pond\",\n",
    "        metadata={\"id\": 1, \"location\": \"pond\", \"topic\": \"animals\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"ducks are also found in the pond\",\n",
    "        metadata={\"id\": 2, \"location\": \"pond\", \"topic\": \"animals\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"fresh apples are available at the market\",\n",
    "        metadata={\"id\": 3, \"location\": \"market\", \"topic\": \"food\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the market also sells fresh oranges\",\n",
    "        metadata={\"id\": 4, \"location\": \"market\", \"topic\": \"food\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the new art exhibit is fascinating\",\n",
    "        metadata={\"id\": 5, \"location\": \"museum\", \"topic\": \"art\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"a sculpture exhibit is also at the museum\",\n",
    "        metadata={\"id\": 6, \"location\": \"museum\", \"topic\": \"art\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"a new coffee shop opened on Main Street\",\n",
    "        metadata={\"id\": 7, \"location\": \"Main Street\", \"topic\": \"food\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the book club meets at the library\",\n",
    "        metadata={\"id\": 8, \"location\": \"library\", \"topic\": \"reading\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"the library hosts a weekly story time for kids\",\n",
    "        metadata={\"id\": 9, \"location\": \"library\", \"topic\": \"reading\"},\n",
    "    ),\n",
    "    Document(\n",
    "        page_content=\"a cooking class for beginners is offered at the community center\",\n",
    "        metadata={\"id\": 10, \"location\": \"community center\", \"topic\": \"classes\"},\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59f82250-7903-4279-8300-062542c83416",
   "metadata": {},
   "source": [
    "## Filtering Support\n",
    "\n",
    "The vectorstore supports a set of filters that can be applied against the metadata fields of the documents.\n",
    "\n",
    "| Operator | Meaning/Category        |\n",
    "|----------|-------------------------|\n",
    "| \\$eq      | Equality (==)           |\n",
    "| \\$ne      | Inequality (!=)         |\n",
    "| \\$lt      | Less than (<)           |\n",
    "| \\$lte     | Less than or equal (<=) |\n",
    "| \\$gt      | Greater than (>)        |\n",
    "| \\$gte     | Greater than or equal (>=) |\n",
    "| \\$in      | Special Cased (in)      |\n",
    "| \\$nin     | Special Cased (not in)  |\n",
    "| \\$between | Special Cased (between) |\n",
    "| \\$like    | Text (like)             |\n",
    "| \\$ilike   | Text (case-insensitive like) |\n",
    "| \\$and     | Logical (and)           |\n",
    "| \\$or      | Logical (or)            |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f15a2359-6dc3-4099-8214-785f167a9ca4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='there are cats in the pond', metadata={'id': 1, 'topic': 'animals', 'location': 'pond'}),\n",
       " Document(page_content='the library hosts a weekly story time for kids', metadata={'id': 9, 'topic': 'reading', 'location': 'library'}),\n",
       " Document(page_content='the new art exhibit is fascinating', metadata={'id': 5, 'topic': 'art', 'location': 'museum'}),\n",
       " Document(page_content='ducks are also found in the pond', metadata={'id': 2, 'topic': 'animals', 'location': 'pond'})]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorstore.similarity_search(\"kitty\", k=10, filter={\"id\": {\"$in\": [1, 5, 2, 9]}})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d92ea049-1b1f-4ae9-9525-35750fe2e52e",
   "metadata": {},
   "source": [
    "If you provide a dict with multiple fields, but no operators, the top level will be interpreted as a logical **AND** filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "88f919e4-e4b0-4b5f-99b3-24c675c26d33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='ducks are also found in the pond', metadata={'id': 2, 'topic': 'animals', 'location': 'pond'}),\n",
       " Document(page_content='there are cats in the pond', metadata={'id': 1, 'topic': 'animals', 'location': 'pond'})]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorstore.similarity_search(\n",
    "    \"ducks\",\n",
    "    k=10,\n",
    "    filter={\"id\": {\"$in\": [1, 5, 2, 9]}, \"location\": {\"$in\": [\"pond\", \"market\"]}},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "88f423a4-6575-4fb8-9be2-a3da01106591",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='ducks are also found in the pond', metadata={'id': 2, 'topic': 'animals', 'location': 'pond'}),\n",
       " Document(page_content='there are cats in the pond', metadata={'id': 1, 'topic': 'animals', 'location': 'pond'})]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorstore.similarity_search(\n",
    "    \"ducks\",\n",
    "    k=10,\n",
    "    filter={\n",
    "        \"$and\": [\n",
    "            {\"id\": {\"$in\": [1, 5, 2, 9]}},\n",
    "            {\"location\": {\"$in\": [\"pond\", \"market\"]}},\n",
    "        ]\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "65133340-2acd-4957-849e-029b6b5d60f0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='the book club meets at the library', metadata={'id': 8, 'topic': 'reading', 'location': 'library'}),\n",
       " Document(page_content='the new art exhibit is fascinating', metadata={'id': 5, 'topic': 'art', 'location': 'museum'}),\n",
       " Document(page_content='the library hosts a weekly story time for kids', metadata={'id': 9, 'topic': 'reading', 'location': 'library'}),\n",
       " Document(page_content='a sculpture exhibit is also at the museum', metadata={'id': 6, 'topic': 'art', 'location': 'museum'}),\n",
       " Document(page_content='the market also sells fresh oranges', metadata={'id': 4, 'topic': 'food', 'location': 'market'}),\n",
       " Document(page_content='a cooking class for beginners is offered at the community center', metadata={'id': 10, 'topic': 'classes', 'location': 'community center'}),\n",
       " Document(page_content='a new coffee shop opened on Main Street', metadata={'id': 7, 'topic': 'food', 'location': 'Main Street'}),\n",
       " Document(page_content='fresh apples are available at the market', metadata={'id': 3, 'topic': 'food', 'location': 'market'})]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorstore.similarity_search(\"bird\", k=10, filter={\"location\": {\"$ne\": \"pond\"}})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}