Integration_Canopy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pinecone Canopy with Anyscale\n",
    "\n",
    "**Canopy** is a Software Development Kit (SDK) for AI applications. Canopy allows you to test, build and package Retrieval Augmented Applications with Pinecone Vector Database. \n",
    "\n",
    "This notebook introduces the quick start steps for working with Canopy and Anyscale Endpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "install canopy library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -qU canopy-sdk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, Canopy uses Pinecone and Anyscale so we need to configure the related API keys.\n",
    "\n",
    "To get Pinecone free trial API key and environment register or log into your Pinecone account in the [console](https://app.pinecone.io/). You can access your API key from the \"API Keys\" section in the sidebar of your dashboard, and find the environment name next to it.\n",
    "\n",
    "You can find your free Anyscale API key [here](https://https://app.endpoints.anyscale.com/). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"PINECONE_API_KEY\"] = PINECONE_API_KEY\n",
    "os.environ[\"PINECONE_ENVIRONMENT\"] = PINECONE_ENVIRONMENT\n",
    "os.environ[\"ANYSCALE_API_KEY\"]=ANYSCALE_API_KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each record in this dataset represents a single page in Pinecone's documentation. Each row contains a unique id, the raw text of the page in markdown language, the url of the page as \"source\" and some metadata. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pinecone Documentation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>source</th>\n",
       "      <th>metadata</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>728aeea1-1dcf-5d0a-91f2-ecccd4dd4272</td>\n",
       "      <td># Scale indexes\\n\\n[Suggest Edits](/edit/scali...</td>\n",
       "      <td>https://docs.pinecone.io/docs/scaling-indexes</td>\n",
       "      <td>{'created_at': '2023_10_25', 'title': 'scaling...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2f19f269-171f-5556-93f3-a2d7eabbe50f</td>\n",
       "      <td># Understanding organizations\\n\\n[Suggest Edit...</td>\n",
       "      <td>https://docs.pinecone.io/docs/organizations</td>\n",
       "      <td>{'created_at': '2023_10_25', 'title': 'organiz...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b2a71cb3-5148-5090-86d5-7f4156edd7cf</td>\n",
       "      <td># Manage datasets\\n\\n[Suggest Edits](/edit/dat...</td>\n",
       "      <td>https://docs.pinecone.io/docs/datasets</td>\n",
       "      <td>{'created_at': '2023_10_25', 'title': 'datasets'}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1dafe68a-2e78-57f7-a97a-93e043462196</td>\n",
       "      <td># Architecture\\n\\n[Suggest Edits](/edit/archit...</td>\n",
       "      <td>https://docs.pinecone.io/docs/architecture</td>\n",
       "      <td>{'created_at': '2023_10_25', 'title': 'archite...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8b07b24d-4ec2-58a1-ac91-c8e6267b9ffd</td>\n",
       "      <td># Moving to production\\n\\n[Suggest Edits](/edi...</td>\n",
       "      <td>https://docs.pinecone.io/docs/moving-to-produc...</td>\n",
       "      <td>{'created_at': '2023_10_25', 'title': 'moving-...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     id  \\\n",
       "0  728aeea1-1dcf-5d0a-91f2-ecccd4dd4272   \n",
       "1  2f19f269-171f-5556-93f3-a2d7eabbe50f   \n",
       "2  b2a71cb3-5148-5090-86d5-7f4156edd7cf   \n",
       "3  1dafe68a-2e78-57f7-a97a-93e043462196   \n",
       "4  8b07b24d-4ec2-58a1-ac91-c8e6267b9ffd   \n",
       "\n",
       "                                                text  \\\n",
       "0  # Scale indexes\\n\\n[Suggest Edits](/edit/scali...   \n",
       "1  # Understanding organizations\\n\\n[Suggest Edit...   \n",
       "2  # Manage datasets\\n\\n[Suggest Edits](/edit/dat...   \n",
       "3  # Architecture\\n\\n[Suggest Edits](/edit/archit...   \n",
       "4  # Moving to production\\n\\n[Suggest Edits](/edi...   \n",
       "\n",
       "                                              source  \\\n",
       "0      https://docs.pinecone.io/docs/scaling-indexes   \n",
       "1        https://docs.pinecone.io/docs/organizations   \n",
       "2             https://docs.pinecone.io/docs/datasets   \n",
       "3         https://docs.pinecone.io/docs/architecture   \n",
       "4  https://docs.pinecone.io/docs/moving-to-produc...   \n",
       "\n",
       "                                            metadata  \n",
       "0  {'created_at': '2023_10_25', 'title': 'scaling...  \n",
       "1  {'created_at': '2023_10_25', 'title': 'organiz...  \n",
       "2  {'created_at': '2023_10_25', 'title': 'datasets'}  \n",
       "3  {'created_at': '2023_10_25', 'title': 'archite...  \n",
       "4  {'created_at': '2023_10_25', 'title': 'moving-...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "data = pd.read_parquet(\"https://storage.googleapis.com/pinecone-datasets-dev/pinecone_docs_ada-002/raw/file1.parquet\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Init a Tokenizer\n",
    "\n",
    "\n",
    "Many of Canopy's components are using tokenization, which is a process that splits text into tokens - basic units of text (like word or sub-words) that are used for processing. Therefore, Canopy uses a singleton `Tokenizer` object which needs to be initialized once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.tokenizer import Tokenizer, LlamaTokenizer\n",
    "Tokenizer.initialize(tokenizer_class=LlamaTokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['▁Hello', '▁world', '!']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer = Tokenizer()\n",
    "tokenizer.tokenize(\"Hello world!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a KnowledgBase to store our data for search\n",
    "\n",
    "The `KnowledgeBase` object is responsible for storing and indexing textual documents.\n",
    "\n",
    "Once documents are indexed, the `KnowledgeBase` can be queried with a new unseen text passage, for which the most relevant document chunks are retrieved.\n",
    "\n",
    "The `KnowledgeBase` holds a connection to a Pinecone index and provides a simple API to insert, delete and search textual documents.\n",
    "\n",
    "The `KnowledgeBase`'s `upsert()` operation is used to index new documents, or update already stored documents. The `upsert` process splits each document's text into smaller chunks, transforms these chunks to vector embeddings, then upserts those vectors to the underlying Pinecone index. At Query time, the `KnowledgeBase` transforms the textual query text to a vector in a similar manner, then queries the underlying Pinecone index to retrieve the top-k most closely matched document chunks.\n",
    "\n",
    "Here we create a `KnowledgeBase` with our desired index name: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.knowledge_base import KnowledgeBase\n",
    "from canopy.knowledge_base.record_encoder import AnyscaleRecordEncoder\n",
    "INDEX_NAME = \"PINECONE_INDEX_NA\"\n",
    "kb = KnowledgeBase(index_name=INDEX_NAME,\n",
    "                  record_encoder=AnyscaleRecordEncoder())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.knowledge_base import list_canopy_indexes\n",
    "\n",
    "if not any(name.endswith(INDEX_NAME) for name in list_canopy_indexes()):\n",
    "    kb.create_canopy_index(indexed_fields=[\"title\"], dimension=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "kb.connect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 💡 Note: a knowledge base must be connected to an index before executing any operation. You should call `kb.connect()` to connect  an existing index or call `kb.create_canopy_index(INDEX_NANE)` before calling any other method of the KB "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upsert data to our KnowledgBase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.models.data_models import Document\n",
    "documents = [Document(**row) for _, row in data.iterrows()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm\n",
    "\n",
    "batch_size = 5\n",
    "\n",
    "for i in tqdm(range(0, len(documents), batch_size)):\n",
    "    kb.upsert(documents[i: i+batch_size])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query the KnowledgeBase\n",
    "\n",
    "Now we can query the knowledge base. The KnowledgeBase will use its default parameters like `top_k` to execute the query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_query_results(results):\n",
    "    for query_results in results:\n",
    "        print('query: ' + query_results.query + '\\n')\n",
    "        for document in query_results.documents:\n",
    "            print('document: ' + document.text.replace(\"\\n\", \"\\\\n\"))\n",
    "            print(\"title: \" + document.metadata[\"title\"])\n",
    "            print('source: ' + document.source)\n",
    "            print(f\"score: {document.score}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.models.data_models import Query\n",
    "results = kb.query([Query(text=\"p1 pod capacity\")])\n",
    "\n",
    "print_query_results(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can change the `top_k` parameter, to determine the number of top query results that will be returned and also to provide a [metadata filter](https://docs.pinecone.io/docs/metadata-filtering)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.models.data_models import Query\n",
    "results = kb.query([Query(text=\"p1 pod capacity\",\n",
    "                          #metadata_filter={\"source\": \"https://docs.pinecone.io/docs/limits\"},\n",
    "                          top_k=2)])\n",
    "\n",
    "print_query_results(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, using the metadata filter we get results only from the \"limits\" page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query the Context Engine\n",
    "\n",
    "`ContextEngine` is an object responsible for retrieving the most relevant context for a given query and token budget.  \n",
    "\n",
    "While `KnowledgeBase` retrieves the full `top-k` structured documents for each query including all the metadata related to them, the context engine in charge of transforming this information to a \"prompt ready\" context that can later feeded to an LLM. To achieve this the context engine holds a `ContextBuilder` object that takes query results from the knowledge base and returns a `Context` object. The `ContextEngine`'s default behavior is to use a `StuffingContextBuilder`, which simply stacks retrieved document chunks in a JSON-like manner, hard limiting by the number of chunks that fit the `max_context_tokens` budget. More complex behaviors can be achieved by providing a custom `ContextBuilder` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.context_engine import ContextEngine\n",
    "context_engine = ContextEngine(kb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "result = context_engine.query([Query(text=\"capacity of p1 pods\", top_k=5)], max_context_tokens=512)\n",
    "\n",
    "print(result.to_text(indent=2))\n",
    "print(f\"\\n# tokens in context returned: {result.num_tokens}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, although we set `top_k=5`, context engine retreived only 3 results in order to satisfy the 512 tokens limit. Also, the documents in the context contain only the text and source and not all the metadata that is not necessarily needed by the LLM. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Knowledgeable chat engine\n",
    "\n",
    "Now we are ready to start chatting with our data!\n",
    "\n",
    "Canopy's `ChatEngine` is a one-stop-shop RAG-infused Chatbot. The `ChatEngine` wraps an underlying LLM such as Llama models, enhancing it by providing relevant context from the user's knowledge base. It also automatically phrases search queries out of the chat history and send them to the knowledge base."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from canopy.chat_engine import ChatEngine\n",
    "from canopy.llm import BaseLLM, AnyscaleLLM\n",
    "from canopy.chat_engine.query_generator import LastMessageQueryGenerator\n",
    "chat_engine = ChatEngine(context_engine,\n",
    "                         llm=AnyscaleLLM(),\n",
    "                         query_builder=LastMessageQueryGenerator())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from typing import Tuple\n",
    "from canopy.models.data_models import Messages, UserMessage, AssistantMessage\n",
    "\n",
    "def chat(new_message: str, history: Messages) -> Tuple[str, Messages]:\n",
    "    messages = history + [UserMessage(content=new_message)]\n",
    "    response = chat_engine.chat(messages)\n",
    "    assistant_response = response.choices[0].message.content\n",
    "    return assistant_response, messages + [AssistantMessage(content=assistant_response)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "  According to the context, the capacity of p1 pods is approximately 1M vectors with 768 dimensions. This information is provided in the document titled \"Limits\" in the context.\n",
       "\n",
       "Source: <https://docs.pinecone.io/docs/limits/>"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "history = []\n",
    "response, history = chat(\"What is the capacity of p1 pods?\", history)\n",
    "display(Markdown(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[UserMessage(role=<Role.USER: 'user'>, content='What is the capacity of p1 pods?'),\n",
       " AssistantMessage(role=<Role.ASSISTANT: 'assistant'>, content='  According to the context, the capacity of p1 pods is approximately 1M vectors with 768 dimensions. This information is provided in the document titled \"Limits\" in the context.\\n\\nSource: <https://docs.pinecone.io/docs/limits/>')]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "  According to the context, p1 pods are optimized for very low query latencies, with latencies of less than 100ms. This means that they are suitable for applications that require fast query responses, such as real-time analytics or dashboards.\n",
       "\n",
       "Source: <https://docs.pinecone.io/docs/choosing-index-type-and-size/>\n",
       "\n",
       "In contrast, p2 pods are optimized for greater query throughput with lower latency, with query latencies of less than 10ms for vectors with fewer than 128 dimensions. This makes them more suitable for applications with moderate to high query volumes, such as data-intensive analytics or machine learning workloads.\n",
       "\n",
       "Source: <https://docs.pinecone.io/docs/p2-pods>\n",
       "\n",
       "Therefore, the choice of pod type depends on the specific latency requirements of the application. If very low latencies are required, p1 pods may be a better fit, while if moderate to high query volumes are involved, p2 pods may be more suitable."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response, history = chat(\"And for what latency requirements does it fit?\", history)\n",
    "display(Markdown(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[UserMessage(role=<Role.USER: 'user'>, content='What is the capacity of p1 pods?'),\n",
       " AssistantMessage(role=<Role.ASSISTANT: 'assistant'>, content='  According to the context, the capacity of p1 pods is approximately 1M vectors with 768 dimensions. This information is provided in the document titled \"Limits\" in the context.\\n\\nSource: <https://docs.pinecone.io/docs/limits/>'),\n",
       " UserMessage(role=<Role.USER: 'user'>, content='And for what latency requirements does it fit?'),\n",
       " AssistantMessage(role=<Role.ASSISTANT: 'assistant'>, content='  According to the context, p1 pods are optimized for very low query latencies, with latencies of less than 100ms. This means that they are suitable for applications that require fast query responses, such as real-time analytics or dashboards.\\n\\nSource: <https://docs.pinecone.io/docs/choosing-index-type-and-size/>\\n\\nIn contrast, p2 pods are optimized for greater query throughput with lower latency, with query latencies of less than 10ms for vectors with fewer than 128 dimensions. This makes them more suitable for applications with moderate to high query volumes, such as data-intensive analytics or machine learning workloads.\\n\\nSource: <https://docs.pinecone.io/docs/p2-pods>\\n\\nTherefore, the choice of pod type depends on the specific latency requirements of the application. If very low latencies are required, p1 pods may be a better fit, while if moderate to high query volumes are involved, p2 pods may be more suitable.')]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 💡 Note: Canopy calls the underlying LLM, providing both the user-provided chat history and a generated `Context` prompt. This might surpass the `ChatEngine`'s configured `max_prompt_tokens`. By default, the `ChatEngine` would truncate the oldest messages in the chat history to avoid exceeding this limit. This behavior in configurable, as explained in the [documentation](https://github.com/pinecone-io/canopy/blob/main/src/canopy/chat_engine/chat_engine.py)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "9e9b81017be88d4d093a2a92984a986685ce96a6b6736b12c233fdf6b743e185"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}