tutorials/milvus_llamaindex_search_and_retrieval_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XN9mbpT2SWSx"
   },
   "source": [
    "<center>\n",
    "    <p style=\"text-align:center\">\n",
    "        <img alt=\"arize llama-index logos\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/milvus_tutorial/arize_zilliz.png\" width=\"400\">\n",
    "        <br>\n",
    "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
    "        |\n",
    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
    "        |\n",
    "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
    "    </p>\n",
    "</center>\n",
    "<h1 align=\"center\">Evaluating and Improving a LlamaIndex Search and Retrieval Application using Milvus as a Vector Store</h1>\n",
    "\n",
    "Imagine you're an engineer at Arize AI and you've built and deployed a documentation question-answering service using LlamaIndex. Users send questions about Arize's core product via a chat interface, and your service retrieves documents from your documentation in order to generate a response to the user. As the engineer in charge of evaluating and maintaining this system, you want to evaluate the quality of the responses from your service.\n",
    "\n",
    "Phoenix helps you:\n",
    "- identify gaps in your documentation\n",
    "- detect queries for which the LLM gave bad responses\n",
    "- detect failures to retrieve relevant context\n",
    "\n",
    "In this tutorial, you will:\n",
    "\n",
    "- Load pre-built knowledge data into a Milvus vector store on Zilliz Cloud\n",
    "- Download a pre-indexed knowledge base of the Arize documentation and run a LlamaIndex application\n",
    "- Visualize user queries and knowledge base documents to identify areas of user interest not answered by your documentation\n",
    "- Find clusters of responses with negative user feedback\n",
    "- Identify failed retrievals using cosine similarity, Euclidean distance, and LLM-assisted ranking metrics\n",
    "\n",
    "\n",
    "## Chatbot Architecture\n",
    "\n",
    "Your chatbot was built using LlamaIndex's low-level API. The architecture of your chatbot is shown below and can be explained in five steps.\n",
    "\n",
    "![llama-index chatbot architecture](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llama-index-knowledge-base-tutorial/llama_index_chatbot_architecture.png)\n",
    "\n",
    "1. The user sends a query about Arize to your service.\n",
    "1. `langchain.embeddings.OpenAIEmbeddings` makes a request to the OpenAI embeddings API to embed the user query using the `text-embedding-ada-002` model.\n",
    "1. `llama_index.retrievers.RetrieverQueryEngine` does a similarity search against the entries of your index knowledge base for the two most similar pieces of context by cosine similarity.\n",
    "1. `llama_index.indices.query.ResponseSynthesizer` generates a response by formatting the query and retrieved context into a single prompt and sending a request to OpenAI chat completions API with the `gpt-3.5-turbo`.\n",
    "2. The response is returned to the user.\n",
    "\n",
    "Phoenix makes your search and retrieval system *observable* by capturing the inputs and outputs of these steps for analysis, including:\n",
    "\n",
    "- your query embeddings\n",
    "- the retrieved context and similarity scores for each query\n",
    "- the generated response that is return to the user\n",
    "\n",
    "With that overview in mind, let's dive into the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l3qXam7rSWSy"
   },
   "source": [
    "## 1. Install Dependencies and Import Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HXJ6OHBsYubW"
   },
   "source": [
    "Install Phoenix, LlamaIndex, and Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install gcsfs \"langchain>=0.0.334\" \"arize-phoenix[evals,llama-index]>=3.20.0\" \"openai>=1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pymilvus==2.2.15\n",
    "!pip install --upgrade --force-reinstall grpcio==1.56.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KS2ONAM3Y1XD"
   },
   "source": [
    "###**Please restart your runtime here. Colab grpc version might cause issues during import.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KZpntsbPSWSz"
   },
   "source": [
    "Import the necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "import os\n",
    "import sys\n",
    "import textwrap\n",
    "import urllib.request\n",
    "from datetime import timedelta\n",
    "from getpass import getpass\n",
    "\n",
    "import numpy as np\n",
    "import openai\n",
    "import pandas as pd\n",
    "import phoenix as px\n",
    "from IPython.display import YouTubeVideo\n",
    "from llama_index import (\n",
    "    ServiceContext,\n",
    "    VectorStoreIndex,\n",
    ")\n",
    "from llama_index.callbacks import CallbackManager, OpenInferenceCallbackHandler\n",
    "from llama_index.callbacks.open_inference_callback import as_dataframe\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "from llama_index.llms import OpenAI\n",
    "from llama_index.schema import NodeRelationship, RelatedNodeInfo, TextNode\n",
    "from llama_index.vector_stores import MilvusVectorStore\n",
    "from llama_index.vector_stores.utils import (\n",
    "    metadata_dict_to_node,\n",
    ")\n",
    "from phoenix.evals.retrievals import (\n",
    "    classify_relevance,\n",
    "    compute_precisions_at_k,\n",
    ")\n",
    "\n",
    "logging.disable(sys.maxsize)\n",
    "pd.set_option(\"display.max_colwidth\", 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kkoc1IC1SWSz"
   },
   "source": [
    "## 2. Configure your OpenAI API Key\n",
    "💭 Configure your OpenAI API key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
    "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "openai.api_key = openai_api_key\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HC9YmlPSSWS0"
   },
   "source": [
    "## 3. Download Your Proprietary Data\n",
    "Download your Arize documentation data from cloud storage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://storage.googleapis.com/arize-assets/xander/milvus-workshop/milvus_dataset.json\"\n",
    "\n",
    "with urllib.request.urlopen(url) as response:\n",
    "    buffer = response.read()\n",
    "    data = json.loads(buffer.decode(\"utf-8\"))\n",
    "    rows = data[\"rows\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jv6wN7kYSWS0"
   },
   "source": [
    "## 4. Create your Milvus Vector Store (Knowledge Base) with Zilliz Cloud\n",
    "\n",
    "We will be using Zilliz Cloud and create a knowledge base using Milvus Vector Store. In order to get it running, we need the following steps.\n",
    "\n",
    "1. Go to [Zilliz Cloud](https://cloud.zilliz.com/) and create an account for free.\n",
    "2. Define an organization and project within Zilliz to get started.\n",
    "3. Add a standard cluster to your project. You should have received 100$ free credit on sign-up, so standard cluster will come at no cost for this tutorial.\n",
    "4. Copy/paste your public endpoint and token into the cells below after creating the cluster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENDPOINT = getpass(prompt=\"Please set your public endpoint from Zilliz Cloud: \")\n",
    "TOKEN = getpass(prompt=\"Enter your token from Zilliz Cloud: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to Zilliz Cloud and instantiate a Milvus Vector Store\n",
    "vector_store = MilvusVectorStore(\n",
    "    uri=ENDPOINT,\n",
    "    token=TOKEN,\n",
    "    collection_name=\"colab_collection\",\n",
    "    dim=1536,\n",
    "    embedding_field=\"embedding\",\n",
    "    doc_id_field=\"doc_id\",\n",
    "    overwrite=True,\n",
    ")\n",
    "\n",
    "# Insert the downloaded Arize documentation data to the vector store\n",
    "nodes = []\n",
    "for row in rows:\n",
    "    node = TextNode(\n",
    "        embedding=row[\"embedding\"],\n",
    "        text=row[\"text\"],\n",
    "        id_=row[\"id\"],\n",
    "        relationships={NodeRelationship.SOURCE: RelatedNodeInfo(node_id=row[\"doc_id\"])},\n",
    "    )\n",
    "    nodes.append(node)\n",
    "\n",
    "vector_store.add(nodes)\n",
    "print(\"Successfully added Arize documentation data into the vector store!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TBZpavN4bsg9"
   },
   "source": [
    "## 5. Run Your Question-Answering Service\n",
    "💭 Start a LlamaIndex application from your downloaded index. Use the OpenInferenceCallbackHandler to store your data in OpenInference format, an open standard for capturing and storing AI model inferences that enables production LLMapp servers to seamlessly integrate with LLM observability solutions such as Arize and Phoenix.\n",
    "\n",
    "After starting the application, ask a few questions of your question-answering service and view the responses.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "callback_handler = OpenInferenceCallbackHandler()\n",
    "\n",
    "service_context = ServiceContext.from_defaults(\n",
    "    llm=OpenAI(model_name=\"gpt-3.5-turbo\", temperature=0),\n",
    "    embed_model=OpenAIEmbedding(model=\"text-embedding-ada-002\"),\n",
    "    callback_manager=CallbackManager(handlers=[callback_handler]),\n",
    ")\n",
    "\n",
    "index = VectorStoreIndex.from_vector_store(vector_store, service_context=service_context)\n",
    "\n",
    "query_engine = index.as_query_engine()\n",
    "\n",
    "max_line_length = 80\n",
    "for query in [\n",
    "    \"How do I get an Arize API key?\",\n",
    "    \"Can I create monitors with an API?\",\n",
    "    \"How do I need to format timestamps?\",\n",
    "    \"What is the price of the Arize platform\",\n",
    "]:\n",
    "    print(\"Query\")\n",
    "    print(\"=====\")\n",
    "    print()\n",
    "    print(textwrap.fill(query, max_line_length))\n",
    "    print()\n",
    "    response = query_engine.query(query)\n",
    "    print(\"Response\")\n",
    "    print(\"========\")\n",
    "    print()\n",
    "    print(textwrap.fill(str(response), max_line_length))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y3IQgDOtSWS0"
   },
   "source": [
    "## 6. Load Your Data Into Pandas Dataframes\n",
    "\n",
    "\n",
    "To use Phoenix, you must load your data into pandas dataframes.\n",
    "\n",
    "Your query data is saved in a buffer on the callback handler you defined previously. Load the data from the buffer into a dataframe.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_data_buffer = callback_handler.flush_query_data_buffer()\n",
    "sample_query_df = as_dataframe(query_data_buffer)\n",
    "sample_query_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8bLAhaKD8RMw"
   },
   "source": [
    "The columns of the dataframe are:\n",
    "\n",
    "* **:id.id::** the query ID\n",
    "* **:timestamp.iso_8601::** the time at which the query was made\n",
    "* **:feature.text:prompt:** the query text\n",
    "* **:feature.[float].embedding:prompt:** the embedding representation of the query\n",
    "* **:prediction.text:response:** the final response presented to the user\n",
    "* **:feature.[str].retrieved_document_ids:prompt:** the list of IDs of the retrieved documents\n",
    "* **:feature.[float].retrieved_document_scores:prompt:** the lists of cosine similarities between the query and retrieved documents\n",
    "\n",
    "The column names are in OpenInference format and describe the category, data type and intent of each column.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eLbghTsSc9BH"
   },
   "source": [
    "Running queries against a large dataset takes a long time. Download a dataframe containing query data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_df = pd.read_parquet(\n",
    "    \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/llm/llama-index/arize-docs/query_data_complete3.parquet\",\n",
    ")\n",
    "query_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jZR7uwBHSWS0"
   },
   "source": [
    "\n",
    "\n",
    "In addition to the columns of the previous dataframe, this data has a few additional fields:\n",
    "\n",
    "* **:tag.float:user_feedback:** approval or rejection from the user (-1 means thumbs down, +1 means thumbs up)\n",
    "* **:tag.str:openai_relevance_0:** a binary classification (relevant vs. irrelevant) by GPT-4 predicting whether the first retrieved document is relevant to the query\n",
    "* **:tag.str:openai_relevance_1:** a binary classification (relevant vs. irrelevant) by GPT-4 predicting whether the second retrieved document is relevant to the query\n",
    "\n",
    "\n",
    "We'll go over how to compute the relevance classifications in section 7.\n",
    "\n",
    "Next load your knowledge base into a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use vector store client to query the knowledge base and load our knowledge base into a dataframe\n",
    "myclient = vector_store.client\n",
    "document_embeddings, document_ids, document_texts = [], [], []\n",
    "i = 0\n",
    "\n",
    "all_data = myclient.query(collection_name=\"colab_collection\", filter='id >= \"\"')\n",
    "for x in all_data:\n",
    "    node = metadata_dict_to_node({\"_node_content\": all_data[i][\"_node_content\"]})\n",
    "    document_embeddings.append(np.array(x[\"embedding\"]))\n",
    "    document_ids.append(node.hash)\n",
    "    document_texts.append(node.text)\n",
    "    i = i + 1\n",
    "\n",
    "database_df = pd.DataFrame(\n",
    "    {\n",
    "        \"document_id\": document_ids,\n",
    "        \"text\": document_texts,\n",
    "        \"text_vector\": document_embeddings,\n",
    "    }\n",
    ")\n",
    "database_df = database_df[database_df[\"text\"] != \"\\n\"]\n",
    "database_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c2tv37jrSWS1"
   },
   "source": [
    "The columns of your dataframe are:\n",
    "\n",
    "- **document_id:** the ID of the chunked document\n",
    "- **text:** the chunked text in your knowledge base\n",
    "- **text_vector:** the embedding vector for the text, computed during the LlamaIndex build using \"text-embedding-ada-002\" from OpenAI\n",
    "\n",
    "The query and database datasets are drawn from different distributions; the queries are short questions while the database entries are several sentences to a paragraph. The embeddings from OpenAI's \"text-embedding-ada-002\" capture these differences and naturally separate the query and context embeddings into distinct regions of the embedding space. When using Phoenix, you want to \"overlay\" the query and context embedding distributions so that queries appear close to their retrieved context in the Phoenix point cloud. To achieve this, we compute a centroid for each dataset that represents an average point in the embedding distribution and center the two distributions so they overlap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "database_embedding_column_name = \"text_vector\"\n",
    "database_centroid = database_df[database_embedding_column_name].mean()\n",
    "database_df[database_embedding_column_name] = database_df[database_embedding_column_name].apply(\n",
    "    lambda x: x - database_centroid\n",
    ")\n",
    "query_embedding_column_name = \":feature.[float].embedding:prompt\"\n",
    "query_centroid = query_df[query_embedding_column_name].mean()\n",
    "query_df[query_embedding_column_name] = query_df[query_embedding_column_name].apply(\n",
    "    lambda x: x - query_centroid\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fWDdxBElSWS1"
   },
   "source": [
    "## 7. Run LLM-Assisted Evaluations\n",
    "\n",
    "Cosine similarity and Euclidean distance are reasonable proxies for retrieval quality, but they don't always work perfectly. A novel idea is to use LLMs to measure retrieval quality by simply asking the LLM whether each retrieved document is relevant to the corresponding query.\n",
    "\n",
    "💭 Use OpenAI to predict whether each retrieved document is relevant or irrelevant to the query.\n",
    "\n",
    "⚠️ It's strongly recommended to use GPT-4 for evaluations if you have access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evals_model_name = \"gpt-3.5-turbo\"\n",
    "# evals_model_name = \"gpt-4\"  # use GPT-4 if you have access\n",
    "query_texts = sample_query_df[\":feature.text:prompt\"].tolist()\n",
    "list_of_document_id_lists = sample_query_df[\":feature.[str].retrieved_document_ids:prompt\"].tolist()\n",
    "document_id_to_text = dict(zip(database_df[\"document_id\"].to_list(), database_df[\"text\"].to_list()))\n",
    "doc_texts = []\n",
    "for document_index in [0]:\n",
    "    for document_ids in list_of_document_id_lists:\n",
    "        doc_texts.append(document_id_to_text[document_ids[document_index]])\n",
    "\n",
    "relevance = []\n",
    "for query_text, document_text in zip(query_texts, doc_texts):\n",
    "    relevance.append(classify_relevance(query_text, document_text, evals_model_name))\n",
    "\n",
    "sample_query_df = sample_query_df.assign(\n",
    "    retrieved_document_text_0=doc_texts,\n",
    "    relevance_0=relevance,\n",
    ")\n",
    "sample_query_df[\n",
    "    [\n",
    "        \":feature.text:prompt\",\n",
    "        \"retrieved_document_text_0\",\n",
    "        \"relevance_0\",\n",
    "    ]\n",
    "].rename(columns={\":feature.text:prompt\": \"query_text\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EcC6A04RSWS1"
   },
   "source": [
    "## 8. Compute Ranking Metrics\n",
    "\n",
    "Now that you know whether each piece of retrieved context is relevant or irrelevant to the corresponding query, you can compute precision@k for k = 1, 2 for each query. This metric tells you what percentage of the retrieved context is relevant to the corresponding query.\n",
    "\n",
    "precision@k = (# of top-k retrieved documents that are relevant) / (k retrieved documents)\n",
    "\n",
    "If your precision@2 is greater than zero for a particular query, your LlamaIndex application successfully retrieved at least one relevant piece of context with which to answer the query. If the precision@k is zero for a particular query, that means that no relevant piece of context was retrieved.\n",
    "\n",
    "Compute precision@k for k = 1, 2 and view the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_document_relevances = [\n",
    "    {\"relevant\": True, \"irrelevant\": False}.get(rel)\n",
    "    for rel in query_df[\":tag.str:openai_relevance_0\"].tolist()\n",
    "]\n",
    "second_document_relevances = [\n",
    "    {\"relevant\": True, \"irrelevant\": False}.get(rel)\n",
    "    for rel in query_df[\":tag.str:openai_relevance_1\"].tolist()\n",
    "]\n",
    "\n",
    "list_of_precisions_at_k_lists = [\n",
    "    compute_precisions_at_k([rel0, rel1])\n",
    "    for rel0, rel1 in zip(first_document_relevances, second_document_relevances)\n",
    "]\n",
    "precisions_at_1, precisions_at_2 = [\n",
    "    [precisions_at_k[index] for precisions_at_k in list_of_precisions_at_k_lists]\n",
    "    for index in [0, 1]\n",
    "]\n",
    "query_df[\":tag.float:openai_precision_at_1\"] = precisions_at_1\n",
    "query_df[\":tag.float:openai_precision_at_2\"] = precisions_at_2\n",
    "query_df[\n",
    "    [\n",
    "        \":tag.str:openai_relevance_0\",\n",
    "        \":tag.str:openai_relevance_1\",\n",
    "        \":tag.float:openai_precision_at_1\",\n",
    "        \":tag.float:openai_precision_at_2\",\n",
    "    ]\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f_7lFqvmSWS1"
   },
   "source": [
    "# 9. Launch phoenix\n",
    "\n",
    "Define your knowledge base dataset with a schema that specifies the meaning of each column (features, predictions, actuals, tags, embeddings, etc.). See the [docs](https://docs.arize.com/phoenix/) for guides on how to define your own schema and API reference on `phoenix.Schema` and `phoenix.EmbeddingColumnNames`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "database_schema = px.Schema(\n",
    "    prediction_id_column_name=\"document_id\",\n",
    "    prompt_column_names=px.EmbeddingColumnNames(\n",
    "        vector_column_name=\"text_vector\",\n",
    "        raw_data_column_name=\"text\",\n",
    "    ),\n",
    ")\n",
    "database_ds = px.Inferences(\n",
    "    dataframe=database_df,\n",
    "    schema=database_schema,\n",
    "    name=\"database\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z-JeimqdSWS1"
   },
   "source": [
    "Define your query dataset. Because the query dataframe is in OpenInference format, Phoenix is able to infer the meaning of each column without a user-defined schema by using the phoenix.Dataset.from_open_inference class method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_ds = px.Inferences.from_open_inference(query_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-xUuyryhSWS1"
   },
   "source": [
    "Launch Phoenix. Follow the instructions in the cell output to open the Phoenix UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = px.launch_app(primary=query_ds, corpus=database_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VNeNrPUUSWS1"
   },
   "source": [
    "# Surface Problematic Clusters and Data Points\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "Phoenix helps you:\n",
    "\n",
    "- visualize your embeddings\n",
    "- color the resulting point cloud using evaluation metrics\n",
    "- cluster the points and surface up problematic clusters based on whatever metric you care about\n",
    "\n",
    "Follow along with the tutorial walkthrough [here](https://youtu.be/hbQYDpJayFw?t=1782), or view the video in your notebook by running the cell below. The video will show you how to investigate your query and knowledge base and identify problematic clusters of data points using Phoenix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_time_in_seconds = int(timedelta(hours=0, minutes=29, seconds=42).total_seconds())\n",
    "YouTubeVideo(\"hbQYDpJayFw\", start=start_time_in_seconds, width=560, height=315)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}