llm_observability_langchain_tracing_tutorial_v3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gc9NI3_Y9WCv"
      },
      "source": [
        "<center>\n",
        "    <p style=\"text-align:center\">\n",
        "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-assets/phoenix/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
        "        <br>\n",
        "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
        "        |\n",
        "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
        "        |\n",
        "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
        "    </p>\n",
        "</center>\n",
        "<h1 align=\"center\">Tracing and Evaluating a LangChain Application</h1>\n",
        "\n",
        "LangChain provides high-level APIs that enable users to build powerful applications in a few lines of code. However, it can be challenging to understand what is going on under the hood and to pinpoint the cause of issues. Phoenix makes your LLM applications *observable* by visualizing the underlying structure of each call to your query engine and surfacing problematic \"spans\" of execution based on latency, token count, or other evaluation metrics.\n",
        "\n",
        "In this tutorial, you will:\n",
        "- Build a simple question and answer app using LangChain that uses retrieval-augmented generation to answer questions over the Arize documentation,\n",
        "- Record trace data in OpenInference format,\n",
        "- Inspect the traces and spans of your application to identify sources of latency and cost,\n",
        "- Export your trace data as a pandas dataframe and run an LLM-assisted evaluation to measure the precision@k of your retrieval step.\n",
        "\n",
        "ℹ️ This notebook requires an OpenAI API key."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i58AlLRv9WCw"
      },
      "source": [
        "## 1. Install Dependencies and Import Libraries\n",
        "\n",
        "Install Phoenix, LangChain, and OpenAI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xgp9l19K9WCx"
      },
      "outputs": [],
      "source": [
        "!pip install \"langchain>=0.0.334\" \"openai>=1\" arize-phoenix tiktoken nest-asyncio"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# import os\n",
        "# from getpass import getpass\n",
        "# # Uses your PHOENIX_COLLECTOR_ENDPOINT Key\n",
        "# if not (collector_endpoint := os.getenv(\"PHOENIX_COLLECTOR_ENDPOINT\")):\n",
        "#     collector_endpoint = getpass(\"🔑 Enter your Phoenix Collector Endpoint: \")\n",
        "\n",
        "# os.environ[\"PHOENIX_COLLECTOR_ENDPOINT\"] = collector_endpoint"
      ],
      "metadata": {
        "id": "BrJ2CqJo9wvo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IYchLndz9WCx"
      },
      "source": [
        "Import libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Cr2m-cu49WCy"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import os\n",
        "from getpass import getpass\n",
        "from urllib.request import urlopen\n",
        "\n",
        "import nest_asyncio\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import phoenix as px\n",
        "from langchain.chains import RetrievalQA\n",
        "from langchain.chat_models import ChatOpenAI\n",
        "from langchain.embeddings import OpenAIEmbeddings\n",
        "from langchain.retrievers import KNNRetriever\n",
        "from phoenix.experimental.evals import (\n",
        "    HallucinationEvaluator,\n",
        "    OpenAIModel,\n",
        "    QAEvaluator,\n",
        "    RelevanceEvaluator,\n",
        "    run_evals,\n",
        ")\n",
        "from phoenix.session.evaluation import get_qa_with_reference, get_retrieved_documents\n",
        "from phoenix.trace import DocumentEvaluations, SpanEvaluations\n",
        "from phoenix.trace.langchain import LangChainInstrumentor\n",
        "from tqdm import tqdm\n",
        "\n",
        "nest_asyncio.apply()  # needed for concurrent evals in notebook environments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Us3S6xid9WCy"
      },
      "source": [
        "## 2. Launch Phoenix\n",
        "\n",
        "You can run Phoenix in the background to collect trace data emitted by any LangChain application that has been instrumented with the `OpenInferenceTracer`.\n",
        "\n",
        "Launch Phoenix and follow the instructions in the cell output to open the Phoenix UI (the UI should be empty because we have yet to run a LangChain application)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D2562XBX9WCz"
      },
      "outputs": [],
      "source": [
        "session = px.launch_app()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zAW8dAno9WCz"
      },
      "source": [
        "## 3. Configure Your OpenAI API Key\n",
        "\n",
        "Set your OpenAI API key if it is not already set as an environment variable."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K2h57bNP9WCz"
      },
      "outputs": [],
      "source": [
        "if os.environ.get(\"OPENAI_API_KEY\") is None:\n",
        "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
        "    os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WEfY34dM9WCz"
      },
      "source": [
        "### 4. Build Your LangChain Application\n",
        "\n",
        "This example uses a `RetrievalQA` chain over a pre-built index of the Arize documentation, but you can use whatever LangChain application you like.\n",
        "\n",
        "Download your pre-built index from cloud storage and instantiate your storage context."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JExlrHMB9WCz"
      },
      "outputs": [],
      "source": [
        "embeddings = OpenAIEmbeddings(model=\"text-embedding-ada-002\")\n",
        "df = pd.read_parquet(\n",
        "    \"http://storage.googleapis.com/arize-assets/phoenix/datasets/\"\n",
        "    \"unstructured/llm/context-retrieval/langchain-pinecone/database.parquet\"\n",
        ")\n",
        "knn_retriever = KNNRetriever(\n",
        "    index=np.stack(df[\"text_vector\"]),\n",
        "    texts=df[\"text\"].tolist(),\n",
        "    embeddings=OpenAIEmbeddings(),\n",
        ")\n",
        "chain_type = \"stuff\"  # stuff, refine, map_reduce, and map_rerank\n",
        "chat_model_name = \"gpt-3.5-turbo\"\n",
        "llm = ChatOpenAI(model_name=chat_model_name)\n",
        "chain = RetrievalQA.from_chain_type(\n",
        "    llm=llm,\n",
        "    chain_type=chain_type,\n",
        "    retriever=knn_retriever,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SV6bc7529WC0"
      },
      "source": [
        "Instrument LangChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q7JbjzVa9WC0"
      },
      "outputs": [],
      "source": [
        "LangChainInstrumentor().instrument()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vuq8QtWe9WC0"
      },
      "source": [
        "## 5. Run Your Query Engine and View Your Traces in Phoenix\n",
        "\n",
        "Download a sample of queries commonly asked of the Arize documentation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jR4e9bbN9WC0"
      },
      "outputs": [],
      "source": [
        "url = \"http://storage.googleapis.com/arize-assets/phoenix/datasets/unstructured/llm/context-retrieval/arize_docs_queries.jsonl\"\n",
        "queries = []\n",
        "with urlopen(url) as response:\n",
        "    for line in response:\n",
        "        line = line.decode(\"utf-8\").strip()\n",
        "        data = json.loads(line)\n",
        "        queries.append(data[\"query\"])\n",
        "queries[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p_7PJnQV9WC0"
      },
      "source": [
        "Run a few queries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TQVaq_0j9WC0"
      },
      "outputs": [],
      "source": [
        "for query in tqdm(queries[:10]):\n",
        "    chain.run(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t1uGY1YS9WC0"
      },
      "source": [
        "Check the Phoenix UI as your queries run. Your traces should appear in real time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iowSk_8c9WC1"
      },
      "source": [
        "## 6. Export and Evaluate Your Trace Data\n",
        "\n",
        "You can export your trace data as a pandas dataframe for further analysis and evaluation.\n",
        "\n",
        "In this case, we will export our `retriever` spans into two separate dataframes:\n",
        "- `queries_df`, in which the retrieved documents for each query are concatenated into a single column,\n",
        "- `retrieved_documents_df`, in which each retrieved document is \"exploded\" into its own row to enable the evaluation of each query-document pair in isolation.\n",
        "\n",
        "This will enable us to compute multiple kinds of evaluations, including:\n",
        "- relevance: Are the retrieved documents grounded in the response?\n",
        "- Q&A correctness: Are your application's responses grounded in the retrieved context?\n",
        "- hallucinations: Is your application making up false information?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Yumo5DY9WC1"
      },
      "outputs": [],
      "source": [
        "queries_df = get_qa_with_reference(px.Client())\n",
        "retrieved_documents_df = get_retrieved_documents(px.Client())"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# from datetime import datetime, timedelta\n",
        "# from typing import List, Optional, Protocol, Union, cast\n",
        "\n",
        "\n",
        "# from phoenix.trace.dsl import SpanQuery\n",
        "# from phoenix.trace.semantic_conventions import (\n",
        "#     DOCUMENT_CONTENT,\n",
        "#     DOCUMENT_SCORE,\n",
        "#     INPUT_VALUE,\n",
        "#     OUTPUT_VALUE,\n",
        "#     RETRIEVAL_DOCUMENTS,\n",
        "# )\n",
        "\n",
        "# INPUT = {\"input\": INPUT_VALUE}\n",
        "# OUTPUT = {\"output\": OUTPUT_VALUE}\n",
        "# IO = {**INPUT, **OUTPUT}\n",
        "\n",
        "# IS_ROOT = \"parent_id is None\"\n",
        "# IS_LLM = \"span_kind == 'LLM'\"\n",
        "# IS_RETRIEVER = \"span_kind == 'RETRIEVER'\"\n",
        "\n",
        "# class CanQuerySpans(Protocol):\n",
        "#     def query_spans(self, *query: SpanQuery) -> Optional[Union[pd.DataFrame, List[pd.DataFrame]]]:\n",
        "#         ...\n",
        "\n",
        "# def get_qa_with_reference(obj: CanQuerySpans, start_time: Optional[datetime] = None, stop_time: Optional[datetime] = None) -> pd.DataFrame:\n",
        "#     results = obj.query_spans(\n",
        "#         SpanQuery().select(**IO).where(IS_ROOT),\n",
        "#         SpanQuery()\n",
        "#         .where(IS_RETRIEVER)\n",
        "#         .select(span_id=\"parent_id\")\n",
        "#         .concat(\n",
        "#             RETRIEVAL_DOCUMENTS,\n",
        "#             reference=DOCUMENT_CONTENT,\n",
        "#         ),\n",
        "#         start_time=start_time,\n",
        "#         stop_time=stop_time\n",
        "#     )\n",
        "\n",
        "#     # Concatenate results if they are in a list, otherwise use directly\n",
        "#     if isinstance(results, list):\n",
        "#         df = pd.concat(results, axis=1, join='inner')\n",
        "#     else:\n",
        "#         df = results\n",
        "\n",
        "#     return df\n",
        "\n",
        "\n",
        "# def get_retrieved_documents(obj: CanQuerySpans, start_time: Optional[datetime] = None, stop_time: Optional[datetime] = None) -> pd.DataFrame:\n",
        "#     results = obj.query_spans(\n",
        "#         SpanQuery()\n",
        "#         .where(IS_RETRIEVER)\n",
        "#         .select(\"trace_id\", **INPUT)\n",
        "#         .explode(\n",
        "#             RETRIEVAL_DOCUMENTS,\n",
        "#             reference=DOCUMENT_CONTENT,\n",
        "#             document_score=DOCUMENT_SCORE,\n",
        "#         ),\n",
        "#         start_time=start_time,\n",
        "#         stop_time=stop_time\n",
        "#     )\n",
        "\n",
        "#     # If results are in a list, concatenate, otherwise use directly\n",
        "#     if isinstance(results, list):\n",
        "#         df = pd.concat(results, ignore_index=True)\n",
        "#     else:\n",
        "#         df = results\n",
        "\n",
        "#     return df\n"
      ],
      "metadata": {
        "id": "7UjxC2ZpDCrt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from datetime import datetime, timedelta\n",
        "# now = datetime.now()\n",
        "\n",
        "# # Subtract 1 hour from the current datetime\n",
        "# start_time = now - timedelta(minutes=5)\n",
        "\n",
        "\n",
        "# my_qa = get_qa_with_reference(px.Client(), start_time, datetime.now())\n",
        "# my_qa.head(5)\n"
      ],
      "metadata": {
        "id": "Lz2qhKfEDZmn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# my_retrieved = get_retrieved_documents(px.Client(), start_time, datetime.now())\n",
        "# my_retrieved.head(5)"
      ],
      "metadata": {
        "id": "JFlOq9boOq2a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# print(\"Number of rows:\", my_qa.shape[0])\n",
        "# print(\"Number of rows:\", my_retrieved.shape[0])"
      ],
      "metadata": {
        "id": "j2Pe-ezH_Amr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "py3YoWl99WC1"
      },
      "source": [
        "Next, define your evaluation model and your evaluators.\n",
        "\n",
        "Evaluators are built on top of language models and prompt the LLM to assess the quality of responses, the relevance of retrieved documents, etc., and provide a quality signal even in the absence of human-labeled data. Pick an evaluator type and instantiate it with the language model you want to use to perform evaluations using our battle-tested evaluation templates."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7KNcYgcX9WC1"
      },
      "outputs": [],
      "source": [
        "eval_model = OpenAIModel(\n",
        "    model_name=\"gpt-4-turbo-preview\",\n",
        ")\n",
        "hallucination_evaluator = HallucinationEvaluator(eval_model)\n",
        "qa_correctness_evaluator = QAEvaluator(eval_model)\n",
        "relevance_evaluator = RelevanceEvaluator(eval_model)\n",
        "\n",
        "hallucination_eval_df, qa_correctness_eval_df = run_evals(\n",
        "    dataframe=queries_df, # my_qa,\n",
        "    evaluators=[hallucination_evaluator, qa_correctness_evaluator],\n",
        "    provide_explanation=True,\n",
        ")\n",
        "relevance_eval_df = run_evals(\n",
        "    dataframe=retrieved_documents_df, # my_retrieved,\n",
        "    evaluators=[relevance_evaluator],\n",
        "    provide_explanation=True,\n",
        ")[0]\n",
        "\n",
        "px.log_evaluations(\n",
        "    SpanEvaluations(eval_name=\"Hallucination\", dataframe=hallucination_eval_df),\n",
        "    SpanEvaluations(eval_name=\"QA Correctness\", dataframe=qa_correctness_eval_df),\n",
        ")\n",
        "px.log_evaluations(DocumentEvaluations(eval_name=\"Relevance\", dataframe=relevance_eval_df))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPA_PL-e9WC1"
      },
      "source": [
        "Your evaluations should now appear as annotations on the appropriate spans in Phoenix."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AXGlmg8H9WC1"
      },
      "outputs": [],
      "source": [
        "print(f\"🚀 Open the Phoenix UI if you haven't already: {session.url}\")\n",
        "session = px.active_session()\n",
        "session.view()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lqZ5zRJ49WC1"
      },
      "source": [
        "## 7. Final Thoughts\n",
        "\n",
        "LLM Traces and the accompanying OpenInference Tracing specification is designed to be a category of telemetry data that is used to understand the execution of LLMs and the surrounding application context such as retrieval from vector stores and the usage of external tools such as search engines or APIs. It lets you understand the inner workings of the individual steps your application takes wile also giving you visibility into how your system is running and performing as a whole.\n",
        "\n",
        "LLM Evals are designed for simple, fast, and accurate LLM-based evaluations. They let you quickly benchmark the performance of your LLM application and help you identify the problematic spans of execution.\n",
        "\n",
        "For more details on Phoenix, LLM Tracing, and LLM Evals, checkout our [documentation](https://docs.arize.com/phoenix/)."
      ]
    }
  ],
  "metadata": {
    "language_info": {
      "name": "python"
    },
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}