tutorials/llm_summarization_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RlUREBQZeACX"
   },
   "source": [
    "<center>\n",
    "    <p style=\"text-align:center\">\n",
    "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
    "        <br>\n",
    "        <a href=\"https://docs.arize.com/phoenix/\">Docs</a>\n",
    "        |\n",
    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
    "        |\n",
    "        <a href=\"https://join.slack.com/t/arize-ai/shared_invite/zt-1px8dcmlf-fmThhDFD_V_48oU7ALan4Q\">Community</a>\n",
    "    </p>\n",
    "</center>\n",
    "<h1 align=\"center\">Troubleshooting an LLM Summarization Task</h1>\n",
    "\n",
    "Imagine you're responsible for your media company's summarization model that condenses daily news into concise summaries. Your model's performance has recently declined, leading to negative feedback from readers around the globe.\n",
    "\n",
    "Phoenix helps you find the root-cause of LLM performance issues by analyzing prompt-response pairs.\n",
    "\n",
    "In this tutorial, you will:\n",
    "\n",
    "- Download curated LLM data for this walkthrough\n",
    "- Compute embeddings for each prompt (article) and response (summary)\n",
    "- Calculate ROUGE-L scores to evaluate the quality of your LLM-generated summaries against human-written reference summaries\n",
    "- Use Phoenix to find articles that your LLM is struggling to summarize\n",
    "\n",
    "⚠️ This tutorial runs faster with a GPU.\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "## Install Dependencies and Import Libraries\n",
    "\n",
    "Install Phoenix and the Arize SDK, which provides convenience methods for extracting embeddings and computing LLM evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"arize-phoenix>=3.20.0\" \"arize[AutoEmbeddings, LLM_Evaluation]\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QPVgTptWeACa"
   },
   "source": [
    "Import libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import phoenix as px\n",
    "from arize.pandas.embeddings import EmbeddingGenerator, UseCases\n",
    "from arize.pandas.generative.llm_evaluation import rouge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EmXtLNOxeACa"
   },
   "source": [
    "## Download the Data\n",
    "\n",
    "Download your production data. Split it into two dataframes, one containing older baseline data and the other containing the most recent data. Inspect a few rows of your baseline data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_parquet(\n",
    "    \"http://storage.googleapis.com/arize-phoenix-assets/datasets/unstructured/llm/summarization/llm_summarization.parquet\"\n",
    ")\n",
    "baseline_df = df[:300]\n",
    "recent_df = df[300:]\n",
    "baseline_df = baseline_df.reset_index(\n",
    "    drop=True\n",
    ")  # recommended when using EmbeddingGenerator.generate_embeddings\n",
    "recent_df = recent_df.reset_index(\n",
    "    drop=True\n",
    ")  # recommended when using EmbeddingGenerator.generate_embeddings\n",
    "baseline_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NIqnIdbbeACa"
   },
   "source": [
    "The columns of the dataframe are:\n",
    "\n",
    "- **prediction_timestamp:**\n",
    "- **article:** the news article to be summarized\n",
    "- **summary:** the LLM-generated summary created using the prompt template: \"Please summarize the following document in English: {article}\"\n",
    "- **reference_summary:** the reference summary written by a human and used to compute ROUGE score\n",
    "\n",
    "## Compute LLM Evaluation Metrics\n",
    "\n",
    "Compute ROUGE-L scores to compare the LLM-generated summary with the human-written reference summary. A high ROUGE-L score mean that the LLM's summary closely matches the human reference summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_rougeL_scores(df: pd.DataFrame) -> pd.Series:\n",
    "    return rouge(\n",
    "        response_col=df[\"summary\"],\n",
    "        references_col=df[\"reference_summary\"],\n",
    "        rouge_types=[\"rougeL\"],\n",
    "    )[\"rougeL\"]\n",
    "\n",
    "\n",
    "baseline_df[\"rougeL_score\"] = compute_rougeL_scores(baseline_df)\n",
    "recent_df[\"rougeL_score\"] = compute_rougeL_scores(recent_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9YRnupWReqOl"
   },
   "source": [
    "## Compute Embeddings for Prompts and Responses\n",
    "\n",
    "Compute embeddings for articles and summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generator = EmbeddingGenerator.from_use_case(\n",
    "    use_case=UseCases.NLP.SUMMARIZATION,\n",
    "    model_name=\"distilbert-base-uncased\",\n",
    ")\n",
    "baseline_df[\"article_vector\"] = generator.generate_embeddings(text_col=baseline_df[\"article\"])\n",
    "baseline_df[\"summary_vector\"] = generator.generate_embeddings(text_col=baseline_df[\"summary\"])\n",
    "recent_df[\"article_vector\"] = generator.generate_embeddings(text_col=recent_df[\"article\"])\n",
    "recent_df[\"summary_vector\"] = generator.generate_embeddings(text_col=recent_df[\"summary\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xHKW34QUeACc"
   },
   "source": [
    "## Launch Phoenix\n",
    "\n",
    "Define a schema to tell Phoenix what the columns of your dataframe represent (tags, prompts, responses, etc.). See the [docs](https://docs.arize.com/phoenix/) for guides on how to define your own schema and API reference on `phoenix.Schema` and `phoenix.EmbeddingColumnNames`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = px.Schema(\n",
    "    timestamp_column_name=\"prediction_timestamp\",\n",
    "    tag_column_names=[\n",
    "        \"rougeL_score\",\n",
    "        \"reference_summary\",\n",
    "    ],\n",
    "    prompt_column_names=px.EmbeddingColumnNames(\n",
    "        vector_column_name=\"article_vector\", raw_data_column_name=\"article\"\n",
    "    ),\n",
    "    response_column_names=px.EmbeddingColumnNames(\n",
    "        vector_column_name=\"summary_vector\", raw_data_column_name=\"summary\"\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fR59QDEXeACc"
   },
   "source": [
    "Create Phoenix datasets that wrap your dataframes with schemas that describe them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_ds = px.Inferences(dataframe=baseline_df, schema=schema, name=\"baseline\")\n",
    "recent_ds = px.Inferences(dataframe=recent_df, schema=schema, name=\"recent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OfyuDyEVeACc"
   },
   "source": [
    "Launch Phoenix. Follow the instructions in the cell output to open the Phoenix UI in the notebook or in a new browser tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = px.launch_app(primary=recent_ds, reference=baseline_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_RJq17-xeACc"
   },
   "source": [
    "## Find the Root-Cause of Your Model Performance Issue\n",
    "\n",
    "Use Phoenix to find the root-cause of your LLM's performance issue.\n",
    "\n",
    "Click on \"article_vector\" to go to the embeddings view for your prompts (the input news articles).\n",
    "\n",
    "![click on article vector](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llm_summarization/click_on_article_vector.png)\n",
    "\n",
    "Select a period of high drift.\n",
    "\n",
    "![select period of high drift](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llm_summarization/select_period_of_high_drift.png)\n",
    "\n",
    "Color your data by the \"rougeL_score\" dimension. The problematic clusters have low ROUGE-L score in blue, the well-performing clusters have high ROUGE-L score in green.\n",
    "\n",
    "![color by rouge score](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llm_summarization/color_by_rouge_score.png)\n",
    "\n",
    "Use the lasso to select part of your data and inspect the prompt-response pairs.\n",
    "\n",
    "![select points with lasso](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llm_summarization/select_points_with_lasso.png)\n",
    "\n",
    "Select each clusters in the left panel and look at the prompt-response pairs. Notice that the LLM is doing a good job summarizing the English articles in the green cluster (high ROUGE-L score), but is struggling to summarize Dutch articles in the blue cluster (low ROUGE-L score).\n",
    "\n",
    "![select clusters](http://storage.googleapis.com/arize-phoenix-assets/assets/docs/notebooks/llm_summarization/select_clusters.png)\n",
    "\n",
    "Congrats! You've discovered that your LLM is struggling to summarize Dutch news articles. You should modify your prompt template to see if you can improve your ROUGE-L score for Dutch articles."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}