notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_llm_evaluation_for_summarization_task.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# AutoSxS: Evaluate a LLM in Vertex AI Model Registry against a third-party model\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_llm_evaluation_for_summarization_task.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_llm_evaluation_for_summarization_task.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_llm_evaluation_for_summarization_task.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>                                                                                               \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to use Vertex AI automatic side-by-side (AutoSxS) to evaluate the performance between a generative AI model in Vertex AI Model Registry and a third-party language model.\n",
        "\n",
        "AutoSxS is a model-assisted evaluation tool that helps you compare two large language models (LLMs) side by side. As part of AutoSxS's preview release, we only support comparing models for summarization and question answering tasks. We will support more tasks and customization in the future.\n",
        "\n",
        "Learn more about [Vertex AI AutoSxS Model Evaluation](https://cloud.google.com/vertex-ai/docs/generative-ai/models/side-by-side-eval#autosxs)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use `Vertex AI Pipelines` and `google_cloud_pipeline_components` to evaluate the performance between two LLM models:\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Cloud Storage\n",
        "- Vertex AI PaLM API\n",
        "- Vertex AI Pipelines\n",
        "- Vertex AI Batch Prediction\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "\n",
        "- Fetch the dataset from the public source.\n",
        "- Preprocess the data locally and save test data in Cloud Storage.\n",
        "- Create and run a Vertex AI AutoSxS Pipeline that generates the judgments and evaluates the two candidate models using the generated judgments.\n",
        "- Print the judgments and evaluation metrics.\n",
        "- Clean up the resources created in this notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08d289fa873f"
      },
      "source": [
        "### Dataset\n",
        "\n",
        "The dataset used for this tutorial is [Extreme Summarization (XSum)](https://huggingface.co/datasets/EdinburghNLP/xsum?row=2#dataset-card-for-xsum). The dataset consists of BBC articles and accompanying single sentence summaries which are typically professionally written by the author of the article. That dataset has 226,711 articles divided into training (90%, 204,045), validation (5%, 11,332), and test (5%, 11,334) sets.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdQSn1vrSzVJ"
      },
      "source": [
        "#### Citation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mVHPL1MtSwH0"
      },
      "outputs": [],
      "source": [
        "# @InProceedings{xsum-emnlp,\n",
        "#   author =      \"Shashi Narayan and Shay B. Cohen and Mirella Lapata\",\n",
        "#   title =       \"Don't Give Me the Details, Just the Summary! {T}opic-Aware Convolutional Neural Networks for Extreme Summarization\",\n",
        "#   booktitle =   \"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing \",\n",
        "#   year =        \"2018\",\n",
        "#   address =     \"Brussels, Belgium\",\n",
        "# }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HpCDDsBC5eip"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade \\\n",
        "    google-cloud-aiplatform \\\n",
        "    google-cloud-pipeline-components==2.13.0 \\\n",
        "    gcsfs \\\n",
        "    datasets \\\n",
        "    fsspec==2023.9.2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You may change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for AutoSxS.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-southeast1`\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "veaqVTqxIxGy"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74ccc9e52986"
      },
      "source": [
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de775a3773ba"
      },
      "source": [
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "254614fa0c46"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef21552ccea8"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "603adbbf0532"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6b2ccc891ed"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1jikHIo07CF"
      },
      "source": [
        "### UUID\n",
        "\n",
        "We define a UUID generation function to avoid resource name collisions on resources created within the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mPKCYjn_0_3c"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "def generate_uuid(length: int = 8) -> str:\n",
        "    \"\"\"Generate a uuid of a specified length (default=8).\"\"\"\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts to the AutoSxS pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = \"gs://[your-bucket-name-unique]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "Create your Cloud Storage bucket if it doesn't already exist."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "if (\n",
        "    BUCKET_URI == \"\"\n",
        "    or BUCKET_URI is None\n",
        "    or BUCKET_URI == \"gs://[your-bucket-name-unique]\"\n",
        "):\n",
        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"-aip-\" + UUID\n",
        "\n",
        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poijnGfZCFYi"
      },
      "source": [
        "Import the Vertex AI Python SDK and other required Python libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "import pandas as pd\n",
        "from google.cloud import aiplatform\n",
        "from google_cloud_pipeline_components.preview import model_evaluation\n",
        "from kfp import compiler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex SDK for Python for your project and corresponding bucket.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AFm7t-wmIxGz"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljpNHjC9NYtG"
      },
      "source": [
        "## Tutorial"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eha2l9nkNxZs"
      },
      "source": [
        "### Generate evaluation dataset for AutoSxS\n",
        "\n",
        "Below you create your dataset, specifying the set of prompts to evaluate on.\n",
        "\n",
        "In this notebook, we:\n",
        "- Download the Extreme Summarization (XSum) from the public resource.\n",
        "- Use 10 examples from the original dataset to create the evaluation dataset for AutoSxS.\n",
        "  - Data in column `document` will be treated as model prompts.\n",
        "  - Data in column `summary` will be treated as responses for model B, because model B is a third-party model in this notebook.\n",
        "- Store it as JSON file in Cloud Storage.\n",
        "\n",
        "#### **Note: For best results we recommend using at least 100 examples. There are diminishing returns past 400 examples.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "_hvNRQRTN3gb"
      },
      "outputs": [],
      "source": [
        "import datasets\n",
        "\n",
        "# Download the dataset.\n",
        "raw_datasets = datasets.load_dataset(\"xsum\", split=\"train\")\n",
        "\n",
        "# Fetch 10 examples from the original dataset.\n",
        "datasets_10 = raw_datasets.select(range(40, 50))\n",
        "print(\"dataset structure: \\n\", datasets_10)\n",
        "\n",
        "# Create the evaluation dataset with 10 examples.\n",
        "prompts = datasets_10[\"document\"]\n",
        "summaries = datasets_10[\"summary\"]\n",
        "examples = pd.DataFrame({\"content\": prompts, \"summary\": summaries})\n",
        "\n",
        "examples.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v75OwRumW_co"
      },
      "source": [
        "#### [Optional] Load your JSONL evaluation dataset from Cloud Storage"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v8Uab7UZN6Fv"
      },
      "source": [
        "Alternatively, you can load your own JSONL dataset from Cloud Storage."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O1QmcxdVZr4M"
      },
      "outputs": [],
      "source": [
        "# # Uncomment to read from Cloud Storage.\n",
        "# GCS_PATH = 'gs://your-own-evaluation-dataset.jsonl'\n",
        "# examples = pd.read_json(GCS_PATH, lines=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NY1Jsj4aOCe1"
      },
      "source": [
        "#### Upload your dataset to Cloud Storage\n",
        "\n",
        "Finally, we upload our evaluation dataset to Cloud Storage to be used as input for AutoSxS."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "vykmkhp-ODKg"
      },
      "outputs": [],
      "source": [
        "examples.to_json(\"evaluation_dataset.json\", orient=\"records\", lines=True)\n",
        "! gsutil cp evaluation_dataset.json $BUCKET_URI/input/evaluation_dataset.json\n",
        "DATASET = f\"{BUCKET_URI}/input/evaluation_dataset.json\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qgdk_qNIOFik"
      },
      "source": [
        "### Create and run AutoSxS job\n",
        "\n",
        "In order to run AutoSxS, we need to define a `autosxs_pipeline` job with the following parameters. More details of the AutoSxS pipeline configuration can be found [here](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-2.9.0/api/preview/model_evaluation.html#preview.model_evaluation.autosxs_pipeline).\n",
        "\n",
        "\n",
        "\n",
        "**Required Parameters:**\n",
        "  - **evaluation_dataset:** A list of Cloud Storage paths to a JSONL dataset containing\n",
        "      evaluation examples.\n",
        "  - **task:** Evaluation task in the form {task}@{version}. task can be one of\n",
        "      \"summarization\", \"question_answering\". Version is an integer with 3 digits or\n",
        "      \"latest\". Ex: summarization@001 or question_answering@latest.\n",
        "  - **id_columns:** The columns which distinguish unique evaluation examples.\n",
        "  - **autorater_prompt_parameters:** Map of autorater prompt parameters to columns\n",
        "      or templates. The expected parameters are:\n",
        "      - inference_instruction - Details\n",
        "      on how to perform a task.\n",
        "      - inference_context - Content to reference to\n",
        "      perform the task.\n",
        "\n",
        "Additionally, we need to specify where the predictions for the candidate models (Model A and Model B) come from. AutoSxS can either run Vertex Batch Prediction to get predictions, or a predefined predictions column can be provided in the evaluation dataset.\n",
        "\n",
        "**Model Parameters if using Batch Prediction (assuming Model A):**\n",
        "  - **model_a:** A fully-qualified model resource name. This parameter is optional\n",
        "      if Model A responses are specified.\n",
        "  - **model_a_prompt_parameters:** Map of Model A prompt template parameters to\n",
        "      columns or templates. In the case of [text-bison](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text#request_body), the only parameter needed is `prompt`.\n",
        "  - **model_a_parameters:** The parameters that govern the predictions from model A such as the model temperature.\n",
        "\n",
        "**Model Parameters if bringing your own predictions (assuming Model A):**\n",
        "  - **response_column_a:** The column containing responses for model A. Required if\n",
        "      any response tables are provided for model A.\n",
        "\n",
        "Lastly, there are parameters that configure additional features such as exporting the judgments or comparing judgments to a human-preference dataset to check the AutoRater's alignment with human raters.\n",
        "  - **judgments_format:** The format to write judgments to. Can be either 'json' or\n",
        "      'bigquery'.\n",
        "  - **bigquery_destination_prefix:** BigQuery table to write judgments to if the\n",
        "      specified format is 'bigquery'.\n",
        "  - **human_preference_column:** The column containing ground truths. Only required\n",
        "      when users want to check the autorater alignment against human preference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "veq26QZ7OMoC"
      },
      "source": [
        "In this notebook, we will evaluate a third-party model's predictions (located in the `summary` column of `DATASET`) against the output of `text-bison@001` using a built-in summarization instruction. The task being performed is summarization.\n",
        "\n",
        "First, compile the AutoSxS pipeline locally."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C2NGZzOMOJPV"
      },
      "outputs": [],
      "source": [
        "template_uri = \"pipeline.yaml\"\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=model_evaluation.autosxs_pipeline,\n",
        "    package_path=template_uri,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0aMBhoqOTXF"
      },
      "source": [
        "The following code starts a Vertex Pipeline job, viewable from the Vertex UI. This pipeline job will take ~15 mins.\n",
        "\n",
        "The logs here will include to the URL to the current pipeline, so you can follow the pipline progress and access/view pipeline outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "tRdA3ovUOV6j"
      },
      "outputs": [],
      "source": [
        "display_name = f\"autosxs-summarization-{generate_uuid()}\"\n",
        "prompt_column = \"content\"\n",
        "response_column_b = \"summary\"\n",
        "DATASET = f\"{BUCKET_URI}/input/evaluation_dataset.json\"\n",
        "parameters = {\n",
        "    \"evaluation_dataset\": DATASET,\n",
        "    \"id_columns\": [prompt_column],\n",
        "    \"autorater_prompt_parameters\": {\n",
        "        \"inference_context\": {\"column\": prompt_column},\n",
        "        \"inference_instruction\": {\"template\": \"{{ default_instruction }}\"},\n",
        "    },\n",
        "    \"task\": \"summarization\",\n",
        "    \"model_a\": \"publishers/google/models/text-bison@001\",\n",
        "    \"model_a_prompt_parameters\": {\n",
        "        \"prompt\": {\n",
        "            \"template\": \"{{ default_instruction }}: {{\" + prompt_column + \"}}.\",\n",
        "            # 'template': 'Summarize the following: {{' + prompt_column + \"}}.\",  - This is also okay.\n",
        "        },\n",
        "    },\n",
        "    \"response_column_b\": response_column_b,\n",
        "}\n",
        "\n",
        "job = aiplatform.PipelineJob(\n",
        "    job_id=display_name,\n",
        "    display_name=display_name,\n",
        "    pipeline_root=os.path.join(BUCKET_URI, display_name),\n",
        "    template_path=template_uri,\n",
        "    parameter_values=parameters,\n",
        "    enable_caching=False,\n",
        ")\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EinPbr3XOYPQ"
      },
      "source": [
        "### Get the judgments and AutoSxS win-rate metrics\n",
        "Next, we can load judgments from the completed AutoSxS job.\n",
        "\n",
        "The results are written to the Cloud Storage output bucket you specified in the AutoSxS job request."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "V_9yMfhrOZDk"
      },
      "outputs": [],
      "source": [
        "# To use an existing pipeline, override job using the line below.\n",
        "# job = aiplatform.PipelineJob.get('projects/[PROJECT_NUMBER]/locations/[REGION]/pipelineJobs/[PIPELINE_RUN_NAME]')\n",
        "\n",
        "for details in job.task_details:\n",
        "    if details.task_name == \"online-evaluation-pairwise\":\n",
        "        break\n",
        "\n",
        "# Judgments\n",
        "judgments_uri = details.outputs[\"judgments\"].artifacts[0].uri\n",
        "judgments_df = pd.read_json(judgments_uri, lines=True)\n",
        "judgments_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SiIVQPoA4ckL"
      },
      "source": [
        "If any example failed to get the result in AutoSxS, their error messages will be stored in an error table. If the error table is empty, it implies there's no failed examples during the evaluation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "xtP1RabVNrlT"
      },
      "outputs": [],
      "source": [
        "for details in job.task_details:\n",
        "    if details.task_name == \"online-evaluation-pairwise\":\n",
        "        break\n",
        "\n",
        "# Error table\n",
        "error_messages_uri = details.outputs[\"error_messages\"].artifacts[0].uri\n",
        "errors_df = pd.read_json(error_messages_uri, lines=True)\n",
        "errors_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlKXu5Ze4tD3"
      },
      "source": [
        "We can also look at metrics computed from the judgments. AutoSxS outputs the win rate to show how often one model outperformed another."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "G7meI2Eq4muT"
      },
      "outputs": [],
      "source": [
        "# Metrics\n",
        "for details in job.task_details:\n",
        "    if details.task_name == \"model-evaluation-text-generation-pairwise\":\n",
        "        break\n",
        "pd.DataFrame([details.outputs[\"autosxs_metrics\"].artifacts[0].metadata])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "Set `delete_bucket` to **True** to delete the Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "job.delete()\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "delete_bucket = False\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "autosxs_llm_evaluation_for_summarization_task.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}