notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_check_alignment_against_human_preference_data.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# AutoSxS: Check autorater alignment against a human-preference dataset\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "\n",
        "  <td>\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_check_alignment_against_human_preference_data.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_check_alignment_against_human_preference_data.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
        "      View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/model_evaluation/model_based_llm_evaluation/autosxs_check_alignment_against_human_preference_data.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
        "      Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>                                                                                               \n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to use Vertex AI automatic side-by-side (AutoSxS) to check how well the autorater aligns with the human rater.\n",
        "\n",
        "Automatic side-by-side (AutoSxS) is a model-assisted evaluation tool that helps you compare two large language models (LLMs) side by side. As part of AutoSxS's preview release, we only support comparing models for summarization and question answering tasks. We will support more tasks and customization in the future.\n",
        "\n",
        "Learn more about [Vertex AI AutoSxS Model Evaluation](https://cloud.google.com/vertex-ai/docs/generative-ai/models/side-by-side-eval#autosxs)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you learn how to use `Vertex AI Pipelines` and `google_cloud_pipeline_components` to check autorater alignment using human-preference data:\n",
        "\n",
        "This tutorial uses the following Google Cloud ML services and resources:\n",
        "\n",
        "- Cloud Storage\n",
        "- Vertex AI PaLM API\n",
        "- Vertex AI Pipelines\n",
        "- Vertex AI Batch Prediction\n",
        "\n",
        "\n",
        "The steps performed include:\n",
        "- Create a evaluation dataset with predictions and human preference data.\n",
        "- Preprocess the data locally and save it in Cloud Storage.\n",
        "- Create and run a Vertex AI AutoSxS Pipeline that generates the judgments and a set of AutoSxS metrics using the generated judgments.\n",
        "- Print the judgments and AutoSxS metrics.\n",
        "- Clean up the resources created in this notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7EUnXsZhAGF"
      },
      "source": [
        "## Installation\n",
        "\n",
        "Install the following packages required to execute this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "HpCDDsBC5eip"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade \\\n",
        "    google-cloud-aiplatform \\\n",
        "    google-cloud-pipeline-components==2.13.0 \\\n",
        "    gcsfs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58707a750154"
      },
      "source": [
        "### Colab only: Uncomment the following cell to restart the kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f200f10a1da3"
      },
      "outputs": [],
      "source": [
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BF1j6f9HApxa"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n",
        "\n",
        "3. [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WReHDGG5g0XY"
      },
      "source": [
        "#### Set your project ID\n",
        "\n",
        "**If you don't know your project ID**, try the following:\n",
        "* Run `gcloud config list`.\n",
        "* Run `gcloud projects list`.\n",
        "* See the support page: [Locate the project ID](https://support.google.com/googleapi/answer/7014113)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oM1iC_MfAts1"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the project id\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "region"
      },
      "source": [
        "#### Region\n",
        "\n",
        "You may change the `REGION` variable, which is used for operations\n",
        "throughout the rest of this notebook.  Below are regions supported for AutoSxS.\n",
        "\n",
        "- Americas: `us-central1`\n",
        "- Europe: `europe-west4`\n",
        "- Asia Pacific: `asia-southeast1`\n",
        "\n",
        "You can also change the `REGION` variable used by Vertex AI. Learn more about [Vertex AI regions](https://cloud.google.com/vertex-ai/docs/general/locations)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "veaqVTqxIxGy"
      },
      "outputs": [],
      "source": [
        "REGION = \"us-central1\"  # @param {type: \"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sBCra4QMA2wR"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74ccc9e52986"
      },
      "source": [
        "**1. Vertex AI Workbench**\n",
        "* Do nothing as you are already authenticated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "de775a3773ba"
      },
      "source": [
        "**2. Local JupyterLab instance, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "254614fa0c46"
      },
      "outputs": [],
      "source": [
        "# ! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef21552ccea8"
      },
      "source": [
        "**3. Colab, uncomment and run:**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "603adbbf0532"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f6b2ccc891ed"
      },
      "source": [
        "**4. Service account or other**\n",
        "* See how to grant Cloud Storage permissions to your service account at https://cloud.google.com/storage/docs/gsutil/commands/iam#ch-examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1jikHIo07CF"
      },
      "source": [
        "### UUID\n",
        "\n",
        "We define a UUID generation function to avoid resource name collisions on resources created within the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mPKCYjn_0_3c"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "\n",
        "def generate_uuid(length: int = 8) -> str:\n",
        "    \"\"\"Generate a uuid of a specified length (default=8).\"\"\"\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))\n",
        "\n",
        "\n",
        "UUID = generate_uuid()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store intermediate artifacts to the AutoSxS pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_URI = \"gs://[your-bucket-name-unique]\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**Only if your bucket doesn't already exist:** Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "U8ZIu6UBVVRE"
      },
      "outputs": [],
      "source": [
        "if (\n",
        "    BUCKET_URI == \"\"\n",
        "    or BUCKET_URI is None\n",
        "    or BUCKET_URI == \"gs://[your-bucket-name-unique]\"\n",
        "):\n",
        "    BUCKET_URI = \"gs://\" + PROJECT_ID + \"-aip-\" + UUID\n",
        "\n",
        "! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "poijnGfZCFYi"
      },
      "source": [
        "Import the Vertex AI Python SDK and other required Python libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "import pandas as pd\n",
        "from google.cloud import aiplatform\n",
        "from google_cloud_pipeline_components.preview import model_evaluation\n",
        "from kfp import compiler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "init_aip:mbsdk,all"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python\n",
        "\n",
        "Initialize the Vertex SDK for Python for your project and corresponding bucket.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AFm7t-wmIxGz"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljpNHjC9NYtG"
      },
      "source": [
        "## Tutorial\n",
        "It is unlikely that the autorater will perform at the same level as human raters in all customer use cases, especially in cases where human raters are expected to have specialized knowledge.\n",
        "\n",
        "The tutorial below shows how AutoSxS helps to determine if you can trust the autorater once you have the ground-truth human-preference data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eha2l9nkNxZs"
      },
      "source": [
        "### Generate evaluation dataset for AutoSxS human alignment checking\n",
        "\n",
        "Below you create your dataset, specifying the set of prompts, predictions from two models and the human-preference data.\n",
        "\n",
        "In this notebook, we:\n",
        "- Create a evaluation dataset with 10 examples for AutoSxS.\n",
        "  - Data in column `prompt` will be treated as model prompts.\n",
        "  - Data in column `pred_a` will be treated as responses for model A.\n",
        "  - Data in column `pred_b` will be treated as responses for model B.\n",
        "  - Data in column `actuals` will be treated as the human-preference data.\n",
        "- Store it as JSON file in Cloud Storage.\n",
        "\n",
        "#### **Note: For best results we recommend using at least 100 examples. There are diminishing returns past 400 examples.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "_hvNRQRTN3gb"
      },
      "outputs": [],
      "source": [
        "# Define context, questions, predictions and human preference data.\n",
        "context = [\n",
        "    \"Beginning in the late 1910s and early 1920s, Whitehead gradually turned his attention from mathematics to philosophy of science, and finally to metaphysics. He developed a comprehensive metaphysical system which radically departed from most of western philosophy. Whitehead argued that reality consists of processes rather than material objects, and that processes are best defined by their relations with other processes, thus rejecting the theory that reality is fundamentally constructed by bits of matter that exist independently of one another. Today Whitehead's philosophical works – particularly Process and Reality – are regarded as the foundational texts of process philosophy.\",\n",
        "    \"The gills have an adnate attachment to the cap, are narrow to moderately broad, closely spaced, and eventually separate from the stem. Young gills are cinnamon-brown in color, with lighter edges, but darken in maturity because they become covered with the dark spores. The stem is 6 to 8 cm (2+3⁄8 to 3+1⁄8 in) long by 1.5 to 2 mm (1⁄16 to 3⁄32 in) thick, and roughly equal in width throughout except for a slightly enlarged base. The lower region of the stem is brownish in color and has silky 'hairs' pressed against the stem; the upper region is grayish and pruinose (lightly dusted with powdery white granules). The flesh turns slightly bluish or greenish where it has been injured. The application of a drop of dilute potassium hydroxide solution on the cap or flesh will cause a color change to pale to dark yellowish to reddish brown; a drop on the stem produces a less intense or no color change.\",\n",
        "    \"Go to Device Support. Choose your device. Scroll to Getting started and select Hardware &amp; phone details. Choose Insert or remove SIM card and follow the steps. Review the Account Summary page for details. Image 13 Activate online Go to att.com/activateprepaid ((att.com/activarprepaid for Spanish)) and follow the prompts. Activate over the phone Call us at 877.426.0525 for automated instructions. You will need to know your SIM/eSIM ICCID &amp; IMEI number for activation. Note: Look for your SIM (( ICCID )) number on your box or SIM card Now youre ready to activate your phone 1. Start with your new device powered off. 2. To activate a new line of service or a replacement device, please go to the AT&amp;T Activation site or call 866.895.1099. You download the eSIM to your device over Wi-Fi®. The eSIM connects your device to our wireless network. How do I activate my phone with an eSIM? Turn your phone on, connect to Wi-Fi, and follow the prompts. Swap active SIM cards AT&amp;T Wireless SM SIM Card Turn your device off. Remove the old SIM card. Insert the new one. Turn on your device.\",\n",
        "    \"According to chief astronaut Deke Slayton's autobiography, he chose Bassett for Gemini 9 because he was 'strong enough to carry' both himself and See. Slayton had also assigned Bassett as command module pilot for the second backup Apollo crew, alongside Frank Borman and William Anders.\",\n",
        "    \"Adaptation of the endosymbiont to the host's lifestyle leads to many changes in the endosymbiont–the foremost being drastic reduction in its genome size. This is due to many genes being lost during the process of metabolism, and DNA repair and recombination. While important genes participating in the DNA to RNA transcription, protein translation and DNA/RNA replication are retained. That is, a decrease in genome size is due to loss of protein coding genes and not due to lessening of inter-genic regions or open reading frame (ORF) size. Thus, species that are naturally evolving and contain reduced sizes of genes can be accounted for an increased number of noticeable differences between them, thereby leading to changes in their evolutionary rates. As the endosymbiotic bacteria related with these insects are passed on to the offspring strictly via vertical genetic transmission, intracellular bacteria goes through many hurdles during the process, resulting in the decrease in effective population sizes when compared to the free living bacteria. This incapability of the endosymbiotic bacteria to reinstate its wild type phenotype via a recombination process is called as Muller's ratchet phenomenon. Muller's ratchet phenomenon together with less effective population sizes has led to an accretion of deleterious mutations in the non-essential genes of the intracellular bacteria. This could have been due to lack of selection mechanisms prevailing in the rich environment of the host.\",\n",
        "    \"The National Archives Building in downtown Washington holds record collections such as all existing federal census records, ships' passenger lists, military unit records from the American Revolution to the Philippine–American War, records of the Confederate government, the Freedmen's Bureau records, and pension and land records.\",\n",
        "    \"Standard 35mm photographic film used for cinema projection has a much higher image resolution than HDTV systems, and is exposed and projected at a rate of 24 frames per second (frame/s). To be shown on standard television, in PAL-system countries, cinema film is scanned at the TV rate of 25 frame/s, causing a speedup of 4.1 percent, which is generally considered acceptable. In NTSC-system countries, the TV scan rate of 30 frame/s would cause a perceptible speedup if the same were attempted, and the necessary correction is performed by a technique called 3:2 Pulldown: Over each successive pair of film frames, one is held for three video fields (1/20 of a second) and the next is held for two video fields (1/30 of a second), giving a total time for the two frames of 1/12 of a second and thus achieving the correct average film frame rate.\",\n",
        "    \"Maria Deraismes was initiated into Freemasonry in 1882, then resigned to allow her lodge to rejoin their Grand Lodge. Having failed to achieve acceptance from any masonic governing body, she and Georges Martin started a mixed masonic lodge that actually worked masonic ritual. Annie Besant spread the phenomenon to the English speaking world. Disagreements over ritual led to the formation of exclusively female bodies of Freemasons in England, which spread to other countries. Meanwhile, the French had re-invented Adoption as an all-female lodge in 1901, only to cast it aside again in 1935. The lodges, however, continued to meet, which gave rise, in 1959, to a body of women practising continental Freemasonry.\",\n",
        "    \"Excavation of the foundations began in November 1906, with an average of 275 workers during the day shift and 100 workers during the night shift. The excavation was required to be completed in 120 days. To remove the spoils from the foundation, three temporary wooden platforms were constructed to street level. Hoisting engines were installed to place the beams for the foundation, while the piers were sunk into the ground under their own weight. Because of the lack of space in the area, the contractors' offices were housed beneath the temporary platforms. During the process of excavation, the Gilsey Building's foundations were underpinned or shored up, because that building had relatively shallow foundations descending only 18 feet (5.5 m) below Broadway.\",\n",
        "    \"Dopamine consumed in food cannot act on the brain, because it cannot cross the blood–brain barrier. However, there are also a variety of plants that contain L-DOPA, the metabolic precursor of dopamine. The highest concentrations are found in the leaves and bean pods of plants of the genus Mucuna, especially in Mucuna pruriens (velvet beans), which have been used as a source for L-DOPA as a drug. Another plant containing substantial amounts of L-DOPA is Vicia faba, the plant that produces fava beans (also known as 'broad beans'). The level of L-DOPA in the beans, however, is much lower than in the pod shells and other parts of the plant. The seeds of Cassia and Bauhinia trees also contain substantial amounts of L-DOPA.\",\n",
        "]\n",
        "\n",
        "questions = [\n",
        "    \"What was the predominant theory of reality that Whitehead opposed?\",\n",
        "    \"Why do the gills on the Psilocybe pelliculosa mushroom darken as they mature?\",\n",
        "    \"user: How do I provision my AT&T SIM card?\",\n",
        "    \"Why did chief astronaut Deke Slayton choose Charles Bassett for Gemini 9, according to Slayton's autobiography?\",\n",
        "    \"What is the main alteration in an endosymbiont when it adapts to a host?\",\n",
        "    \"What's the earliest war The National Archives Building has military unit records for\",\n",
        "    \"To be shown on SDTV in PAL-system countries, at what rate is cinema film scanned?\",\n",
        "    \"What year was the all-female masonic lodge cast aside?\",\n",
        "    \"Why did the Gilsey Building have underpinned and shored up foundations?\",\n",
        "    \"Why can dopamine consumed in food not act on the brain?\",\n",
        "]\n",
        "predictions_a = [\n",
        "    \"bits of matter that exist independently of one another\",\n",
        "    \"The gills darken in maturity because they become covered with the dark spores.\",\n",
        "    \"Go to Device Support. Choose your device. Scroll to Getting started and select Hardware &amp; phone details. Choose Insert or remove SIM card and follow the steps.\",\n",
        "    \"he was 'smart enough to carry' both himself and See\",\n",
        "    \"drastic reduction in its genome size\",\n",
        "    \"American Revolution to the Philippine–American War\",\n",
        "    \"Cinema film is scanned at the TV rate of 25 frame/s.\",\n",
        "    \"1935\",\n",
        "    \"The Gilsey Building's foundations were shored up because they were only 18 feet below Broadway.\",\n",
        "    \"The blood–brain barrier does not allow dopamine consumed in food to enter the brain.\",\n",
        "]\n",
        "predictions_b = [\n",
        "    \"independent bits of matter\",\n",
        "    \"Young gills are cinnamon-brown in color, with lighter edges, but darken in maturity because they become covered with the dark spores.\",\n",
        "    \"Go to Device Support.\",\n",
        "    \"he was 'strong enough to carry' both himself and See, as stated by chief astronaut Deke Slayton in his autobiography\",\n",
        "    \"its genome size decrease\",\n",
        "    \"American Revolution\",\n",
        "    \"25 frame/s, causing a speedup of 4.1 percent\",\n",
        "    \"1901\",\n",
        "    \"The Gilsey Building's foundations were underpinned or shored up.\",\n",
        "    \"Mucuna pruriens (velvet beans) have been used as a source for L-DOPA as a drug. Another plant containing substantial amounts of L-DOPA is Vicia faba, the plant that produces fava beans (also known as 'broad beans').\",\n",
        "]\n",
        "\n",
        "human_preference = [\n",
        "    \"A\",\n",
        "    \"B\",\n",
        "    \"A\",\n",
        "    \"B\",\n",
        "    \"A\",\n",
        "    \"A\",\n",
        "    \"B\",\n",
        "    \"A\",\n",
        "    \"A\",\n",
        "    \"A\",\n",
        "]\n",
        "\n",
        "# Create the evaluation dataset with context, questions, predictions and human preference data.\n",
        "examples = pd.DataFrame(\n",
        "    {\n",
        "        \"context\": context,\n",
        "        \"questions\": questions,\n",
        "        \"pred_a\": predictions_a,\n",
        "        \"pred_b\": predictions_b,\n",
        "        \"actuals\": human_preference,\n",
        "    }\n",
        ")\n",
        "examples.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v75OwRumW_co"
      },
      "source": [
        "#### [Optional] Load your JSONL evaluation dataset from Cloud Storage.\n",
        "\n",
        "Alternatively, you can load your own JSONL dataset from Cloud Storage.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O1QmcxdVZr4M"
      },
      "outputs": [],
      "source": [
        "# # Uncomment to read from Cloud Storage.\n",
        "# GCS_PATH = 'gs://your-own-evaluation-dataset-with-human-preference-data.jsonl'\n",
        "# preds = pd.read_json(GCS_PATH, lines=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NY1Jsj4aOCe1"
      },
      "source": [
        "#### Upload your dataset to Cloud Storage\n",
        "\n",
        "Finally, we upload our evaluation dataset to Cloud Storage to be used as input for AutoSxS."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "vykmkhp-ODKg"
      },
      "outputs": [],
      "source": [
        "# Upload predictions to the Cloud Storage bucket.\n",
        "examples.to_json(\n",
        "    \"evaluation_dataset_with_human_preference.json\", orient=\"records\", lines=True\n",
        ")\n",
        "! gsutil cp evaluation_dataset_with_human_preference.json $BUCKET_URI/input/evaluation_dataset_with_human_preference.json\n",
        "DATASET = f\"{BUCKET_URI}/input/evaluation_dataset_with_human_preference.json\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qgdk_qNIOFik"
      },
      "source": [
        "### Create and run AutoSxS job\n",
        "\n",
        "In order to run AutoSxS, we need to define a `autosxs_pipeline` job with the following parameters. More details of the AutoSxS pipeline configuration can be found [here](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-2.9.0/api/preview/model_evaluation.html#preview.model_evaluation.autosxs_pipeline).\n",
        "\n",
        "**Required Parameters:**\n",
        "  - **evaluation_dataset:** A list of Cloud Storage paths to a JSONL dataset containing\n",
        "      evaluation examples.\n",
        "  - **task:** Evaluation task in the form {task}@{version}. task can be one of\n",
        "      \"summarization\", \"question_answering\". Version is an integer with 3 digits or\n",
        "      \"latest\". Ex: summarization@001 or question_answering@latest.\n",
        "  - **id_columns:** The columns which distinguish unique evaluation examples.\n",
        "  - **autorater_prompt_parameters:** Map of autorater prompt parameters to columns\n",
        "      or templates. The expected parameters are:\n",
        "      - inference_instruction - Details\n",
        "      on how to perform a task.\n",
        "      - inference_context - Content to reference to\n",
        "      perform the task.\n",
        "\n",
        "Additionally, we need to specify where the predictions for the candidate models (Model A and Model B) are coming from. AutoSxS can either run Vertex Batch Prediction to get predictions, or a predefined predictions column can be provided in the evaluation dataset.\n",
        "\n",
        "**Model Parameters if using Batch Prediction (assuming Model A):**\n",
        "  - **model_a:** A fully-qualified model resource name. This parameter is optional\n",
        "      if Model A responses are specified.\n",
        "  - **model_a_prompt_parameters:** Map of Model A prompt template parameters to\n",
        "      columns or templates. In the case of [text-bison](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text#request_body), the only parameter needed is `prompt`.\n",
        "  - **model_a_parameters:** The parameters that govern the predictions from model A such as the model temperature.\n",
        "\n",
        "**Model Parameters if bringing your own predictions (assuming Model A):**\n",
        "  - **response_column_a:** The column containing responses for model A. Required if\n",
        "      any response tables are provided for model A.\n",
        "\n",
        "Lastly, there are parameters that configure additional features such as exporting the judgments or comparing judgments to a human-preference dataset to check the AutoRater's alignment with human raters.\n",
        "  - **judgments_format:** The format to write judgments to. Can be either 'json' or\n",
        "      'bigquery'.\n",
        "  - **bigquery_destination_prefix:** BigQuery table to write judgments to if the\n",
        "      specified format is 'bigquery'.\n",
        "  - **human_preference_column:** The column containing ground truths. Only required\n",
        "      when users want to check the autorater alignment against human preference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "veq26QZ7OMoC"
      },
      "source": [
        "In this notebook, we will evaluate how well the autorater aligns with the human rater using two model's predictions (located in the `pred_a` column and `pred_b` column of `PREDS` dataset) and the human preference data (located in the `actuals` column of `PREDS` dataset). The task being performed is question answering.\n",
        "\n",
        "First, compile the AutoSxS pipeline locally."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C2NGZzOMOJPV"
      },
      "outputs": [],
      "source": [
        "template_uri = \"pipeline.yaml\"\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=model_evaluation.autosxs_pipeline,\n",
        "    package_path=template_uri,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0aMBhoqOTXF"
      },
      "source": [
        "The following code starts a Vertex Pipeline job, viewable from the Vertex UI. This pipeline job will take ~10 mins.\n",
        "\n",
        "The logs here will include to the URL to the current pipeline, so you can follow the pipline progress and access/view pipeline outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "tRdA3ovUOV6j"
      },
      "outputs": [],
      "source": [
        "display_name = f\"autosxs-question-answering-human-alignment-checking-{generate_uuid()}\"\n",
        "context_column = \"context\"\n",
        "question_column = \"questions\"\n",
        "response_column_a = \"pred_a\"\n",
        "response_column_b = \"pred_b\"\n",
        "human_preference_column = \"actuals\"\n",
        "parameters = {\n",
        "    \"evaluation_dataset\": DATASET,\n",
        "    \"id_columns\": [question_column],\n",
        "    \"autorater_prompt_parameters\": {\n",
        "        \"inference_context\": {\"column\": context_column},\n",
        "        \"inference_instruction\": {\"column\": question_column},\n",
        "    },\n",
        "    \"task\": \"question_answering\",\n",
        "    \"response_column_a\": response_column_a,\n",
        "    \"response_column_b\": response_column_b,\n",
        "    \"human_preference_column\": human_preference_column,\n",
        "}\n",
        "\n",
        "job = aiplatform.PipelineJob(\n",
        "    job_id=display_name,\n",
        "    display_name=display_name,\n",
        "    pipeline_root=os.path.join(BUCKET_URI, display_name),\n",
        "    template_path=template_uri,\n",
        "    parameter_values=parameters,\n",
        "    enable_caching=False,\n",
        ")\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EinPbr3XOYPQ"
      },
      "source": [
        "### Get the judgments and AutoSxS metrics\n",
        "Next, we can load judgments from the completed AutoSxS job.\n",
        "\n",
        "The results are written to the Cloud Storage output bucket you specified in the AutoSxS job request."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "V_9yMfhrOZDk"
      },
      "outputs": [],
      "source": [
        "# To use an existing pipeline, override job using the line below.\n",
        "# job = aiplatform.PipelineJob.get('projects/[PROJECT_NUMBER]/locations/[REGION]/pipelineJobs/[PIPELINE_RUN_NAME]')\n",
        "\n",
        "for details in job.task_details:\n",
        "    if details.task_name == \"online-evaluation-pairwise\":\n",
        "        break\n",
        "\n",
        "# Judgments\n",
        "judgments_uri = details.outputs[\"judgments\"].artifacts[0].uri\n",
        "judgments_df = pd.read_json(judgments_uri, lines=True)\n",
        "judgments_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SiIVQPoA4ckL"
      },
      "source": [
        "If any example failed to get the result in AutoSxS, their error messages will be stored in an error table. If the error table is empty, it implies there's no failed examples during the evaluation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "04o7lBQl4WZd"
      },
      "outputs": [],
      "source": [
        "for details in job.task_details:\n",
        "    if details.task_name == \"online-evaluation-pairwise\":\n",
        "        break\n",
        "\n",
        "# Error table\n",
        "error_messages_uri = details.outputs[\"error_messages\"].artifacts[0].uri\n",
        "errors_df = pd.read_json(error_messages_uri, lines=True)\n",
        "errors_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlKXu5Ze4tD3"
      },
      "source": [
        "We can also look at AutoSxS metrics computed from the judgments.\n",
        "\n",
        "In the case of human-preference data been provided, AutoSxS outputs the win rate from the AutoRater and a set of human-preference alignment metrics. You can find more details of AutoSxS metrics [here](https://cloud.google.com/vertex-ai/docs/generative-ai/models/side-by-side-eval#human-metrics)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "G7meI2Eq4muT"
      },
      "outputs": [],
      "source": [
        "# Metrics\n",
        "for details in job.task_details:\n",
        "    if details.task_name == \"model-evaluation-text-generation-pairwise\":\n",
        "        break\n",
        "pd.DataFrame([details.outputs[\"autosxs_metrics\"].artifacts[0].metadata])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "Set `delete_bucket` to **True** to delete the Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "job.delete()\n",
        "\n",
        "# Delete Cloud Storage objects that were created\n",
        "delete_bucket = False\n",
        "if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
        "    ! gsutil -m rm -r $BUCKET_URI"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "autosxs_check_alignment_against_human_preference_data.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}