notebooks/community/model_garden/model_garden_autosxs_evaluation_llama3_1.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Vertex AI Model Garden - Evaluate Llama 3.1 models using Vertex AI AutoSxS\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_autosxs_evaluation_llama3_1.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fmodel_garden%2Fmodel_garden_autosxs_evaluation_llama3_1.ipynb\"\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_autosxs_evaluation_llama3_1.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_autosxs_evaluation_llama3_1.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demostrates how to use the Vertex AI automatic side-by-side (AutoSxS) tool to evaluate Llama 3.1 models for a question-answering task.\n",
        "\n",
        "### Objective\n",
        "\n",
        "- Choose the Llama 3.1 models you want to compare.\n",
        "\n",
        "- Create an evaluation dataset with question-answer data.\n",
        "\n",
        "- Create and run a Vertex AI AutoSxS pipeline that generates judgments and a set of AutoSxS metrics using the generated judgments.\n",
        "\n",
        "- Print the judgments and AutoSxS metrics.\n",
        "\n",
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK for Python and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "! pip3 install --upgrade --user --quiet google-cloud-aiplatform google-cloud-pipeline-components\n",
        "! pip3 install --upgrade --user --quiet openai gcsfs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com). Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"<your-project-id>\"  # @param {type:\"string\"}\n",
        "\n",
        "# Set the region of the instance\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zgPO1eR3CYjk"
      },
      "source": [
        "### Create a Cloud Storage bucket\n",
        "\n",
        "Create a storage bucket to store tutorial artifacts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MzGDU7TWdts_"
      },
      "outputs": [],
      "source": [
        "BUCKET_NAME = \"<your-bucket-name>\"  # @param {type:\"string\"}\n",
        "\n",
        "BUCKET_URI = f\"gs://{BUCKET_NAME}\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EcIXiGsCePi"
      },
      "source": [
        "**If your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NIq7R4HZCfIc"
      },
      "outputs": [],
      "source": [
        "! gsutil mb -l {LOCATION} -p {PROJECT_ID} {BUCKET_URI}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Wn8ZkcV86KR"
      },
      "source": [
        "### Initialize Vertex AI SDK for Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B8DawN9D9NLU"
      },
      "outputs": [],
      "source": [
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION, staging_bucket=BUCKET_URI)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jVYoyDl165EE"
      },
      "source": [
        "### Import libraries\n",
        "\n",
        "Import libraries to use in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c1tEW-U968h8"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import string\n",
        "\n",
        "import openai\n",
        "import pandas as pd\n",
        "from google.auth import default, transport\n",
        "from google.cloud import aiplatform\n",
        "from google_cloud_pipeline_components.v1 import model_evaluation\n",
        "from kfp import compiler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZXnx1_CtEV5L"
      },
      "source": [
        "### Set variables\n",
        "\n",
        "Before starting, you must decide how to access Llama 3.1 models. You can access Llama 3.1 models in just a few clicks using Model-as-a-Service (MaaS) without any setup or infrastructure hassles. You can also access Llama models for self-service in Vertex AI Model Garden, allowing you to choose your preferred infrastructure.\n",
        "\n",
        "This tutorial assumes that you deploy a self-managed instance of the Llama 3.1 model and compare it with Llama 3 405b using Model-as-a-Service (MaaS). Notice, only `us-central1` is supported region for Llama 3.1 models using Model-as-a-Service (MaaS).\n",
        "\n",
        "[Check out Llama 3 model card](https://console.cloud.google.com/vertex-ai/publishers/meta/model-garden/llama3?_ga=2.31261500.2048242469.1721714335-1107467625.1721655511) to learn how to deploy a Llama 3.1 models on Vertex AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XASp0SPNEX10"
      },
      "outputs": [],
      "source": [
        "SELF_DEPLOYED_ENDPOINT_REGION = \"<your-endpoint-region>\"  # @param {type:\"string\"}\n",
        "SELF_DEPLOYED_ENDPOINT_ID = \"<your-endpoint-id>\"  # @param {type:\"string\"}\n",
        "MODEL_LOCATION = \"us-central1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_sdmrDed2aHd"
      },
      "source": [
        "### Helpers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pACHdEUf2bfq"
      },
      "outputs": [],
      "source": [
        "def self_model_generate(\n",
        "    question,\n",
        "    context,\n",
        "    endpoint_id=SELF_DEPLOYED_ENDPOINT_ID,\n",
        "    endpoint_location=SELF_DEPLOYED_ENDPOINT_REGION,\n",
        "    **model_kwargs,\n",
        "):\n",
        "    \"\"\"Generate a response from a self-managed Llama 3.1 model.\"\"\"\n",
        "\n",
        "    aiplatform.init(project=PROJECT_ID, location=endpoint_location)\n",
        "\n",
        "    prompt = \"\"\"You are an AI assistant. Your goal is to answer questions using the pieces of context. \"\"\"\n",
        "    prompt += f\"\"\"Question: {question}.\"\"\"\n",
        "    prompt += f\"\"\"Context: {context}.\"\"\"\n",
        "    prompt += \"\"\"Answer:\"\"\"\n",
        "\n",
        "    instance = {\"prompt\": prompt}\n",
        "    instance.update(model_kwargs)\n",
        "    instances = [instance]\n",
        "\n",
        "    endpoint = aiplatform.Endpoint(endpoint_id)\n",
        "    response = endpoint.predict(instances=instances)\n",
        "    return response.predictions[0][len(prompt) + 1 :]\n",
        "\n",
        "\n",
        "def maas_generate(\n",
        "    question,\n",
        "    context,\n",
        "    model=\"meta/llama3-405b-instruct-maas\",\n",
        "    model_location=MODEL_LOCATION,\n",
        "    **model_kwargs,\n",
        "):\n",
        "    \"\"\"Generate a response from a MaaS Llama 3.1 model.\"\"\"\n",
        "\n",
        "    creds, _ = default()\n",
        "    auth_req = transport.requests.Request()\n",
        "    creds.refresh(auth_req)\n",
        "    if model_kwargs is None:\n",
        "        model_kwargs = {}\n",
        "\n",
        "    client = openai.OpenAI(\n",
        "        base_url=f\"https://{model_location}-aiplatform.googleapis.com/v1beta1/projects/{PROJECT_ID}/locations/{model_location}/endpoints/openapi/chat/completions?\",\n",
        "        api_key=creds.token,\n",
        "    )\n",
        "\n",
        "    response = client.chat.completions.create(\n",
        "        model=model,\n",
        "        messages=[\n",
        "            {\n",
        "                \"role\": \"system\",\n",
        "                \"content\": \"\"\"You are an AI assistant. Your goal is to answer questions using the pieces of context. If you don't know the answer, say that you don't know.\"\"\",\n",
        "            },\n",
        "            {\"role\": \"user\", \"content\": question},\n",
        "            {\"role\": \"assistant\", \"content\": context},\n",
        "        ],\n",
        "        **model_kwargs,\n",
        "    )\n",
        "\n",
        "    return response.choices[0].message.content\n",
        "\n",
        "\n",
        "def generate_uuid(length: int = 8) -> str:\n",
        "    \"\"\"Generate a uuid of a specified length (default=8).\"\"\"\n",
        "    return \"\".join(random.choices(string.ascii_lowercase + string.digits, k=length))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eha2l9nkNxZs"
      },
      "source": [
        "### Generate evaluation dataset for AutoSxS\n",
        "\n",
        "Below you create your evaluation dataset, you specify a set of prompts to evaluate on.\n",
        "\n",
        "In this notebook, you:\n",
        "\n",
        "- Use 10 examples from the original dataset to create an evaluation dataset for AutoSxS.\n",
        "  - Data in the `contexts` column will be treated as model context.\n",
        "  - Data in the `questions` column will be treated as model instruction.\n",
        "  - Data in the `response_a` column will be treated as responses for model A.\n",
        "  - Data in the `response_b` will be treated as responses for model B.\n",
        "\n",
        "- Store the data in a JSON file in Google sCloud Storage.\n",
        "\n",
        "#### **Note: For the best results we recommend using at least 100 examples. There are diminishing returns when using more than 400 examples.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kuVd8Y7GHbp8"
      },
      "source": [
        "#### Provide context and question"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j_OuH0yh_PMe"
      },
      "outputs": [],
      "source": [
        "contexts = [\n",
        "    \"Beginning in the late 1910s and early 1920s, Whitehead gradually turned his attention from mathematics to philosophy of science, and finally to metaphysics. He developed a comprehensive metaphysical system which radically departed from most of western philosophy. Whitehead argued that reality consists of processes rather than material objects, and that processes are best defined by their relations with other processes, thus rejecting the theory that reality is fundamentally constructed by bits of matter that exist independently of one another. Today Whitehead's philosophical works – particularly Process and Reality – are regarded as the foundational texts of process philosophy.\",\n",
        "    \"The gills have an adnate attachment to the cap, are narrow to moderately broad, closely spaced, and eventually separate from the stem. Young gills are cinnamon-brown in color, with lighter edges, but darken in maturity because they become covered with the dark spores. The stem is 6 to 8 cm (2+3⁄8 to 3+1⁄8 in) long by 1.5 to 2 mm (1⁄16 to 3⁄32 in) thick, and roughly equal in width throughout except for a slightly enlarged base. The lower region of the stem is brownish in color and has silky 'hairs' pressed against the stem; the upper region is grayish and pruinose (lightly dusted with powdery white granules). The flesh turns slightly bluish or greenish where it has been injured. The application of a drop of dilute potassium hydroxide solution on the cap or flesh will cause a color change to pale to dark yellowish to reddish brown; a drop on the stem produces a less intense or no color change.\",\n",
        "    \"Go to Device Support. Choose your device. Scroll to Getting started and select Hardware & phone details. Choose Insert or remove SIM card and follow the steps. Review the Account Summary page for details. Image 13 Activate online Go to att.com/activateprepaid ((att.com/activarprepaid for Spanish)) and follow the prompts. Activate over the phone Call us at 877.426.0525 for automated instructions. You will need to know your SIM/eSIM ICCID & IMEI number for activation. Note: Look for your SIM (( ICCID )) number on your box or SIM card Now youre ready to activate your phone 1. Start with your new device powered off. 2. To activate a new line of service or a replacement device, please go to the AT&T Activation site or call 866.895.1099. You download the eSIM to your device over Wi-Fi®. The eSIM connects your device to our wireless network. How do I activate my phone with an eSIM? Turn your phone on, connect to Wi-Fi, and follow the prompts. Swap active SIM cards AT&T Wireless SM SIM Card Turn your device off. Remove the old SIM card. Insert the new one. Turn on your device.\",\n",
        "    \"According to chief astronaut Deke Slayton's autobiography, he chose Bassett for Gemini 9 because he was 'strong enough to carry' both himself and See. Slayton had also assigned Bassett as command module pilot for the second backup Apollo crew, alongside Frank Borman and William Anders.\",\n",
        "    \"Adaptation of the endosymbiont to the host's lifestyle leads to many changes in the endosymbiont–the foremost being drastic reduction in its genome size. This is due to many genes being lost during the process of metabolism, and DNA repair and recombination. While important genes participating in the DNA to RNA transcription, protein translation and DNA/RNA replication are retained. That is, a decrease in genome size is due to loss of protein coding genes and not due to lessening of inter-genic regions or open reading frame (ORF) size. Thus, species that are naturally evolving and contain reduced sizes of genes can be accounted for an increased number of noticeable differences between them, thereby leading to changes in their evolutionary rates. As the endosymbiotic bacteria related with these insects are passed on to the offspring strictly via vertical genetic transmission, intracellular bacteria goes through many hurdles during the process, resulting in the decrease in effective population sizes when compared to the free living bacteria. This incapability of the endosymbiotic bacteria to reinstate its wild type phenotype via a recombination process is called as Muller's ratchet phenomenon. Muller's ratchet phenomenon together with less effective population sizes has led to an accretion of deleterious mutations in the non-essential genes of the intracellular bacteria. This could have been due to lack of selection mechanisms prevailing in the rich environment of the host.\",\n",
        "    \"The National Archives Building in downtown Washington holds record collections such as all existing federal census records, ships' passenger lists, military unit records from the American Revolution to the Philippine–American War, records of the Confederate government, the Freedmen's Bureau records, and pension and land records.\",\n",
        "    \"Standard 35mm photographic film used for cinema projection has a much higher image resolution than HDTV systems, and is exposed and projected at a rate of 24 frames per second (frame/s). To be shown on standard television, in PAL-system countries, cinema film is scanned at the TV rate of 25 frame/s, causing a speedup of 4.1 percent, which is generally considered acceptable. In NTSC-system countries, the TV scan rate of 30 frame/s would cause a perceptible speedup if the same were attempted, and the necessary correction is performed by a technique called 3:2 Pulldown: Over each successive pair of film frames, one is held for three video fields (1/20 of a second) and the next is held for two video fields (1/30 of a second), giving a total time for the two frames of 1/12 of a second and thus achieving the correct average film frame rate.\",\n",
        "    \"Maria Deraismes was initiated into Freemasonry in 1882, then resigned to allow her lodge to rejoin their Grand Lodge. Having failed to achieve acceptance from any masonic governing body, she and Georges Martin started a mixed masonic lodge that actually worked masonic ritual. Annie Besant spread the phenomenon to the English speaking world. Disagreements over ritual led to the formation of exclusively female bodies of Freemasons in England, which spread to other countries. Meanwhile, the French had re-invented Adoption as an all-female lodge in 1901, only to cast it aside again in 1935. The lodges, however, continued to meet, which gave rise, in 1959, to a body of women practising continental Freemasonry.\",\n",
        "    \"Excavation of the foundations began in November 1906, with an average of 275 workers during the day shift and 100 workers during the night shift. The excavation was required to be completed in 120 days. To remove the spoils from the foundation, three temporary wooden platforms were constructed to street level. Hoisting engines were installed to place the beams for the foundation, while the piers were sunk into the ground under their own weight. Because of the lack of space in the area, the contractors' offices were housed beneath the temporary platforms. During the process of excavation, the Gilsey Building's foundations were underpinned or shored up, because that building had relatively shallow foundations descending only 18 feet (5.5 m) below Broadway.\",\n",
        "    \"Dopamine consumed in food cannot act on the brain, because it cannot cross the blood–brain barrier. However, there are also a variety of plants that contain L-DOPA, the metabolic precursor of dopamine. The highest concentrations are found in the leaves and bean pods of plants of the genus Mucuna, especially in Mucuna pruriens (velvet beans), which have been used as a source for L-DOPA as a drug. Another plant containing substantial amounts of L-DOPA is Vicia faba, the plant that produces fava beans (also known as 'broad beans'). The level of L-DOPA in the beans, however, is much lower than in the pod shells and other parts of the plant. The seeds of Cassia and Bauhinia trees also contain substantial amounts of L-DOPA.\",\n",
        "]\n",
        "\n",
        "questions = [\n",
        "    \"What was the predominant theory of reality that Whitehead opposed?\",\n",
        "    \"Why do the gills on the Psilocybe pelliculosa mushroom darken as they mature?\",\n",
        "    \"user: How do I provision my AT&T SIM card?\",\n",
        "    \"Why did chief astronaut Deke Slayton choose Charles Bassett for Gemini 9, according to Slayton's autobiography?\",\n",
        "    \"What is the main alteration in an endosymbiont when it adapts to a host?\",\n",
        "    \"What's the earliest war The National Archives Building has military unit records for\",\n",
        "    \"To be shown on SDTV in PAL-system countries, at what rate is cinema film scanned?\",\n",
        "    \"What year was the all-female masonic lodge cast aside?\",\n",
        "    \"Why did the Gilsey Building have underpinned and shored up foundations?\",\n",
        "    \"Why can dopamine consumed in food not act on the brain?\",\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Oiwr677h_cSk"
      },
      "outputs": [],
      "source": [
        "examples = pd.DataFrame(\n",
        "    {\n",
        "        \"questions\": questions,\n",
        "        \"context\": contexts,\n",
        "    }\n",
        ")\n",
        "examples.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2je-Rs8e_65p"
      },
      "outputs": [],
      "source": [
        "examples[\"response_a\"] = examples.apply(\n",
        "    lambda x: self_model_generate(\n",
        "        x[\"questions\"], x[\"context\"], max_tokens=2500, temperature=0.5\n",
        "    ),\n",
        "    axis=1,\n",
        ")\n",
        "examples.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VGIpgBk9Br_G"
      },
      "outputs": [],
      "source": [
        "examples[\"response_b\"] = examples.apply(\n",
        "    lambda x: maas_generate(\n",
        "        x[\"questions\"], x[\"context\"], max_tokens=2500, temperature=0.5\n",
        "    ),\n",
        "    axis=1,\n",
        ")\n",
        "examples.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NY1Jsj4aOCe1"
      },
      "source": [
        "#### Upload your dataset to Cloud Storage\n",
        "\n",
        "Finally, we upload our evaluation dataset to Cloud Storage to be used as input for AutoSxS."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vykmkhp-ODKg"
      },
      "outputs": [],
      "source": [
        "examples.to_json(f\"{BUCKET_URI}/evaluation_dataset.json\", orient=\"records\", lines=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qgdk_qNIOFik"
      },
      "source": [
        "### Create and run AutoSxS job\n",
        "\n",
        "In order to run AutoSxS, we need to define a `autosxs_pipeline` job with the following parameters.\n",
        "\n",
        "More details of the AutoSxS pipeline configuration can be found [here](https://google-cloud-pipeline-components.readthedocs.io/en/google-cloud-pipeline-components-2.9.0/api/preview/model_evaluation.html#preview.model_evaluation.autosxs_pipeline)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "veq26QZ7OMoC"
      },
      "source": [
        "First, compile the AutoSxS pipeline locally."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "C2NGZzOMOJPV"
      },
      "outputs": [],
      "source": [
        "template_uri = \"pipeline.yaml\"\n",
        "compiler.Compiler().compile(\n",
        "    pipeline_func=model_evaluation.autosxs_pipeline,\n",
        "    package_path=template_uri,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0aMBhoqOTXF"
      },
      "source": [
        "The following code starts a Vertex Pipeline job, viewable from the Vertex UI. This pipeline job will take ~15 mins. This pipeline is made for batch prediction at a much larger scale than this example, so the time won't scale up linearly if there were thousands of Q&A pairs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tRdA3ovUOV6j"
      },
      "outputs": [],
      "source": [
        "display_name = f\"autosxs-qa-{generate_uuid()}\"\n",
        "context_column = \"context\"\n",
        "question_column = \"questions\"\n",
        "response_column_a = \"response_a\"\n",
        "response_column_b = \"response_b\"\n",
        "\n",
        "parameters = {\n",
        "    \"evaluation_dataset\": BUCKET_URI + \"/evaluation_dataset.json\",\n",
        "    \"id_columns\": [question_column],\n",
        "    \"autorater_prompt_parameters\": {\n",
        "        \"inference_context\": {\"column\": context_column},\n",
        "        \"inference_instruction\": {\"column\": question_column},\n",
        "    },\n",
        "    \"task\": \"question_answering\",\n",
        "    \"response_column_a\": response_column_a,\n",
        "    \"response_column_b\": response_column_b,\n",
        "}\n",
        "\n",
        "job = aiplatform.PipelineJob(\n",
        "    job_id=display_name,\n",
        "    display_name=display_name,\n",
        "    pipeline_root=BUCKET_URI + \"/pipeline\",\n",
        "    template_path=template_uri,\n",
        "    parameter_values=parameters,\n",
        "    enable_caching=False,\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        ")\n",
        "job.run()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EinPbr3XOYPQ"
      },
      "source": [
        "### Get the judgments and AutoSxS metrics\n",
        "Next, you can review judgments from the completed AutoSxS job."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V_9yMfhrOZDk"
      },
      "outputs": [],
      "source": [
        "for details in job.task_details:\n",
        "    if details.task_name == \"online-evaluation-pairwise\":\n",
        "        break\n",
        "\n",
        "judgments_uri = details.outputs[\"judgments\"].artifacts[0].uri\n",
        "judgments_df = pd.read_json(judgments_uri, lines=True)\n",
        "judgments_df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BlKXu5Ze4tD3"
      },
      "source": [
        "You can also review AutoSxS metrics computed from the judgments.\n",
        "\n",
        "You can find more details of AutoSxS metrics [here](https://cloud.google.com/vertex-ai/generative-ai/docs/models/side-by-side-eval#aggregate-metrics)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G7meI2Eq4muT"
      },
      "outputs": [],
      "source": [
        "for details in job.task_details:\n",
        "    if details.task_name == \"model-evaluation-text-generation-pairwise\":\n",
        "        break\n",
        "pd.DataFrame([details.outputs[\"autosxs_metrics\"].artifacts[0].metadata])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TpV-iwP9qw9c"
      },
      "source": [
        "## Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial:\n",
        "\n",
        "Set `delete_bucket` to **True** to delete the Cloud Storage bucket."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sx_vKniMq9ZX"
      },
      "outputs": [],
      "source": [
        "delete_pipeline_job = False  # @param {type:\"boolean\"}\n",
        "delete_bucket = False  # @param {type:\"boolean\"}\n",
        "\n",
        "if delete_pipeline_job:\n",
        "    job.delete()\n",
        "\n",
        "if delete_bucket:\n",
        "    ! gsutil rm -r gs://{BUCKET_NAME}"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "model_garden_autosxs_evaluation_llama3_1.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}