trulens_eval/examples/expositional/models/bedrock_finetuning_experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af2ee7fb-e888-4e38-a349-c7c40dfd2963",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "# Deploy, Fine-tune Foundation Models with AWS Sagemaker, Iterate and Monitor with TruEra"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0679163a-387a-4ba6-8ce0-d5571614c0dc",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "SageMaker JumpStart provides a variety of pretrained open source and proprietary models such as Llama-2, Anthropic’s Claude and Cohere Command that can be quickly deployed in the Sagemaker environment. In many cases however, these foundation models are not sufficient on their own for production use cases, needing to be adapted to a particular style or new tasks. One way to surface this need is by evaluating the model against a curated ground truth dataset. Once the need to adapt the foundation model is clear, one could leverage a set of techniques to carry that out. A popular approach is to fine-tune the model on a dataset that is tailored to the use case.\n",
    "\n",
    "One challenge with this approach is that curated ground truth datasets are expensive to create. In this blog post, we address this challenge by augmenting this workflow with a framework for extensible, automated evaluations. We start off with a baseline foundation model from SageMaker JumpStart and evaluate it with TruLens, an open source library for evaluating & tracking LLM apps. Once we identify the need for adaptation, we can leverage fine-tuning in Sagemaker Jumpstart and confirm improvement with TruLens.\n",
    "\n",
    "TruLens evaluations make use of an abstraction of feedback functions. These functions can be implemented in several ways, including BERT-style models, appropriately prompted Large Language Models, and more. TruLens’ integration with AWS Bedrock allows you to easily run evaluations using LLMs available from AWS Bedrock. The reliability of Bedrock’s infrastructure is particularly valuable for use in performing evaluations across development and production.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "251624f9-1eb6-4051-a774-0a4ba83cabf5",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "---\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy pre-trained Llama 2 model as well as fine-tune it for your dataset in domain adaptation or instruction tuning format. We will also use TruLens to identify performance issues with the base model and validate improvement of the fine-tuned model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85addd9d-ec89-44a7-9fb5-9bc24fe9993b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! pip install trulens_eval==0.20.3 sagemaker datasets boto3 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13274b9b-87bd-4090-a6aa-294570c31e0e",
   "metadata": {},
   "source": [
    "## Deploy Pre-trained Model\n",
    "\n",
    "---\n",
    "\n",
    "First we will deploy the Llama-2 model as a SageMaker endpoint. To train/deploy 13B and 70B models, please change model_id to \"meta-textgenerated_text-llama-2-7b\" and \"meta-textgenerated_text-llama-2-70b\" respectively.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8071a2d3",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, model_version = \"meta-textgeneration-llama-2-7b\", \"*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1722b230-b7bc-487f-b4ee-98ca42848423",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "pretrained_model = JumpStartModel(model_id=model_id)\n",
    "pretrained_predictor = pretrained_model.deploy(accept_eula=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8017c4ef-eb89-4da6-8e28-c800adbfc4b8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Invoke the endpoint\n",
    "\n",
    "---\n",
    "Next, we invoke the endpoint with some sample queries. Later, in this notebook, we will fine-tune this model with a custom dataset and carry out inference using the fine-tuned model. We will also show comparison between results obtained via the pre-trained and the fine-tuned models.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b795a085-048f-42b2-945f-0cd339c1cf91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_response(payload, response):\n",
    "    print(payload[\"inputs\"])\n",
    "    print(f\"> {response[0]['generated_text']}\")\n",
    "    print(\"\\n==================================\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dd833f8-1ddc-4805-80b2-19e7db629880",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "payload = {\n",
    "    \"inputs\": \"I believe the meaning of life is\",\n",
    "    \"parameters\": {\n",
    "        \"max_new_tokens\": 64,\n",
    "        \"top_p\": 0.9,\n",
    "        \"temperature\": 0.6,\n",
    "        \"return_full_text\": False,\n",
    "    },\n",
    "}\n",
    "try:\n",
    "    response = pretrained_predictor.predict(payload, custom_attributes=\"accept_eula=true\")\n",
    "    print_response(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a6773e6-7cf2-4cea-bce6-905d5995d857",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "To learn about additional use cases of pre-trained model, please checkout the notebook [Text completion: Run Llama 2 models in SageMaker JumpStart](https://github.com/aws/amazon-sagemaker-examples/blob/main/introduction_to_amazon_algorithms/jumpstart-foundation-models/llama-2-text-completion.ipynb).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e19e16f-d459-40c6-9d6b-0272938b3878",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Dataset preparation for fine-tuning\n",
    "\n",
    "---\n",
    "\n",
    "You can fine-tune on the dataset with domain adaptation format or instruction tuning format. Please find more details in the section [Dataset instruction](#Dataset-instruction). In this demo, we will use a subset of [Dolly dataset](https://huggingface.co/datasets/databricks/databricks-dolly-15k) in an instruction tuning format. Dolly dataset contains roughly 15,000 instruction following records for various categories such as question answering, summarization, information extraction etc. It is available under Apache 2.0 license. We will select the summarization examples for fine-tuning.\n",
    "\n",
    "\n",
    "Training data is formatted in JSON lines (.jsonl) format, where each line is a dictionary representing a single data sample. All training data must be in a single folder, however it can be saved in multiple jsonl files. The training folder can also contain a template.json file describing the input and output formats.\n",
    "\n",
    "To train your model on a collection of unstructured dataset (text files), please see the section [Example fine-tuning with Domain-Adaptation dataset format](#Example-fine-tuning-with-Domain-Adaptation-dataset-format) in the Appendix.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dd20a0d-15a5-49b0-a330-a75755d046ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dolly_dataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")\n",
    "\n",
    "# To train for question answering/information extraction, you can replace the assertion in next line to example[\"category\"] == \"closed_qa\"/\"information_extraction\".\n",
    "summarization_dataset = dolly_dataset.filter(lambda example: example[\"category\"] == \"summarization\")\n",
    "summarization_dataset = summarization_dataset.remove_columns(\"category\")\n",
    "\n",
    "# We split the dataset into two where test data is used to evaluate at the end.\n",
    "train_and_test_dataset = summarization_dataset.train_test_split(test_size=0.1)\n",
    "\n",
    "# Dumping the training data to a local file to be used for training.\n",
    "train_and_test_dataset[\"train\"].to_json(\"train.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9fbf002-3ee3-4cc8-8fce-871939f1bd19",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_and_test_dataset[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b2e5489-33dc-4623-92da-f6fc97bd25ab",
   "metadata": {},
   "source": [
    "---\n",
    "Next, we create a prompt template for using the data in an instruction / input format for the training job (since we are instruction fine-tuning the model in this example), and also for inferencing the deployed endpoint.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90451114-7cf5-445c-88e3-02ccaa5d3a4b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "template = {\n",
    "    \"prompt\": \"Below is an instruction that describes a task, paired with an input that provides further context. \"\n",
    "    \"Write a response that appropriately completes the request.\\n\\n\"\n",
    "    \"### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}\\n\\n\",\n",
    "    \"completion\": \" {response}\",\n",
    "}\n",
    "with open(\"template.json\", \"w\") as f:\n",
    "    json.dump(template, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a22171b1-1cec-4cec-9ce4-db62761633d9",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Upload dataset to S3\n",
    "---\n",
    "\n",
    "We will upload the prepared dataset to S3 which will be used for fine-tuning.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e1ee29a-8439-4788-8088-35a433fe2110",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "local_data_file = \"train.jsonl\"\n",
    "train_data_location = f\"s3://{output_bucket}/dolly_dataset\"\n",
    "S3Uploader.upload(local_data_file, train_data_location)\n",
    "S3Uploader.upload(\"template.json\", train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e61340-bc81-477d-aaf1-f37e8c554863",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train the model\n",
    "---\n",
    "Next, we fine-tune the LLaMA v2 7B model on the summarization dataset from Dolly. Finetuning scripts are based on scripts provided by [this repo](https://github.com/facebookresearch/llama-recipes/tree/main). To learn more about the fine-tuning scripts, please checkout section [5. Few notes about the fine-tuning method](#5.-Few-notes-about-the-fine-tuning-method). For a list of supported hyper-parameters and their default values, please see section [3. Supported Hyper-parameters for fine-tuning](#3.-Supported-Hyper-parameters-for-fine-tuning).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a71087e-9c9e-42d7-999e-5f3fac07bc4a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    environment={\"accept_eula\": \"true\"},\n",
    "    disable_output_compression=True,  # For Llama-2-70b, add instance_type = \"ml.g5.48xlarge\"\n",
    ")\n",
    "# By default, instruction tuning is set to false. Thus, to use instruction tuning dataset you use\n",
    "estimator.set_hyperparameters(instruction_tuned=\"True\", epoch=\"5\", max_input_length=\"1024\")\n",
    "estimator.fit({\"training\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e3889d9-1567-41ad-9375-fb738db629fa",
   "metadata": {
    "tags": []
   },
   "source": [
    "Studio Kernel Dying issue:  If your studio kernel dies and you lose reference to the estimator object, please see section [6. Studio Kernel Dead/Creating JumpStart Model from the training Job](#6.-Studio-Kernel-Dead/Creating-JumpStart-Model-from-the-training-Job) on how to deploy endpoint using the training job name and the model id. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e9decbf-08c6-4cb4-8644-4a96afb5bebf",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Deploy the fine-tuned model\n",
    "---\n",
    "Next, we deploy fine-tuned model. We will compare the performance of fine-tuned and pre-trained model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83eccacb-fa92-4fee-9734-6c72fec352fa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_predictor = attached_estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "016e591b-63f8-4e0f-941c-4b4e0b9dc6fc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_predictor = attached_estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb57904a-9631-45fe-bc3f-ae2fbb992960",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Evaluate the pre-trained and fine-tuned model\n",
    "---\n",
    "Next, we use TruLens evaluate the performance of the fine-tuned model and compare it with the pre-trained model. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b1adbef-ad38-41f2-b3af-c7053a50f3a5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import display\n",
    "from IPython.display import HTML\n",
    "import pandas as pd\n",
    "\n",
    "test_dataset = train_and_test_dataset[\"test\"]\n",
    "\n",
    "inputs, ground_truth_responses, responses_before_finetuning, responses_after_finetuning = (\n",
    "    [],\n",
    "    [],\n",
    "    [],\n",
    "    [],\n",
    ")\n",
    "\n",
    "\n",
    "def predict_and_print(datapoint):\n",
    "    # For instruction fine-tuning, we insert a special key between input and output\n",
    "    input_output_demarkation_key = \"\\n\\n### Response:\\n\"\n",
    "\n",
    "    payload = {\n",
    "        \"inputs\": template[\"prompt\"].format(\n",
    "            instruction=datapoint[\"instruction\"], context=datapoint[\"context\"]\n",
    "        )\n",
    "        + input_output_demarkation_key,\n",
    "        \"parameters\": {\"max_new_tokens\": 100},\n",
    "    }\n",
    "    inputs.append(payload[\"inputs\"])\n",
    "    ground_truth_responses.append(datapoint[\"response\"])\n",
    "    # Please change the following line to \"accept_eula=True\"\n",
    "    pretrained_response = pretrained_predictor.predict(\n",
    "        payload, custom_attributes=\"accept_eula=true\"\n",
    "    )\n",
    "    responses_before_finetuning.append(pretrained_response[0][\"generated_text\"])\n",
    "    # Please change the following line to \"accept_eula=True\"\n",
    "    finetuned_response = finetuned_predictor.predict(payload, custom_attributes=\"accept_eula=true\")\n",
    "    responses_after_finetuning.append(finetuned_response[0][\"generated_text\"])\n",
    "\n",
    "\n",
    "try:\n",
    "    for i, datapoint in enumerate(test_dataset.select(range(5))):\n",
    "        predict_and_print(datapoint)\n",
    "\n",
    "    df = pd.DataFrame(\n",
    "        {\n",
    "            \"Inputs\": inputs,\n",
    "            \"Ground Truth\": ground_truth_responses,\n",
    "            \"Response from non-finetuned model\": responses_before_finetuning,\n",
    "            \"Response from fine-tuned model\": responses_after_finetuning,\n",
    "        }\n",
    "    )\n",
    "    display(HTML(df.to_html()))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "196d7ac2-95bc-4d28-a763-1d45d36c14f1",
   "metadata": {},
   "source": [
    "### Set up as text to text LLM apps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17fc96a1-001d-47c9-b3b1-6a32438b50cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def base_llm(instruction, context):\n",
    "    # For instruction fine-tuning, we insert a special key between input and output\n",
    "    input_output_demarkation_key = \"\\n\\n### Response:\\n\"\n",
    "    payload = {\n",
    "        \"inputs\": template[\"prompt\"].format(\n",
    "            instruction=instruction, context=context\n",
    "        )\n",
    "        + input_output_demarkation_key,\n",
    "        \"parameters\": {\"max_new_tokens\": 200},\n",
    "    }\n",
    "    \n",
    "    return pretrained_predictor.predict(\n",
    "        payload, custom_attributes=\"accept_eula=true\"\n",
    "    )[0][\"generated_text\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "149c4533-84b9-412e-a431-e2c46e42f4da",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def finetuned_llm(instruction, context):\n",
    "    # For instruction fine-tuning, we insert a special key between input and output\n",
    "    input_output_demarkation_key = \"\\n\\n### Response:\\n\"\n",
    "    payload = {\n",
    "        \"inputs\": template[\"prompt\"].format(\n",
    "            instruction=instruction, context=context\n",
    "        )\n",
    "        + input_output_demarkation_key,\n",
    "        \"parameters\": {\"max_new_tokens\": 200},\n",
    "    }\n",
    "    \n",
    "    return finetuned_predictor.predict(\n",
    "        payload, custom_attributes=\"accept_eula=true\"\n",
    "    )[0][\"generated_text\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cc295b2-9696-4ecd-9e8b-b5aad0e4ecc7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_llm(test_dataset[\"instruction\"][0], test_dataset[\"context\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c22d163-c24b-4862-a18a-761be66f055f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_llm(test_dataset[\"instruction\"][0], test_dataset[\"context\"][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ef4c72-ad8c-40e5-b045-46f729f15630",
   "metadata": {},
   "source": [
    "Use TruLens for automated evaluation and tracking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0771370-e408-40b2-9aa1-eba161bb847c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from trulens_eval.feedback import GroundTruthAgreement, Groundedness\n",
    "from trulens_eval import TruBasicApp, Feedback, Tru, Select\n",
    "import boto3\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34f40abd-0b8b-42fc-9fbd-06d60d8b97ee",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Rename columns\n",
    "test_dataset = pd.DataFrame(test_dataset)\n",
    "test_dataset.rename(columns={\"instruction\": \"query\"}, inplace=True)\n",
    "\n",
    "# Convert DataFrame to a list of dictionaries\n",
    "golden_set = test_dataset[[\"query\",\"response\"]].to_dict(orient='records')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e28b44b8-bd3a-4a19-a054-900e7f480f72",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Instantiate Bedrock\n",
    "from trulens_eval import Bedrock\n",
    "\n",
    "# Initialize Bedrock as feedback function provider\n",
    "bedrock = Bedrock(model_id = \"amazon.titan-text-express-v1\", region_name=\"us-east-1\")\n",
    "\n",
    "# Create a Feedback object for ground truth similarity\n",
    "ground_truth = GroundTruthAgreement(golden_set, provider = bedrock)\n",
    "# Call the agreement measure on the instruction and output\n",
    "f_groundtruth = (Feedback(ground_truth.agreement_measure, name = \"Ground Truth Agreement\")\n",
    "                 .on(Select.Record.calls[0].args.args[0])\n",
    "                 .on_output()\n",
    "                )\n",
    "# Answer Relevance\n",
    "f_answer_relevance = (Feedback(bedrock.relevance_with_cot_reasons, name = \"Answer Relevance\")\n",
    "                      .on(Select.Record.calls[0].args.args[0])\n",
    "                      .on_output()\n",
    "                      )\n",
    "\n",
    "# Context Relevance\n",
    "f_context_relevance = (Feedback(bedrock.qs_relevance_with_cot_reasons, name = \"Context Relevance\")\n",
    "                       .on(Select.Record.calls[0].args.args[0])\n",
    "                       .on(Select.Record.calls[0].args.args[1])\n",
    "                      )\n",
    "\n",
    "# Groundedness\n",
    "grounded = Groundedness(groundedness_provider=bedrock)\n",
    "f_groundedness = (Feedback(grounded.groundedness_measure_with_cot_reasons, name = \"Groundedness\")\n",
    "                .on(Select.Record.calls[0].args.args[1])\n",
    "                .on_output()\n",
    "                .aggregate(grounded.grounded_statements_aggregator)\n",
    "            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0547e49-d998-4608-bed3-8a102dfcd560",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_recorder = TruBasicApp(base_llm, app_id=\"Base LLM\", feedbacks=[f_groundtruth, f_answer_relevance, f_context_relevance, f_groundedness])\n",
    "finetuned_recorder = TruBasicApp(finetuned_llm, app_id=\"Finetuned LLM\", feedbacks=[f_groundtruth, f_answer_relevance, f_context_relevance, f_groundedness])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be4ebe8d-56ac-4326-9f66-78d499f41ded",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i in range(len(test_dataset)):\n",
    "    with base_recorder as recording:\n",
    "        base_recorder.app(test_dataset[\"query\"][i], test_dataset[\"context\"][i])\n",
    "    with finetuned_recorder as recording:\n",
    "        finetuned_recorder.app(test_dataset[\"query\"][i], test_dataset[\"context\"][i])\n",
    "\n",
    "#Ignore minor errors in the stack trace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f2364e2-7cbe-4b76-9243-a2222938448b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Tru().get_records_and_feedback(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddd60b65-8266-4817-bf7f-a0fe65451f48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "records, feedback = Tru().get_leaderboard(app_ids=[\"Base LLM\", \"Finetuned LLM\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bc07af8-8a7d-4b65-b26f-7774637701a4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Tru().get_leaderboard(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ab81aec-7b41-4ccd-baf8-cce9ab90e2a8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Tru().run_dashboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b0a0f5-ef34-40db-8ab7-c24a5d14b525",
   "metadata": {},
   "source": [
    "### Clean up resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7ecb519-46dd-4a2f-aff0-77f53c9819ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete resources\n",
    "pretrained_predictor.delete_model()\n",
    "pretrained_predictor.delete_endpoint()\n",
    "finetuned_predictor.delete_model()\n",
    "finetuned_predictor.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}