    generative_ai/sm-jumpstart_foundation_llama_3_2_vision_language_finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "af2ee7fb-e888-4e38-a349-c7c40dfd2963",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "# Fine-tune LLaMA 3.2 Vision Language Models on SageMaker JumpStart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0679163a-387a-4ba6-8ce0-d5571614c0dc",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "251624f9-1eb6-4051-a774-0a4ba83cabf5",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "---\n",
    "This demo notebook illustrates how to leverage the Amazon SageMaker Python SDK to fine-tune the LLaMA 3.2 vision language model, including the 11B and 90B base and instruct versions, on your custom dataset for a visual question answering task.\"\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d9b99d-639b-40f3-91e3-1fe00ee032a4",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "### Model License information\n",
    "---\n",
    "To perform inference on these models, you need to pass custom_attributes='accept_eula=true' as part of header. This means you have read and accept the end-user-license-agreement (EULA) of the model. EULA can be found in model card description or from https://ai.meta.com/resources/models-and-libraries/llama-downloads/. By default, this notebook sets custom_attributes='accept_eula=false', so all inference requests will fail until you explicitly change this custom attribute.\n",
    "\n",
    "Note: Custom_attributes used to pass EULA are key/value pairs. The key and value are separated by '=' and pairs are separated by ';'. If the user passes the same key more than once, the last value is kept and passed to the script handler (i.e., in this case, used for conditional logic). For example, if 'accept_eula=false; accept_eula=true' is passed to the server, then 'accept_eula=true' is kept and passed to the script handler.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "019c4fcd-d6c5-4381-8425-1d224c0ac197",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "### Set up\n",
    "\n",
    "---\n",
    "We begin by installing and upgrading necessary packages. Restart the kernel after executing the cell below for the first time.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85addd9d-ec89-44a7-9fb5-9bc24fe9993b",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade sagemaker datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90719731-03ed-41bb-964b-12f59a5caa24",
   "metadata": {},
   "source": [
    "\n",
    "Take a look at this [link](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#built-in-algorithms-with-pre-trained-model-table\n",
    ") for all available models\n",
    "\n",
    "If you wish to fine-tune Llama 3.2 Vision Base/Instruct models, you can set model_id to: \n",
    "\n",
    "| Model | Model ID | All Supported Instances Types for fine-tuning |\n",
    "| - | - | - |\n",
    "| Llama 3.2 11B | meta-vlm-llama-3-2-11b-vision | ml.g5.48xlarge, ml.p4de.24xlarge, ml.p4d.24xlarge |\n",
    "| Llama 3.2 11B Instruct | meta-vlm-llama-3-2-11b-vision-instruct | ml.g5.48xlarge, ml.p4de.24xlarge   |\n",
    "| Llama 3.2 90B | meta-vlm-llama-3-2-90b-vision | ml.g5.48xlarge|\n",
    "| Llama 3.2 90B Instruct | meta-vlm-llama-3-2-90b-vision-instruct | ml.g5.48xlarge |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7e01401-82db-4d49-9457-f930f4138618",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, model_version = \"meta-vlm-llama-3-2-11b-vision-instruct\", \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e19e16f-d459-40c6-9d6b-0272938b3878",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Dataset preparation for visual question and answering task\n",
    "\n",
    "---\n",
    "We currently offer fine-tuning for vision question and answering (VQA) task. The vision language model can be fine-tuned on the image-text dataset, provided that the data  \n",
    "is formulated in the expected format. The resulting model can be further deployed for inference. Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** \n",
    "  - A train and an optional validation directory. Train and validation directories should contain one directory named `images` hosting all the image data and one JSON lines (`.jsonl`) file named `metadata.jsonl`.  \n",
    "  - In the `metadata.jsonl` file, each example is a dictionary which contains three keys named `file_name`, `prompt`, and `completion`. The `file_name` defines the path to image data. `prompt` defines the text input prompt and `completion` defines the text completion corresponding to the input prompt.\n",
    "  Below is an example of the contents in the `metadata.jsonl` file.\n",
    "\n",
    "\n",
    "```\n",
    "{\"file_name\": \"images/img_0.jpg\", \"prompt\": \"what is the date mentioned in this letter?\", \"completion\": \"1/8/93\"}\n",
    "{\"file_name\": \"images/img_1.jpg\", \"prompt\": \"what is the contact person name mentioned in letter?\", \"completion\": \"P. Carter\"}\n",
    "{\"file_name\": \"images/img_2.jpg\", \"prompt\": \"Which part of Virginia is this letter sent from\", \"completion\": \"Richmond\"}\n",
    "```\n",
    "\n",
    "- **Output:** A trained model that can be deployed for inference. \n",
    "\n",
    "We provide a subset of [DocVQA](https://www.docvqa.org/datasets) as an example dataset to demonstrate fine-tuning. The full dataset can be downloaded from [here](https://www.docvqa.org/datasets).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ffd7c15-7f59-4702-b544-de1f9b493d47",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "from tqdm import tqdm\n",
    "from datasets import load_dataset\n",
    "import base64\n",
    "from io import BytesIO\n",
    "import json\n",
    "from sagemaker.jumpstart.types import JumpStartSerializablePayload\n",
    "\n",
    "dataset_name = \"HuggingFaceM4/DocumentVQA\"\n",
    "data = load_dataset(\n",
    "    dataset_name, cache_dir=\"./\"\n",
    ")  # you need make sure there is enough space to download the dataset in your instance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8020ab-77fb-4e88-89f5-8805d9d1f0bf",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Process the dataset to the format required in the above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcae3e11-2bd8-4ab0-8499-e33517af4b82",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def process_data(data, output_dir, num_ex):\n",
    "    local_data_file = f\"{output_dir}/metadata.jsonl\"\n",
    "    with open(local_data_file, \"w\") as f:\n",
    "        for i in tqdm(range(num_ex)):\n",
    "            each = data[i]\n",
    "            q = each[\"question\"]\n",
    "            each_img = each[\"image\"]\n",
    "            a = each[\"answers\"][0]\n",
    "\n",
    "            example = {\"file_name\": f\"images/img_{i}.jpg\", \"prompt\": q, \"completion\": a}\n",
    "            json.dump(example, f)\n",
    "            f.write(\"\\n\")\n",
    "\n",
    "            each_img.save(f\"{output_dir}/images/img_{i}.jpg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8ca8eff-9007-4767-8f41-1d93c3399020",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for split, num in [(\"train\", 1000), (\"validation\", 20)]:\n",
    "    os.makedirs(f\"docvqa/{split}\", exist_ok=True)\n",
    "    os.makedirs(f\"docvqa/{split}/images\", exist_ok=True)\n",
    "    process_data(data=data[split], output_dir=f\"./docvqa/{split}\", num_ex=num)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c95ff66-7969-44be-b3ae-6f21e74b2018",
   "metadata": {},
   "source": [
    "### Upload the dataset to the S3\n",
    "\n",
    "Given the dataset contains image, the uploading process will take a while depending on the size of examples you process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e1ee29a-8439-4788-8088-35a433fe2110",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"/{default_bucket_prefix}\"\n",
    "\n",
    "local_data_dir = \"./docvqa/train/\"\n",
    "train_data_location = f\"s3://{output_bucket}{default_bucket_prefix_path}/docvqa-1000-20\"\n",
    "S3Uploader.upload(local_data_dir, train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e61340-bc81-477d-aaf1-f37e8c554863",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train the model\n",
    "---\n",
    "Next, we fine-tune the LIama vision 11B model on a subset of [DocVQA dataset](https://www.docvqa.org/datasets). For the finetuning method, we currently parameter-efficient finetuning [LoRA](https://arxiv.org/abs/2106.09685).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55ab66d0-470d-4559-b413-b7163cc65477",
   "metadata": {},
   "source": [
    "### Retrieve and modify the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02f9ea87-07e4-4b42-b3e5-18981e112544",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "my_hyperparameters = hyperparameters.retrieve_default(\n",
    "    model_id=model_id, model_version=model_version\n",
    ")\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e57cfa-3828-4ea7-a7db-840c8b45e94b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "my_hyperparameters[\"epoch\"] = \"1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "584712ac-adf5-451e-bfd0-a72bcfc20cbf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameters.validate(\n",
    "    model_id=model_id, model_version=model_version, hyperparameters=my_hyperparameters\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a71087e-9c9e-42d7-999e-5f3fac07bc4a",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    environment={\"accept_eula\": \"true\"},  # Please change {\"accept_eula\": \"true\"}\n",
    "    disable_output_compression=True,\n",
    "    instance_type=\"ml.p4de.24xlarge\",\n",
    "    hyperparameters=my_hyperparameters,\n",
    ")\n",
    "estimator.fit({\"training\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e3889d9-1567-41ad-9375-fb738db629fa",
   "metadata": {
    "tags": []
   },
   "source": [
    "Studio Kernel Dying issue:  If your studio kernel dies and you lose reference to the estimator object, please see section [2. Studio Kernel Dead/Creating JumpStart Model from the training Job](#2.-Studio-Kernel-Dead/Creating-JumpStart-Model-from-the-training-Job) on how to deploy endpoint using the training job name and the model id. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e9decbf-08c6-4cb4-8644-4a96afb5bebf",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Deploy the fine-tuned model\n",
    "---\n",
    "Next, we deploy fine-tuned model. We will compare the performance of fine-tuned and pre-trained model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07913134-e9cc-4e57-8fb1-b9e36147cbac",
   "metadata": {},
   "source": [
    "Before deployment, we need to know whether the model is finetuned with chat template. If so, we need deploy it with `MESSAGES_API_ENABLED` and input / output signatures are different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76adeab2-77b2-4aed-a2f9-06e75ac9a37b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "is_chat_template = True if my_hyperparameters[\"chat_template\"] == \"True\" else False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95a6ad72-6826-46a3-884f-6e3b1c939cea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if is_chat_template:\n",
    "    estimator.environment = {\"MESSAGES_API_ENABLED\": \"true\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "016e591b-63f8-4e0f-941c-4b4e0b9dc6fc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb57904a-9631-45fe-bc3f-ae2fbb992960",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Evaluate the fine-tuned model\n",
    "---\n",
    "Next, we use the test data to evaluate the performance of the fine-tuned model on the validation set of DocVQA dataset. We also use [prompt template suggested by Meta on this task](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/eval_details.md#docvqa).\n",
    "\n",
    "Note. If hyperparameter `chat_template` is False (i.e., you did not train the model in chat completion format but text completion format), we need append `### Response:\\n\\n` at the end of prompt. You can see details on argument `instruct` in the following function `formulate_payload`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05c9fd4-97de-48ab-b474-aea2eccacdd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "from PIL import Image\n",
    "import io\n",
    "\n",
    "\n",
    "def display_base64_image(base64_string):\n",
    "    \"\"\"\n",
    "    Displays a base64 encoded image.\n",
    "    \"\"\"\n",
    "\n",
    "    # Decode the base64 string\n",
    "    image_data = base64.b64decode(base64_string)\n",
    "\n",
    "    # Open the image using Pillow\n",
    "    image = Image.open(io.BytesIO(image_data))\n",
    "    new_size = (700, 500)  # Specify the desired width and height\n",
    "    resized_image = image.resize(new_size)\n",
    "    # Display the image\n",
    "    resized_image.show()\n",
    "\n",
    "\n",
    "def get_image_decode_64base(image_path):\n",
    "    with open(image_path, \"rb\") as f:\n",
    "        image = base64.b64encode(f.read()).decode(\"utf-8\")\n",
    "    return image\n",
    "\n",
    "\n",
    "def formulate_payload(q, image, instruct):\n",
    "    img_path = f\"data:image/jpg;base64,{image}\"\n",
    "    if instruct:\n",
    "        payload = {\n",
    "            \"messages\": [\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [\n",
    "                        {\n",
    "                            \"type\": \"text\",\n",
    "                            \"text\": f\"Read the text in the image carefully and answer the question with the text as seen exactly in the image. For yes/no questions, just respond Yes or No. If the answer is numeric, just respond with the number and nothing else. If the answer has multiple words, just respond with the words and absolutely nothing else. Never respond in a sentence or a phrase.\\n Question: {q}\",\n",
    "                        },\n",
    "                        {\"type\": \"image_url\", \"image_url\": {\"url\": img_path}},\n",
    "                    ],\n",
    "                }\n",
    "            ],\n",
    "            \"max_tokens\": 512,\n",
    "            \"logprobs\": False,\n",
    "        }\n",
    "    else:\n",
    "        prompt = f\"![]({img_path})<|image|><|begin_of_text|>Read the text in the image carefully and answer the question with the text as seen exactly in the image. For yes/no questions, just respond Yes or No. If the answer is numeric, just respond with the number and nothing else. If the answer has multiple words, just respond with the words and absolutely nothing else. Never respond in a sentence or a phrase.\\n Question: {q}### Response:\\n\\n\"\n",
    "        payload = {\n",
    "            \"body\": {\n",
    "                \"inputs\": prompt,\n",
    "                \"parameters\": {\n",
    "                    \"max_new_tokens\": 512,\n",
    "                    \"return_full_text\": False,\n",
    "                    # \"temperature\": 0.1,\n",
    "                    \"do_sample\": False,\n",
    "                    # \"top_p\": 0.97\n",
    "                },\n",
    "            },\n",
    "            \"content_type\": \"application/json\",\n",
    "            \"accept\": \"application/json\",\n",
    "        }\n",
    "    return payload\n",
    "\n",
    "\n",
    "start = \"\\033[1m\"\n",
    "end = \"\\033[0;0m\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eda44aef-8a43-44fe-8026-64854c331f43",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"./docvqa/validation/metadata.jsonl\") as f:\n",
    "    data = [json.loads(line) for line in f]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c17d555d-d75f-4521-9737-de907cfaf47b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i, each in enumerate(data[:5]):  # take first 5 observations\n",
    "    q, a, image = (\n",
    "        each[\"prompt\"],\n",
    "        each[\"completion\"],\n",
    "        get_image_decode_64base(image_path=f\"./docvqa/validation/{each['file_name']}\"),\n",
    "    )\n",
    "\n",
    "    payload = formulate_payload(q=q, image=image, instruct=is_chat_template)\n",
    "    # print(payload)\n",
    "    ft_response = finetuned_predictor.predict(payload)\n",
    "    if is_chat_template:\n",
    "        ft_response = ft_response[\"choices\"][0][\"message\"][\"content\"]\n",
    "    else:\n",
    "        ft_response = ft_response[0][\"generated_text\"]\n",
    "    print(f\"==========={start}Example {i}{end}=============\")\n",
    "    print(f\"{start}Prompt{end}: {q}\")\n",
    "    print(f\"{start}FT response{end}: {ft_response}\\n\")\n",
    "    print(f\"{start}GT{end}: {a}\\n\")\n",
    "    display_base64_image(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b0a0f5-ef34-40db-8ab7-c24a5d14b525",
   "metadata": {},
   "source": [
    "### Clean up resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d73ab2da-d00f-46db-90eb-81812898653b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete resources\n",
    "finetuned_predictor.delete_model()\n",
    "finetuned_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "759ce98f-a35a-4c64-9fae-50894b5e9f37",
   "metadata": {},
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b6d023-3487-4571-8b52-f332790c1ad7",
   "metadata": {},
   "source": [
    "### 1. Supported Instance types for fine-tuning Llama 3\n",
    "\n",
    "---\n",
    "We have tested our scripts on the following instances types for fine-tuning Llama 3:\n",
    "\n",
    "| Model | Model ID | All Supported Instances Types for fine-tuning |\n",
    "| - | - | - |\n",
    "| Llama 3.2 11B | meta-vlm-llama-3-2-11b-vision | ml.g5.48xlarge, ml.p4de.24xlarge, ml.p4d.24xlarge |\n",
    "| Llama 3.2 11B Instruct | meta-vlm-llama-3-2-11b-vision-instruct | ml.g5.48xlarge, ml.p4de.24xlarge   |\n",
    "| Llama 3.2 90B | meta-vlm-llama-3-2-90b-vision | ml.g5.48xlarge|\n",
    "| Llama 3.2 90B Instruct | meta-vlm-llama-3-2-90b-vision-instruct | ml.g5.48xlarge |\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "909ce841-3a2c-4c08-a102-b94148036a5a",
   "metadata": {},
   "source": [
    "### 2. Studio Kernel Dead/Creating JumpStart Model from the training Job\n",
    "---\n",
    "Due to the size of the Llama 70B model, training job may take several hours and the studio kernel may die during the training phase. However, during this time, training is still running in SageMaker. If this happens, you can still deploy the endpoint using the training job name with the following code:\n",
    "\n",
    "How to find the training job name? Go to Console -> SageMaker -> Training -> Training Jobs -> Identify the training job name and substitute in the following cell. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfa60a66-1c2f-42df-8079-191319e28a65",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "training_job_name = \"<<Replace this with Training Job Name>>\"\n",
    "\n",
    "attached_estimator = JumpStartEstimator.attach(training_job_name, model_id)\n",
    "attached_estimator.logs()\n",
    "finetuned_predictor = attached_estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e719b3ee-30cc-4995-9d7a-94442e95185f",
   "metadata": {},
   "source": [
    "If you lost your endpoint deployment, you can retrieve it by following code. You need to go to AWS console -> SageMaker -> Endpoint (left pannel) to get the endpoint name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b506ecb-feef-4fc7-a2a5-c957e518af5b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.predictor import retrieve_default\n",
    "\n",
    "finetuned_predictor = retrieve_default(\n",
    "    model_id=model_id, model_version=\"*\", endpoint_name=\"<<Replace this with Endpoint Name>>\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7be1219",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|llama-3-finetuning.ipynb)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}