    generative_ai/sm-jumpstart_llama_2_chat_completion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8af3794b",
   "metadata": {},
   "source": [
    "# Chat Completion: Run Llama 2 Models in SageMaker JumpStart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07389e40",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "---\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "446b1b24",
   "metadata": {},
   "source": [
    "---\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy a JumpStart model for Text Generation using the Llama 2 fine-tuned model optimized for dialogue use cases.\n",
    "\n",
    "To perform inference on these models, you need to pass custom_attributes='accept_eula=true' as part of header. This means you have read and accept the end-user-license-agreement (EULA) of the model. EULA can be found in model card description or from https://ai.meta.com/resources/models-and-libraries/llama-downloads/. By default, this notebook sets custom_attributes='accept_eula=false', so all inference requests will fail until you explicitly change this custom attribute.\n",
    "\n",
    "Note: Custom_attributes used to pass EULA are key/value pairs. The key and value are separated by '=' and pairs are separated by ';'. If the user passes the same key more than once, the last value is kept and passed to the script handler (i.e., in this case, used for conditional logic). For example, if 'accept_eula=false; accept_eula=true' is passed to the server, then 'accept_eula=true' is kept and passed to the script handler.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35642ab2",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b55e677-3429-4668-b100-bd63d2a4c401",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet sagemaker datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d458cf0-02e2-4066-927b-25fa5ef2a07e",
   "metadata": {},
   "source": [
    "***\n",
    "You can continue with the default model or choose a different model: this notebook will run with the following model IDs :\n",
    "- `meta-textgeneration-llama-2-7b-f`\n",
    "- `meta-textgeneration-llama-2-13b-f`\n",
    "- `meta-textgeneration-llama-2-70b-f`\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a882ae62",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, model_version = \"meta-textgeneration-llama-2-7b-f\", \"2.*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11eef0dd",
   "metadata": {},
   "source": [
    "## Deploy model\n",
    "\n",
    "***\n",
    "You can now deploy the model using SageMaker JumpStart.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e52afae-868d-4736-881f-7180f393003a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "model = JumpStartModel(model_id=model_id, model_version=model_version)\n",
    "predictor = model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ef7207e-01ba-4ac2-b4a9-c8f6f0e1c498",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Invoke the endpoint\n",
    "\n",
    "***\n",
    "### Supported Parameters\n",
    "This model supports the following inference payload parameters:\n",
    "\n",
    "* **max_new_tokens:** Model generates text until the output length (excluding the input context length) reaches max_new_tokens. If specified, it must be a positive integer.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "\n",
    "You may specify any subset of the parameters mentioned above while invoking an endpoint. \n",
    "\n",
    "***\n",
    "### Notes\n",
    "- If `max_new_tokens` is not defined, the model may generate up to the maximum total tokens allowed, which is 4K for these models. This may result in endpoint query timeout errors, so it is recommended to set `max_new_tokens` when possible. For 7B, 13B, and 70B models, we recommend to set `max_new_tokens` no greater than 1500, 1000, and 500 respectively, while keeping the total number of tokens less than 4K.\n",
    "- In order to support a 4k context length, this model has restricted query payloads to only utilize a batch size of 1. Payloads with larger batch sizes will receive an endpoint error prior to inference.\n",
    "- This model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...).\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5adf9b4-c7e1-4090-aefe-9cae0d096968",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_dialog(payload, response):\n",
    "    dialog = payload[\"inputs\"][0]\n",
    "    for msg in dialog:\n",
    "        print(f\"{msg['role'].capitalize()}: {msg['content']}\\n\")\n",
    "    print(\n",
    "        f\">>>> {response[0]['generation']['role'].capitalize()}: {response[0]['generation']['content']}\"\n",
    "    )\n",
    "    print(\"\\n==================================\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2fbb9af",
   "metadata": {},
   "source": [
    "### Example 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cbde5e7-1068-41f9-999a-70ef04e1cbbb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        [\n",
    "            {\"role\": \"user\", \"content\": \"what is the recipe of mayonnaise?\"},\n",
    "        ]\n",
    "    ],\n",
    "    \"parameters\": {\"max_new_tokens\": 512, \"top_p\": 0.9, \"temperature\": 0.6},\n",
    "}\n",
    "try:\n",
    "    response = predictor.predict(payload, custom_attributes=\"accept_eula=false\")\n",
    "    print_dialog(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5574e4e2",
   "metadata": {},
   "source": [
    "### Example 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cda81ccf-0188-4117-8355-801ef98aaa48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        [\n",
    "            {\"role\": \"user\", \"content\": \"I am going to Paris, what should I see?\"},\n",
    "            {\n",
    "                \"role\": \"assistant\",\n",
    "                \"content\": \"\"\"\\\n",
    "Paris, the capital of France, is known for its stunning architecture, art museums, historical landmarks, and romantic atmosphere. Here are some of the top attractions to see in Paris:\n",
    "\n",
    "1. The Eiffel Tower: The iconic Eiffel Tower is one of the most recognizable landmarks in the world and offers breathtaking views of the city.\n",
    "2. The Louvre Museum: The Louvre is one of the world's largest and most famous museums, housing an impressive collection of art and artifacts, including the Mona Lisa.\n",
    "3. Notre-Dame Cathedral: This beautiful cathedral is one of the most famous landmarks in Paris and is known for its Gothic architecture and stunning stained glass windows.\n",
    "\n",
    "These are just a few of the many attractions that Paris has to offer. With so much to see and do, it's no wonder that Paris is one of the most popular tourist destinations in the world.\"\"\",\n",
    "            },\n",
    "            {\"role\": \"user\", \"content\": \"What is so great about #1?\"},\n",
    "        ]\n",
    "    ],\n",
    "    \"parameters\": {\"max_new_tokens\": 512, \"top_p\": 0.9, \"temperature\": 0.6},\n",
    "}\n",
    "try:\n",
    "    response = predictor.predict(payload, custom_attributes=\"accept_eula=false\")\n",
    "    print_dialog(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aa8d152",
   "metadata": {},
   "source": [
    "### Example 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de6e8250-88c8-4b1c-a70b-ae5a4976e6ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        [\n",
    "            {\"role\": \"system\", \"content\": \"Always answer with Haiku\"},\n",
    "            {\"role\": \"user\", \"content\": \"I am going to Paris, what should I see?\"},\n",
    "        ]\n",
    "    ],\n",
    "    \"parameters\": {\"max_new_tokens\": 512, \"top_p\": 0.9, \"temperature\": 0.6},\n",
    "}\n",
    "try:\n",
    "    response = predictor.predict(payload, custom_attributes=\"accept_eula=false\")\n",
    "    print_dialog(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076644d4",
   "metadata": {},
   "source": [
    "### Example 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2da83b4a-1e61-495c-b509-38266f5c44eb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "payload = {\n",
    "    \"inputs\": [\n",
    "        [\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": \"Always answer with emojis\",\n",
    "            },\n",
    "            {\"role\": \"user\", \"content\": \"How to go from Beijing to NY?\"},\n",
    "        ]\n",
    "    ],\n",
    "    \"parameters\": {\"max_new_tokens\": 512, \"top_p\": 0.9, \"temperature\": 0.6},\n",
    "}\n",
    "try:\n",
    "    response = predictor.predict(payload, custom_attributes=\"accept_eula=false\")\n",
    "    print_dialog(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c924b9dd-147d-46d2-b003-d1b407596b8e",
   "metadata": {},
   "source": [
    "## Dataset preparation for fine-tuning\n",
    "\n",
    "---\n",
    "\n",
    "You can fine-tune on the dataset with domain adaptation format or instruction tuning format or the chat dataset format. Please find more details in the section [Dataset formatting instruction for training](#1.-Dataset-formatting-instruction-for-training). In this demo, we will use a subset of OpenAssistant's TOP-1 Conversation Threads as an example dataset. It can be downloaded from [here](https://huggingface.co/datasets/OpenAssistant/oasst_top1_2023-08-25). It contains roughly 13,000 samples of conversations between the Assistant and the user. \n",
    "\n",
    "\n",
    "Training data is formatted in JSON lines (.jsonl) format, where each line is a dictionary representing a single set of conversation.\n",
    "\n",
    "To train your model on a collection of unstructured dataset (text files), please see the section [1.3. Example fine-tuning with Domain-Adaptation dataset format](#1.3.-Example-fine-tuning-with-Domain-Adaptation-dataset-format) in the Appendix. To train your model on instruction tuning dataset format, please see the section [ 1.4. Example fine-tuning with Instruction tuning dataset format](#1.4.-Example-fine-tuning-with-Instruction-tuning-dataset-format).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36311650-c33e-4df7-9e51-0412608dab25",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import re\n",
    "\n",
    "# Load the dataset\n",
    "dataset = load_dataset(\"OpenAssistant/oasst_top1_2023-08-25\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84b32e4d-f8ab-440d-975c-3f515c903374",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define a function to transform the data\n",
    "def transform_conversation(example):\n",
    "    conversation_text = example[\"text\"]\n",
    "\n",
    "    segments = re.split(\"<\\|im_start\\|>|<\\|im_end\\|>\", conversation_text)\n",
    "    reformatted_segments = []\n",
    "    dialog_list = []\n",
    "\n",
    "    # Iterate over pairs of segments\n",
    "    for i in range(1, len(segments) - 1, 4):\n",
    "        human_text = segments[i].strip().replace(\"user\", \"\").strip()\n",
    "\n",
    "        # Check if there is a corresponding assistant segment before processing\n",
    "        if i + 1 < len(segments):\n",
    "            assistant_text = segments[i + 2].strip().replace(\"assistant\", \"\").strip()\n",
    "            dialog_list.append({\"role\": \"user\", \"content\": human_text})\n",
    "            dialog_list.append({\"role\": \"assistant\", \"content\": assistant_text})\n",
    "\n",
    "        else:\n",
    "            dialog_list.append({\"role\": \"user\", \"content\": human_text})\n",
    "    return {\"dialog\": dialog_list}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "085eb0a0-1c90-4eb4-b1d8-7ff457d3aab7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "transformed_dataset = dataset.map(transform_conversation).remove_columns(\"text\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04efa358-a657-4c30-9166-6b4baecc4826",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "transformed_dataset[\"train\"].select(range(5000)).to_json(\"train.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6c43bbe-b172-4445-ae4a-581ab3e04a1c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"/{default_bucket_prefix}\"\n",
    "\n",
    "local_data_file = \"train.jsonl\"\n",
    "train_data_location = f\"s3://{output_bucket}{default_bucket_prefix_path}/oasst_top1\"\n",
    "S3Uploader.upload(local_data_file, train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c834800d-5168-4239-8cd3-e9cee1365906",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "---\n",
    "Next, we fine-tune the LLaMA v2 7B model on the summarization dataset from Dolly. Finetuning scripts are based on scripts provided by [this repo](https://github.com/facebookresearch/llama-recipes/tree/main). To learn more about the fine-tuning scripts, please checkout section [5. Few notes about the fine-tuning method](#5.-Few-notes-about-the-fine-tuning-method). For a list of supported hyper-parameters and their default values, please see section [3. Supported Hyper-parameters for fine-tuning](#3.-Supported-Hyper-parameters-for-fine-tuning).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fff9b35c-5ed4-4294-b00f-5932b5574b82",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    environment={\"accept_eula\": \"true\"},\n",
    "    disable_output_compression=True,  # For Llama-2-70b, add instance_type = \"ml.g5.48xlarge\"\n",
    ")\n",
    "\n",
    "estimator.set_hyperparameters(\n",
    "    chat_dataset=\"True\", instruction_tuned=\"False\", epoch=\"1\", max_input_length=\"1024\"\n",
    ")\n",
    "estimator.fit({\"training\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a675254-1866-4913-b1e4-eb3899a0c553",
   "metadata": {},
   "source": [
    "Studio Kernel Dying issue:  If your studio kernel dies and you lose reference to the estimator object, please see section [6. Studio Kernel Dead/Creating JumpStart Model from the training Job](#6.-Studio-Kernel-Dead/Creating-JumpStart-Model-from-the-training-Job) on how to deploy endpoint using the training job name and the model id. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a37f2f03-a3a4-48d6-8096-bbe44bef6c66",
   "metadata": {},
   "source": [
    "### Deploy the fine-tuned model\n",
    "---\n",
    "Next, we deploy fine-tuned model. We will compare the performance of fine-tuned and pre-trained model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef84df10-2565-455c-9d3b-f7b0dc90a0b8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db767525-31a6-434f-b726-090a46395111",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Evaluate the pre-trained and fine-tuned model\n",
    "---\n",
    "Next, we use the test data to evaluate the performance of the fine-tuned model and compare it with the pre-trained model. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20531df9-6b0d-49d5-8521-77648fc41727",
   "metadata": {
    "pycharm": {
     "is_executing": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "test_dataset = transformed_dataset[\"test\"]\n",
    "\n",
    "try:\n",
    "    for i, datapoint in enumerate(test_dataset.select(range(5))):\n",
    "        payload = {\n",
    "            \"inputs\": [datapoint[\"dialog\"][:-1]],\n",
    "            \"parameters\": {\"max_new_tokens\": 512, \"top_p\": 0.9, \"temperature\": 0.6},\n",
    "        }\n",
    "        response = finetuned_predictor.predict(payload, custom_attributes=\"accept_eula=false\")\n",
    "        print_dialog(payload, response)\n",
    "        print(\"Ground Truth Response:\")\n",
    "        print(\n",
    "            f\">>>> {datapoint['dialog'][-1]['role'].capitalize()}: {datapoint['dialog'][-1]['content']}\"\n",
    "        )\n",
    "        print(f\"\\n============End of Example {i+1} ======================\\n\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2189335-4d40-44bb-bef1-4bd3597801b2",
   "metadata": {},
   "source": [
    "### Clean up resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec2391e3-bde2-4a7f-bb5c-7af8d1d1c7ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete resources\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()\n",
    "finetuned_predictor.delete_model()\n",
    "finetuned_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaee53f8-814a-490e-abef-34861cacd4f7",
   "metadata": {},
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0d3bc1-bffe-4a4e-84f5-72991e9a5666",
   "metadata": {},
   "source": [
    "### 1. Dataset formatting instruction for training\n",
    "\n",
    "---\n",
    "\n",
    "####  Fine-tune the Model on a New Dataset\n",
    "We currently offer two types of fine-tuning: instruction fine-tuning and domain adaption fine-tuning. You can easily switch to one of the training \n",
    "methods by specifying parameter `instruction_tuned` being 'True' or 'False'.\n",
    "\n",
    "#### 1.1. Chat fine-tuning\n",
    "\n",
    "\n",
    "The Text generation model can be fine-tuned on the chat dataset, provided that the data is in the expected format. The resulting chat model can be further deployed for inference. Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** A train and an optional validation directory. Train and validation directories should contain one or multiple JSON lines (.jsonl) formatted files. All training data must be in a single folder, however it can be saved in multiple jsonl files. The .jsonl file extension is mandatory.\n",
    "    - The training data must be formatted in a JSON lines (.jsonl) format, where each line is a dictionary representing a single data sample. Each line in the file is a list of conversations between the user and the assistant model. This model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...).\n",
    "- **Output:**  A trained model that can be deployed for inference.\n",
    "\n",
    "The best model is selected according to the validation loss, calculated at the end of each epoch. If a validation set is not given, an (adjustable) percentage of the training data is automatically split and used for validation.The training data must be formatted in a JSON lines (.jsonl) format, where each line is a dictionary representing a single data sample.   \n",
    "\n",
    "Here is an example of a line in the training file:\n",
    "\n",
    "{\"dialog\": [{\"content\":\"what is the height of the empire state building\",\"role\":\"user\"},{\"content\":\"381 meters, or 1,250 feet, is the height of the Empire State Building. If you also account for the antenna, it brings up the total height to 443 meters, or 1,454 feet\",\"role\":\"assistant\"},{\"content\":\"Some people need to pilot an aircraft above it and need to know.\\nSo what is the answer in feet?\",\"role\":\"user\"},{\"content\":\"1454 feet\",\"role\":\"assistant\"}]}\n",
    "\n",
    "\n",
    "\n",
    "#### 1.2. Domain adaptation fine-tuning\n",
    "The Text Generation model can also be fine-tuned on any domain specific dataset. After being fine-tuned on the domain specific dataset, the model\n",
    "is expected to generate domain specific text and solve various NLP tasks in that specific domain with **few shot prompting**.\n",
    "\n",
    "Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** A train and an optional validation directory. Each directory contains a CSV/JSON/TXT file. \n",
    "  - For CSV/JSON files, the train or validation data is used from the column called 'text' or the first column if no column called 'text' is found.\n",
    "  - The number of files under train and validation (if provided) should equal to one, respectively. \n",
    "- **Output:** A trained model that can be deployed for inference. \n",
    "\n",
    "Below is an example of a TXT file for fine-tuning the Text Generation model. The TXT file is SEC filings of Amazon from year 2021 to 2022.\n",
    "\n",
    "```Note About Forward-Looking Statements\n",
    "This report includes estimates, projections, statements relating to our\n",
    "business plans, objectives, and expected operating results that are “forward-\n",
    "looking statements” within the meaning of the Private Securities Litigation\n",
    "Reform Act of 1995, Section 27A of the Securities Act of 1933, and Section 21E\n",
    "of the Securities Exchange Act of 1934. Forward-looking statements may appear\n",
    "throughout this report, including the following sections: “Business” (Part I,\n",
    "Item 1 of this Form 10-K), “Risk Factors” (Part I, Item 1A of this Form 10-K),\n",
    "and “Management’s Discussion and Analysis of Financial Condition and Results\n",
    "of Operations” (Part II, Item 7 of this Form 10-K). These forward-looking\n",
    "statements generally are identified by the words “believe,” “project,”\n",
    "“expect,” “anticipate,” “estimate,” “intend,” “strategy,” “future,”\n",
    "“opportunity,” “plan,” “may,” “should,” “will,” “would,” “will be,” “will\n",
    "continue,” “will likely result,” and similar expressions. Forward-looking\n",
    "statements are based on current expectations and assumptions that are subject\n",
    "to risks and uncertainties that may cause actual results to differ materially.\n",
    "We describe risks and uncertainties that could cause actual results and events\n",
    "to differ materially in “Risk Factors,” “Management’s Discussion and Analysis\n",
    "of Financial Condition and Results of Operations,” and “Quantitative and\n",
    "Qualitative Disclosures about Market Risk” (Part II, Item 7A of this Form\n",
    "10-K). Readers are cautioned not to place undue reliance on forward-looking\n",
    "statements, which speak only as of the date they are made. We undertake no\n",
    "obligation to update or revise publicly any forward-looking statements,\n",
    "whether because of new information, future events, or otherwise.\n",
    "GENERAL\n",
    "Embracing Our Future ...\n",
    "```\n",
    "\n",
    "\n",
    "#### 1.3. Instruction fine-tuning\n",
    "The Text generation model can be instruction-tuned on any text data provided that the data \n",
    "is in the expected format. The instruction-tuned model can be further deployed for inference. \n",
    "Below are the instructions for how the training data should be formatted for input to the \n",
    "model.\n",
    "\n",
    "Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** A train and an optional validation directory. Train and validation directories should contain one or multiple JSON lines (`.jsonl`) formatted files. In particular, train directory can also contain an optional `*.json` file describing the input and output formats. \n",
    "  - The best model is selected according to the validation loss, calculated at the end of each epoch.\n",
    "  If a validation set is not given, an (adjustable) percentage of the training data is\n",
    "  automatically split and used for validation.\n",
    "  - The training data must be formatted in a JSON lines (`.jsonl`) format, where each line is a dictionary\n",
    "representing a single data sample. All training data must be in a single folder, however\n",
    "it can be saved in multiple jsonl files. The `.jsonl` file extension is mandatory. The training\n",
    "folder can also contain a `template.json` file describing the input and output formats. If no\n",
    "template file is given, the following template will be used:\n",
    "  ```json\n",
    "  {\n",
    "    \"prompt\": \"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}\",\n",
    "    \"completion\": \"{response}\"\n",
    "  }\n",
    "  ```\n",
    "  - In this case, the data in the JSON lines entries must include `instruction`, `context` and `response` fields. If a custom template is provided it must also use `prompt` and `completion` keys to define\n",
    "  the input and output templates.\n",
    "  Below is a sample custom template:\n",
    "\n",
    "  ```json\n",
    "  {\n",
    "    \"prompt\": \"question: {question} context: {context}\",\n",
    "    \"completion\": \"{answer}\"\n",
    "  }\n",
    "  ```\n",
    "Here, the data in the JSON lines entries must include `question`, `context` and `answer` fields. \n",
    "- **Output:** A trained model that can be deployed for inference. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bded571-476a-464e-bc06-28cd89ba08ec",
   "metadata": {},
   "source": [
    "#### 1.3. Example fine-tuning with Domain-Adaptation dataset format\n",
    "---\n",
    "We provide a subset of SEC filings data of Amazon in domain adaptation dataset format. It is downloaded from publicly available [EDGAR](https://www.sec.gov/edgar/searchedgar/companysearch). Instruction of accessing the data is shown [here](https://www.sec.gov/os/accessing-edgar-data).\n",
    "\n",
    "License: [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/legalcode).\n",
    "\n",
    "Please uncomment the following code to fine-tune the model on dataset in domain adaptation format.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9b249b1-1167-4850-97b6-fb0a1b0c20f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import boto3\n",
    "# model_id = \"meta-textgeneration-llama-2-7b\"\n",
    "\n",
    "# estimator = JumpStartEstimator(model_id=model_id,  environment={\"accept_eula\": \"true\"},instance_type = \"ml.g5.24xlarge\")\n",
    "# estimator.set_hyperparameters(instruction_tuned=\"False\", epoch=\"5\")\n",
    "# estimator.fit({\"training\": f\"s3://jumpstart-cache-prod-{boto3.Session().region_name}/training-datasets/sec_amazon\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0658b6e-6637-4098-bd4c-c347d82c8d38",
   "metadata": {},
   "source": [
    "#### 1.4. Example fine-tuning with Instruction tuning dataset format\n",
    "---\n",
    "Next, we fine-tune the LLaMA v2 7B model on the summarization dataset from Dolly dataset.\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef7c81d5-0457-497b-83f6-da6c5446afce",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Dataset format\n",
    "\n",
    "from datasets import load_dataset\n",
    "\n",
    "dolly_dataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")\n",
    "\n",
    "# To train for question answering/information extraction, you can replace the assertion in next line to example[\"category\"] == \"closed_qa\"/\"information_extraction\".\n",
    "summarization_dataset = dolly_dataset.filter(lambda example: example[\"category\"] == \"summarization\")\n",
    "summarization_dataset = summarization_dataset.remove_columns(\"category\")\n",
    "\n",
    "# We split the dataset into two where test data is used to evaluate at the end.\n",
    "train_and_test_dataset = summarization_dataset.train_test_split(test_size=0.1)\n",
    "\n",
    "# Dumping the training data to a local file to be used for training.\n",
    "train_and_test_dataset[\"train\"].to_json(\"train.jsonl\")\n",
    "\n",
    "\n",
    "\n",
    "import json\n",
    "\n",
    "template = {\n",
    "    \"prompt\": \"Below is an instruction that describes a task, paired with an input that provides further context. \"\n",
    "    \"Write a response that appropriately completes the request.\\n\\n\"\n",
    "    \"### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}\\n\\n\",\n",
    "    \"completion\": \" {response}\",\n",
    "}\n",
    "with open(\"template.json\", \"w\") as f:\n",
    "    json.dump(template, f)\n",
    "    \n",
    "\n",
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "local_data_file = \"train.jsonl\"\n",
    "train_data_location = f\"s3://{output_bucket}/dolly_dataset\"\n",
    "S3Uploader.upload(local_data_file, train_data_location)\n",
    "S3Uploader.upload(\"template.json\", train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")\n",
    "    \n",
    "\n",
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    environment={\"accept_eula\": \"true\"},\n",
    "    disable_output_compression=True,  # For Llama-2-70b, add instance_type = \"ml.g5.48xlarge\"\n",
    ")\n",
    "# By default, instruction tuning is set to false. Thus, to use instruction tuning dataset you use\n",
    "estimator.set_hyperparameters(instruction_tuned=\"True\", \"chat_dataset\"=\"False\", epoch=\"5\", max_input_length=\"1024\")\n",
    "estimator.fit({\"training\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76ccbb2c-bf5f-4ff2-ae2b-76ff1a7478ea",
   "metadata": {},
   "source": [
    "### 2. Supported Hyper-parameters for fine-tuning\n",
    "---\n",
    "- epoch: The number of passes that the fine-tuning algorithm takes through the training dataset. Must be an integer greater than 1. Default: 5\n",
    "- learning_rate: The rate at which the model weights are updated after working through each batch of training examples. Must be a positive float greater than 0. Default: 1e-4.\n",
    "- instruction_tuned: Whether to instruction-train the model or not. Must be 'True' or 'False'. Default: 'False'\n",
    "- chat_dataset: If True, dataset is assumed to be in chat format. At most one of instruction_tuned and chat_dataset can be True.\n",
    "- add_input_output_demarcation_key: For instruction tuned dataset, if this is True a demarcation key(\\\"### Response:\\\\n\\\") is added between the prompt and completion before training. Default: 'True'.\n",
    "- per_device_train_batch_size: The batch size per GPU core/CPU for training. Must be a positive integer. Default: 4.\n",
    "- per_device_eval_batch_size: The batch size per GPU core/CPU for evaluation. Must be a positive integer. Default: 1\n",
    "- max_train_samples: For debugging purposes or quicker training, truncate the number of training examples to this value. Value -1 means using all of training samples. Must be a positive integer or -1. Default: -1. \n",
    "- max_val_samples: For debugging purposes or quicker training, truncate the number of validation examples to this value. Value -1 means using all of validation samples. Must be a positive integer or -1. Default: -1. \n",
    "- max_input_length: Maximum total input sequence length after tokenization. Sequences longer than this will be truncated. If -1, max_input_length is set to the minimum of 1024 and the maximum model length defined by the tokenizer. If set to a positive value, max_input_length is set to the minimum of the provided value and the model_max_length defined by the tokenizer. Must be a positive integer or -1. Default: -1. \n",
    "- validation_split_ratio: If validation channel is none, ratio of train-validation split from the train data. Must be between 0 and 1. Default: 0.2. \n",
    "- train_data_split_seed: If validation data is not present, this fixes the random splitting of the input training data to training and validation data used by the algorithm. Must be an integer. Default: 0.\n",
    "- preprocessing_num_workers: The number of processes to use for the preprocessing. If None, main process is used for preprocessing. Default: \"None\"\n",
    "- lora_r: Lora R. Must be a positive integer. Default: 8.\n",
    "- lora_alpha: Lora Alpha. Must be a positive integer. Default: 32\n",
    "- lora_dropout: Lora Dropout. must be a positive float between 0 and 1. Default: 0.05. \n",
    "- int8_quantization: If True, model is loaded with 8 bit precision for training. Default for 7B/13B: False. Default for 70B: True.\n",
    "- enable_fsdp: If True, training uses Fully Sharded Data Parallelism. Default for 7B/13B: True. Default for 70B: False.\n",
    "\n",
    "Note 1: int8_quantization is not supported with FSDP. Also, int8_quantization = 'False' and enable_fsdp = 'False' is not supported due to CUDA memory issues for any of the g5 family instances. Thus, we recommend setting exactly one of int8_quantization or enable_fsdp to be 'True'\n",
    "Note 2: Due to the size of the model, 70B model can not be fine-tuned with enable_fsdp = 'True' for any of the supported instance types.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d65b70e8-5f24-4061-b105-efe8a8464232",
   "metadata": {},
   "source": [
    "### 3. Supported Instance types\n",
    "\n",
    "---\n",
    "We have tested our scripts on the following instances types:\n",
    "\n",
    "- 7B, 7B-F: ml.g5.12xlarge, nl.g5.24xlarge, ml.g5.48xlarge, ml.p3dn.24xlarge\n",
    "- 13B, 13B-F: ml.g5.24xlarge, ml.g5.48xlarge, ml.p3dn.24xlarge\n",
    "- 70B, 70B-F: ml.g5.48xlarge\n",
    "\n",
    "Other instance types may also work to fine-tune. Note: When using p3 instances, training will be done with 32 bit precision as bfloat16 is not supported on these instances. Thus, training job would consume double the amount of CUDA memory when training on p3 instances compared to g5 instances.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db5f1f27-8755-443e-af37-07f6254f2745",
   "metadata": {},
   "source": [
    "### 4. Few notes about the fine-tuning method\n",
    "\n",
    "---\n",
    "- Fine-tuning scripts are based on [this repo](https://github.com/facebookresearch/llama-recipes/tree/main). \n",
    "- Instruction tuning dataset is first converted into domain adaptation dataset format before fine-tuning. \n",
    "- Fine-tuning scripts utilize Fully Sharded Data Parallel (FSDP) as well as Low Rank Adaptation (LoRA) method fine-tuning the models\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4093134-a2fa-42df-9d12-e241dff9325b",
   "metadata": {},
   "source": [
    "### 5. Studio Kernel Dead/Creating JumpStart Model from the training Job\n",
    "---\n",
    "Due to the size of the Llama 70B model, training job may take several hours and the studio kernel may die during the training phase. However, during this time, training is still running in SageMaker. If this happens, you can still deploy the endpoint using the training job name with the following code:\n",
    "\n",
    "How to find the training job name? Go to Console -> SageMaker -> Training -> Training Jobs -> Identify the training job name and substitute in the following cell. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5c05c89-3a9e-4efc-97e0-9e3e32f957a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "# training_job_name = <<training_job_name>>\n",
    "\n",
    "# attached_estimator = JumpStartEstimator.attach(training_job_name, model_id)\n",
    "# attached_estimator.logs()\n",
    "# attached_estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b722f8b",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/generative_ai|sm-jumpstart_llama_2_chat_completion.ipynb)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.g5.12xlarge",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}