    generative_ai/sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dbfa4226-79a1-4520-b131-c798e42e34f7",
   "metadata": {},
   "source": [
    "# Fine-tune and deploy LLaMA V2 models on [AWS Trainium](https://aws.amazon.com/ec2/instance-types/trn1/) and [AWS Inferentia](https://aws.amazon.com/ec2/instance-types/inf2/) based instances in SageMaker JumpStart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081de5c0",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29838a99",
   "metadata": {},
   "source": [
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy pre-trained Llama 2 model as well as fine-tune it for your dataset in domain adaptation or instruction tuning format on [AWS Trainium](https://aws.amazon.com/ec2/instance-types/trn1/) and [AWS Inferentia](https://aws.amazon.com/ec2/instance-types/inf2/) based instances.\n",
    "\n",
    "AWS Neuron is an SDK with a compiler, runtime, and profiling tools that unlocks high-performance and cost-effective deep learning (DL) acceleration. It supports high-performance training on AWS Trainium-based Amazon Elastic Compute Cloud (Amazon EC2) Trn1 instances. For model deployment, it supports high-performance and low-latency inference on AWS Inferentia-based Amazon EC2 Inf1 instances and AWS Inferentia2-based Amazon EC2 Inf2 instances. For details, see [Official documentation](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/index.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3d9b99d-639b-40f3-91e3-1fe00ee032a4",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "### Model License information\n",
    "---\n",
    "To perform inference on these models, you need to pass 'accept_eula=True' as part of model.deploy() call. This means you have read and accept the end-user-license-agreement (EULA) of the model. EULA can be found in model card description or from https://ai.meta.com/resources/models-and-libraries/llama-downloads/. By default, this notebook sets 'accept_eula=False', so all inference requests will fail until you explicitly change this custom attribute.\n",
    "\n",
    "Similarly, to perform fine-tuning on these models, you need pass environment variable '{\"accept_eula\": \"true\"}' to `JumpStartEstimator` class.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "019c4fcd-d6c5-4381-8425-1d224c0ac197",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "### Set up\n",
    "\n",
    "---\n",
    "We begin by installing and upgrading necessary packages. Restart the kernel after executing the cell below for the first time.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85addd9d-ec89-44a7-9fb5-9bc24fe9993b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade sagemaker datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13274b9b-87bd-4090-a6aa-294570c31e0e",
   "metadata": {},
   "source": [
    "## Deploy Pre-trained Model\n",
    "\n",
    "\n",
    "***\n",
    "You can now deploy the model using SageMaker JumpStart through 2 options. Option 1 allows you to quickly deploy the endpoint with default setting in two lines of code. Option 2 allows you to have more customized configurations. \n",
    "\n",
    "To deploy a model on [AWS Trainium](https://aws.amazon.com/ec2/instance-types/trn1/) or [AWS Inferentia](https://aws.amazon.com/ec2/instance-types/inf2/) based instances, we will firstly need call PyTorch Neuron ([torch-neuronx](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/arch/neuron-features/neuron-caching.html?highlight=graph#neuron-persistent-cache)) to compile the model into a Neuron specific graph. Then during runtime the graph is executed on the **NeuronCores** of the [AWS Trainium](https://aws.amazon.com/ec2/instance-types/trn1/) or [AWS Inferentia](https://aws.amazon.com/ec2/instance-types/inf2/) based instances. Compiling the graph involves running optimizations that can make use of the NeuronCores efficiently.\n",
    "\n",
    "In SageMaker JumpStart, we pre-compile the neuron graphs for a varieity of configurations such that you do not spend time waiting for compiling the graph during endpoint deployment, as long as the deployment parameters (**environmental variables**) matches one of configurations listed as below. Otherwise, the compilation will be triggered during endpoint deployment, which will take a slightly longer time to deploy a model.\n",
    "\n",
    "|||LLaMA V2 7B and 7B Chat|||\n",
    "|---|---|---|---|---|\n",
    "|Instance type|Context length|Batch size| Tensor parallel degree| Data type |\n",
    "|ml.inf2.xlarge|1024|1|2|fp16|\n",
    "|ml.inf2.8xlarge|2048|1|2|fp16|\n",
    "|ml.inf2.24xlarge|4096|4|4|fp16|\n",
    "|ml.inf2.24xlarge|4096|4|8|fp16|\n",
    "|ml.inf2.24xlarge|4096|4|12|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|4|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|8|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|12|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|24|fp16|\n",
    "\n",
    "\n",
    "|||LLaMA V2 13B and 13B Chat|||\n",
    "|---|---|---|---|---|\n",
    "|Instance type|Context length|Batch size| Tensor parallel degree| Data type |\n",
    "|ml.inf2.8xlarge|1024|1|2|fp16|\n",
    "|ml.inf2.24xlarge|2048|4|4|fp16|\n",
    "|ml.inf2.24xlarge|4096|4|8|fp16|\n",
    "|ml.inf2.24xlarge|4096|4|12|fp16|\n",
    "|ml.inf2.48xlarge|2048|4|4|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|8|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|12|fp16|\n",
    "|ml.inf2.48xlarge|4096|4|24|fp16|\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7e01401-82db-4d49-9457-f930f4138618",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdOnly"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id = \"meta-textgenerationneuron-llama-2-7b\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89769446",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_version = \"1.*\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "971c9087",
   "metadata": {
    "jumpStartAlterations": [
     "dropModelSelection"
    ]
   },
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "from sagemaker.jumpstart.filters import And\n",
    "\n",
    "\n",
    "filter_value = And(\"task == textgenerationneuron\", \"framework == meta\")\n",
    "model_ids_meta_textgenerationneuron = list_jumpstart_models(filter=filter_value)\n",
    "\n",
    "# display the model-ids in a dropdown to select a model for inference.\n",
    "model_dropdown = Dropdown(\n",
    "    options=model_ids_meta_textgenerationneuron,\n",
    "    value=model_id,\n",
    "    description=\"Select a model\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")\n",
    "display(model_dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0b862f0",
   "metadata": {
    "jumpStartAlterations": [
     "dropModelSelection"
    ]
   },
   "outputs": [],
   "source": [
    "model_id = model_dropdown.value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1e648c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "EXAMPLE_ENV = {\n",
    "    \"meta-textgenerationneuron-llama-2-13b-f\": {\n",
    "        \"context_length\": \"1024\",\n",
    "        \"batch_size\": \"1\",\n",
    "        \"tensor_parallel_degree\": \"2\",\n",
    "        \"instance_type\": \"ml.inf2.8xlarge\",\n",
    "    },\n",
    "    \"meta-textgenerationneuron-llama-2-13b\": {\n",
    "        \"context_length\": \"1024\",\n",
    "        \"batch_size\": \"1\",\n",
    "        \"tensor_parallel_degree\": \"2\",\n",
    "        \"instance_type\": \"ml.inf2.8xlarge\",\n",
    "    },\n",
    "    \"meta-textgenerationneuron-llama-2-7b-f\": {\n",
    "        \"context_length\": \"2048\",\n",
    "        \"batch_size\": \"1\",\n",
    "        \"tensor_parallel_degree\": \"2\",\n",
    "        \"instance_type\": \"ml.inf2.8xlarge\",\n",
    "    },\n",
    "    \"meta-textgenerationneuron-llama-2-7b\": {\n",
    "        \"context_length\": \"2048\",\n",
    "        \"batch_size\": \"1\",\n",
    "        \"tensor_parallel_degree\": \"2\",\n",
    "        \"instance_type\": \"ml.inf2.8xlarge\",\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1722b230-b7bc-487f-b4ee-98ca42848423",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "option = \"option1\"\n",
    "\n",
    "if option == \"option1\":\n",
    "    model = JumpStartModel(model_id=model_id)\n",
    "\n",
    "else:\n",
    "    model = JumpStartModel(\n",
    "        model_id=model_id,\n",
    "        env={\n",
    "            \"OPTION_DTYPE\": \"fp16\",  ## correspond to the column `Data type`\n",
    "            \"OPTION_N_POSITIONS\": EXAMPLE_ENV[model_id][\n",
    "                \"context_length\"\n",
    "            ],  ## correspond to the column `Contexnt length`\n",
    "            \"OPTION_TENSOR_PARALLEL_DEGREE\": EXAMPLE_ENV[model_id][\n",
    "                \"tensor_parallel_degree\"\n",
    "            ],  ## correspond to the column `Tensor parallel degree`\n",
    "            \"OPTION_MAX_ROLLING_BATCH_SIZE\": EXAMPLE_ENV[model_id][\n",
    "                \"batch_size\"\n",
    "            ],  ## correspond to the column `Batch size`\n",
    "        },\n",
    "        instance_type=EXAMPLE_ENV[model_id][\n",
    "            \"instance_type\"\n",
    "        ],  ## correspond to the column `Instance type`\n",
    "    )\n",
    "\n",
    "pretrained_predictor = model.deploy(accept_eula=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8017c4ef-eb89-4da6-8e28-c800adbfc4b8",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Invoke the endpoint\n",
    "\n",
    "---\n",
    "Next, we invoke the endpoint with some sample queries. Later, in this notebook, we will fine-tune this model with a custom dataset and carry out inference using the fine-tuned model. We will also show comparison between results obtained via the pre-trained and the fine-tuned models.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b795a085-048f-42b2-945f-0cd339c1cf91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_response(payload, response):\n",
    "    print(payload[\"inputs\"])\n",
    "    print(f\"> {response['generated_text']}\")\n",
    "    print(\"\\n==================================\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dd833f8-1ddc-4805-80b2-19e7db629880",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "payload = {\n",
    "    \"inputs\": \"I believe the meaning of life is\",\n",
    "    \"parameters\": {\n",
    "        \"max_new_tokens\": 64,\n",
    "        \"top_p\": 0.9,\n",
    "        \"temperature\": 0.6,\n",
    "    },\n",
    "}\n",
    "try:\n",
    "    response = pretrained_predictor.predict(payload)\n",
    "    print_response(payload, response)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e19e16f-d459-40c6-9d6b-0272938b3878",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Dataset preparation for fine-tuning\n",
    "\n",
    "---\n",
    "\n",
    "You can fine-tune on the dataset with domain adaptation format or instruction tuning format. Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** A train directory containing either a JSON lines (`.jsonl`) or text (`.txt`) formatted file. \n",
    "  - For JSON lines (JSONL) file, each line is a dictionary, repsentating a dictionary. The key in dictionary (each line) has to be 'text'.\n",
    "  - The number of files under train directory should equal to one. \n",
    "- **Output:** A trained model that can be deployed for inference. \n",
    "\n",
    "In this demo, we will use a subset of [Dolly dataset](https://huggingface.co/datasets/databricks/databricks-dolly-15k) in an instruction tuning format. Dolly dataset contains roughly 15,000 instruction following records for various categories such as question answering, summarization, information extraction etc. It is available under Apache 2.0 license. We will select the summarization examples for fine-tuning.\n",
    "\n",
    "For demonstration of using text file as input, please see [Appendix 2](#2.-Use-text-file-as-input-to-fine-tune-LLaMA-2)\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dd20a0d-15a5-49b0-a330-a75755d046ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "dolly_dataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")\n",
    "\n",
    "task = \"information_extraction\"\n",
    "# To train for summarization/closed question and answering, you can replace the assertion in next line to example[\"category\"] == \"sumarization\"/\"closed_qa\".\n",
    "summarization_dataset = dolly_dataset.filter(lambda example: example[\"category\"] == task)\n",
    "summarization_dataset = summarization_dataset.remove_columns(\"category\")\n",
    "\n",
    "# We split the dataset into two where test data is used to evaluate at the end.\n",
    "train_and_test_dataset = summarization_dataset.train_test_split(test_size=0.1)\n",
    "\n",
    "# Dumping the training data to a local file to be used for training.\n",
    "train_and_test_dataset[\"train\"].to_json(\"train.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9fbf002-3ee3-4cc8-8fce-871939f1bd19",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_and_test_dataset[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b2e5489-33dc-4623-92da-f6fc97bd25ab",
   "metadata": {},
   "source": [
    "---\n",
    "Next, we use a prompt template for preprocessing the data in an instruction / input format for the training job, and also for inferencing the deployed endpoint.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90451114-7cf5-445c-88e3-02ccaa5d3a4b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompt = \"\"\"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}### Response:\\n{response}\\n\\n<s>\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9a19a34-1afd-4483-ac9f-81574d03d9af",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def apply_prompt_template(sample):\n",
    "    return {\n",
    "        \"text\": prompt.format(\n",
    "            instruction=sample[\"instruction\"],\n",
    "            context=sample[\"context\"],\n",
    "            response=sample[\"response\"],\n",
    "        )\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b5af455-3426-4d2e-9377-a57daa2cdd79",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_processed = train_and_test_dataset.map(\n",
    "    apply_prompt_template, remove_columns=list(train_and_test_dataset[\"train\"].features)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1804128d-c3c8-410b-9ab6-b1e0a423c20b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_processed[\"train\"].to_json(f\"dolly/processed-train-{task}.jsonl\")\n",
    "dataset_processed[\"test\"].to_json(f\"dolly/processed-test-{task}.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a22171b1-1cec-4cec-9ce4-db62761633d9",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Upload dataset to S3\n",
    "---\n",
    "\n",
    "We will upload the prepared dataset to S3 which will be used for fine-tuning.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e1ee29a-8439-4788-8088-35a433fe2110",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"/{default_bucket_prefix}\"\n",
    "\n",
    "local_data_file = f\"dolly/processed-train-{task}.jsonl\"\n",
    "train_data_location = f\"s3://{output_bucket}{default_bucket_prefix_path}/dolly_dataset_trn1\"\n",
    "S3Uploader.upload(local_data_file, train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e61340-bc81-477d-aaf1-f37e8c554863",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train the model\n",
    "---\n",
    "Next, we fine-tune the LLaMA v2 model on the summarization dataset from Dolly on [AWS Trainium](https://aws.amazon.com/ec2/instance-types/trn1/) instance. You have two options: `ml.trn1.32xlarge` (default) and `ml.trn1n.32xlarge`. Finetuning scripts are based on scripts provided by [Neuronx-Nemo-Megatron](https://github.com/aws-neuron/neuronx-nemo-megatron). For a list of supported hyper-parameters and their default values, please see [supported hyperparameters for fine-tuning](#3.-Supported-Hyper-parameters-for-fine-tuning).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fc9a6b0-b4df-4420-b55d-1906e215b79e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "my_hyperparameters = hyperparameters.retrieve_default(\n",
    "    model_id=model_id, model_version=model_version\n",
    ")\n",
    "\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69bc46b1-bf7d-4ba3-9bff-4015189ea43e",
   "metadata": {},
   "source": [
    "Overwrite some of the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ee59a63-656d-49cb-b72d-08ebd8afb1c3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# my_hyperparameters[\"max_input_length\"] = \"4096\" # you can increase it up to 4096 for sequence length.\n",
    "my_hyperparameters[\"max_steps\"] = \"25\"\n",
    "my_hyperparameters[\"learning_rate\"] = \"0.0001\"\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35b5a3e2-5b64-4fca-aec7-045ccfa4c24b",
   "metadata": {},
   "source": [
    "Validate hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6685ff5b-5f63-4822-85c0-1a65cca0e713",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameters.validate(\n",
    "    model_id=model_id, model_version=model_version, hyperparameters=my_hyperparameters\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a71087e-9c9e-42d7-999e-5f3fac07bc4a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    hyperparameters=my_hyperparameters,\n",
    "    environment={\n",
    "        \"accept_eula\": \"false\"\n",
    "    },  # please change `accept_eula` to be `true` to accept EULA.\n",
    "    # instance_type=\"ml.trn1n.32xlarge\", if not specified, default `ml.trn1.32xlarge` will be used.\n",
    ")\n",
    "\n",
    "estimator.fit({\"train\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e3889d9-1567-41ad-9375-fb738db629fa",
   "metadata": {
    "tags": []
   },
   "source": [
    "Studio Kernel idle issue:  If your studio kernel goes idle and you lose reference to the estimator object, please see section [4. Studio Kernel Dead/Creating JumpStart Model from the training Job](#4.-Studio-Kernel-Dead/Creating-JumpStart-Model-from-the-training-Job) on how to deploy endpoint using the training job name and the model id. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e9decbf-08c6-4cb4-8644-4a96afb5bebf",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Deploy the fine-tuned model\n",
    "---\n",
    "Next, we deploy the fine-tuned model. We will compare the performance of fine-tuned and pre-trained model.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "016e591b-63f8-4e0f-941c-4b4e0b9dc6fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "finetuned_predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb57904a-9631-45fe-bc3f-ae2fbb992960",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Evaluate the pre-trained and fine-tuned model\n",
    "---\n",
    "Next, we use the test data to evaluate the performance of the fine-tuned model and compare it with the pre-trained model. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c01e892-a70e-487b-989f-c1bbe5caba23",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompt_inference = \"\"\"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\\n\\n### Instruction:\\n{instruction}\\n\\n### Input:\\n{context}\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87085bf6-dc7e-46f3-8563-d2e4aafd0820",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "test_dataset = train_and_test_dataset[\"test\"]\n",
    "\n",
    "inputs, ground_truth_responses, responses_before_finetuning, responses_after_finetuning = (\n",
    "    [],\n",
    "    [],\n",
    "    [],\n",
    "    [],\n",
    ")\n",
    "\n",
    "\n",
    "def predict_and_print(datapoint):\n",
    "    # For instruction fine-tuning, we insert a special key between input and output\n",
    "    input_output_demarkation_key = \"\\n\\n### Response:\\n\"\n",
    "\n",
    "    payload = {\n",
    "        \"inputs\": prompt_inference.format(\n",
    "            instruction=datapoint[\"instruction\"], context=datapoint[\"context\"]\n",
    "        )\n",
    "        + input_output_demarkation_key,\n",
    "        \"parameters\": {\"max_new_tokens\": 100},\n",
    "    }\n",
    "    inputs.append(payload[\"inputs\"])\n",
    "    ground_truth_responses.append(datapoint[\"response\"])\n",
    "    pretrained_response = pretrained_predictor.predict(payload)\n",
    "    responses_before_finetuning.append(pretrained_response[\"generated_text\"])\n",
    "    finetuned_response = finetuned_predictor.predict(payload)\n",
    "    responses_after_finetuning.append(finetuned_response[\"generated_text\"])\n",
    "\n",
    "\n",
    "try:\n",
    "    for i, datapoint in enumerate(test_dataset.select(range(5))):\n",
    "        predict_and_print(datapoint)\n",
    "\n",
    "    df = pd.DataFrame(\n",
    "        {\n",
    "            \"Inputs\": inputs,\n",
    "            \"Ground Truth\": ground_truth_responses,\n",
    "            \"Response from non-finetuned model\": responses_before_finetuning,\n",
    "            \"Response from fine-tuned model\": responses_after_finetuning,\n",
    "        }\n",
    "    )\n",
    "    display(HTML(df.to_html()))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b0a0f5-ef34-40db-8ab7-c24a5d14b525",
   "metadata": {},
   "source": [
    "### Clean up resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d73ab2da-d00f-46db-90eb-81812898653b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete resources\n",
    "pretrained_predictor.delete_model()\n",
    "pretrained_predictor.delete_endpoint()\n",
    "finetuned_predictor.delete_model()\n",
    "finetuned_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "759ce98f-a35a-4c64-9fae-50894b5e9f37",
   "metadata": {},
   "source": [
    "# Appendix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d1c8c86-bfe2-4828-a7aa-dbd7a5ee075f",
   "metadata": {},
   "source": [
    "### 1. Supported Inference Parameters\n",
    "\n",
    "---\n",
    "This model supports the following inference payload parameters:\n",
    "\n",
    "\n",
    "* **max_length:** Model generates text until the output length (which includes the input context length) reaches `max_length`. If specified, it must be a positive integer.\n",
    "* **max_new_tokens:** Model generates text until the output length (excluding the input context length) reaches `max_new_tokens`. If specified, it must be a positive integer.\n",
    "* **num_beams:** Number of beams used in the greedy search. If specified, it must be integer greater than or equal to `num_return_sequences`.\n",
    "* **no_repeat_ngram_size:** Model ensures that a sequence of words of `no_repeat_ngram_size` is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **early_stopping:** If True, text generation is finished when all beam hypotheses reach the end of sentence token. If specified, it must be boolean.\n",
    "* **do_sample:** If True, sample the next word as per the likelihood. If specified, it must be boolean.\n",
    "* **top_k:** In each step of text generation, sample from only the `top_k` most likely words. If specified, it must be a positive integer.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "* **stop**: If specified, it must be a list of strings. Text generation stops if any one of the specified strings is generated.\n",
    "\n",
    "We may specify any subset of the parameters mentioned above while invoking an endpoint.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db86872-cca4-4ec5-8574-6876668ebf12",
   "metadata": {},
   "source": [
    "### 2. Use text file as input to fine-tune LLaMA-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d85c93a9-ebe2-4966-a5d6-af4c053f69f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import boto3\n",
    "# model_id = \"meta-textgenerationneuron-llama-2-7b\" #or  \"meta-textgenerationneuron-llama-2-13b\"\n",
    "\n",
    "# estimator = JumpStartEstimator(model_id=model_id,  environment={\"accept_eula\": \"false\"})\n",
    "# estimator.set_hyperparameters(max_steps=30)\n",
    "# estimator.fit({\"training\": f\"s3://jumpstart-cache-prod-{boto3.Session().region_name}/training-datasets/sec_amazon\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19f7e4f8-970f-4a1d-b6ee-86bc77b8b9a9",
   "metadata": {},
   "source": [
    "### 3. Supported Hyper-parameters for fine-tuning\n",
    "---\n",
    "- max_input_length: Maximum total input sequence length after tokenization. Sequences longer than this will be truncated. Default: 2048.\n",
    "- learning_rate: The rate at which the model weights are updated after working through each batch of training examples. Must be a positive float greater than 0. Default: 6e-6.\n",
    "- min_learning_rate: The learning rate at the last step of learning rate scheduler 'CosineAnnealing'. Default: 1e-06.\n",
    "- global_train_batch_size: The global batch size for training. Based on global_train_batch_size, the gradient accumulation is calculated as global_train_batch_size / (data_parallel_degree * per_device_train_batch_size), where data_parallel_degree is calculated as total number of neuron cores / (tensor_parallel_degree * pipeline_parallel_degree). Default: 256.\n",
    "- per_device_train_batch_size: The batch size per Neuron core for training. Default: 1\n",
    "- layer_norm_epilson: During layer normalization, a value added to the denominator for numerical stability. See [documentation](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html). Default: 0.00001.\n",
    "- preprocessing_num_workers: The number of processors to use for the preprocessing. If None, all of workers (number of vCPUs) are used for preprocessing. Default: \"None\"\n",
    "- weight_decay: The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in `AdamW` optimizer. Default: 0.1.\n",
    "- lr_scheduler_type: Learning rate scheduler type. Default: 'CosineAnnealing' (currently we only support 'CosineAnnealing' scheduler type).\n",
    "- warmup_steps: Linear warmup over warmup steps. Default: 10.\n",
    "- constant_steps: The number of steps for learning rate to be constant after warmup_steps in 'CosineAnnealing' scheduler type. Default: 0.\n",
    "- adam_beta1: The beta1 hyperparameter (exponential decay rate for the first moment estimates) for the AdamW optimizer. Default: 0.9.\n",
    "- adam_beta2: The beta2 hyperparameter (exponential decay rate for the first moment estimates) for the AdamW optimizer. Default: 0.95.\n",
    "- mixed_precision: Whether to use mixed precision. If mixed_precision to be 'True', it means that master weights and optimizer states are stored in fp32, and model weights are saved in bf16. For details, see [reference](https://arxiv.org/pdf/1710.03740.pdf). Default: 'True'.\n",
    "- tensor_parallel_degree: The number of neuron cores which specific model weights, gradients, and optimizer states are split across. For details, see [reference](https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-extended-features-pytorch-tensor-parallelism.html). Default: \"8\" (currently we only support parallel degree as 8).\n",
    "- pipeline_parallel_degree: The number of neuron cores which the layers of a model are partitioned across. Default: \"1\" (currently we only support \"1\" for LLaMA-2 7B and \"4\" for LLaMA-2 13B).\n",
    "- append_eod: Whether to append an `<eod>` token to the end of each example. By setting it to 'True', the fine-tuned model tends to generate succinct output. Default: 'False'.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "909ce841-3a2c-4c08-a102-b94148036a5a",
   "metadata": {},
   "source": [
    "### 4. Studio Kernel goes idle/Creating JumpStart Model from the training Job\n",
    "---\n",
    "Training job may take several hours due to setting of hyperparameters and the studio kernel may be in idle stage during the training phase. However, during this time, training is still running in SageMaker. If this happens, you can still deploy the endpoint using the training job name with the following code:\n",
    "\n",
    "How to find the training job name? Go to Console -> SageMaker -> Training -> Training Jobs -> Identify the training job name and substitute in the following cell. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfa60a66-1c2f-42df-8079-191319e28a65",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "# training_job_name = <<training_job_name>>\n",
    "\n",
    "# attached_estimator = JumpStartEstimator.attach(training_job_name, model_id)\n",
    "# attached_estimator.logs()\n",
    "# attached_estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9708003",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/generative_ai|sm-jumpstart_foundation_trainium_inferentia_finetuning_deployment.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}