    generative_ai/sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e78dc97c",
   "metadata": {},
   "source": [
    "# Fine-tune Llama3.1 405B Base and Instruct Variants on SageMaker JumpStart"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e567a4",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4e36b9",
   "metadata": {},
   "source": [
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to fine-tune Llama3.1 and further deploy the fine-tuned model on SageMaker JumpStart.\n",
    "\n",
    "Below is the content of the notebook.\n",
    "\n",
    "1. [Setup](#1.-Setup)\n",
    "2. [Deploy model](#2.-Deploy-model)\n",
    "3. [Fine-tune model](#3.-Fine-tune-model)\n",
    "4. [Evaluate the pre-trained and fine-tuned model](#4.-Qualitatively-evaluate-the-pre-trained-and-fine-tuned-model)\n",
    "\n",
    "The notebook requires users to specify following variables to start with.\n",
    "* Specify `model_id` (default value: `meta-textgeneration-llama-3-1-405b-fp8`) (other option: `meta-textgeneration-llama-3-1-405b-instruct-fp8`)\n",
    "* Specify `accept_eula` argument to be True in `model.deploy()` to accept the end-user license agreement (EULA) before deployment the model in an endpoint, given Llama model is gated.\n",
    "* Sepcify `{\"accept_eula\": \"true\"}` in argument `environment` of `JumpStartEstimator` to accept the end-user license agreement (EULA) before fine-tuning.\n",
    "\n",
    "#### The fine-tuning job will require more than 5 hours for the Llama3.1 405b model due to the large number of model parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af77efb",
   "metadata": {},
   "source": [
    "## 1. Setup\n",
    "First, upgrade to the latest sagemaker SDK to ensure all available models are deployable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b05b931-992e-4526-978d-f03196874a3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --quiet --upgrade sagemaker jmespath datasets transformers jinja2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dbb6b35",
   "metadata": {},
   "source": [
    "Select the desired model to deploy. The provided dropdown filters all text generation models available in SageMaker JumpStart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f625a488",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "\n",
    "\n",
    "try:\n",
    "    dropdown = Dropdown(\n",
    "        options=list_jumpstart_models(\"search_keywords includes Text Generation\"),\n",
    "        value=\"meta-textgeneration-llama-3-1-405b-fp8\",\n",
    "        description=\"Select a JumpStart text generation model:\",\n",
    "        style={\"description_width\": \"initial\"},\n",
    "        layout={\"width\": \"max-content\"},\n",
    "    )\n",
    "    display(dropdown)\n",
    "except:\n",
    "    dropdown = None\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a40df34",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdOnly"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "if dropdown:\n",
    "    model_id = dropdown.value\n",
    "else:\n",
    "    # Provide model id as meta-textgeneration-llama-3-1-405b-instruct-fp8 for the instruct variant\n",
    "    model_id = \"meta-textgeneration-llama-3-1-405b-fp8\"\n",
    "model_version = \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0a0388",
   "metadata": {},
   "source": [
    "## 2. Deploy model\n",
    "\n",
    "Create a `JumpStartModel` object, which initializes default model configurations conditioned on the selected instance type. JumpStart already sets a default instance type, but you can deploy the model on other instance types by passing `instance_type` to the `JumpStartModel` class. The deployment requires single ml.p5.48xlarge EC2 instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85a2a8e5-789f-4041-9927-221257126653",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "\n",
    "model = JumpStartModel(model_id=model_id, model_version=model_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1259ad4f",
   "metadata": {},
   "source": [
    "You can now deploy the model using SageMaker JumpStart. If the selected model is gated, you will need to accept the end-user license agreement (EULA) prior to deployment. This is accomplished by providing the `accept_eula=True` argument to the `deploy` method. The deployment might take few minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f3b42ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor = model.deploy(\n",
    "    accept_eula=False\n",
    ")  # please change `accept_eula` to be `true` to accept EULA."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10cf06c",
   "metadata": {},
   "source": [
    "### Invoke the endpoint\n",
    "\n",
    "This section demonstrates how to invoke the endpoint using example payloads that are retrieved programmatically from the `JumpStartModel` object. You can replace these example payloads with your own payloads. Inference payload parameters are provided as below.\n",
    "\n",
    "* **max_new_tokens:** Model generates text until the output length (excluding the input context length) reaches max_new_tokens. If specified, it must be a positive integer.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "\n",
    "You may specify any subset of the parameters mentioned above while invoking an endpoint. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6728120a-0f1f-41d1-998b-172ce1f4e594",
   "metadata": {},
   "source": [
    "JumpStart stores model-specific default example payloads in its SDK. You can retrieve and view them using following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbb364d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "example_payloads = model.retrieve_all_examples()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf5899c8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for payload in example_payloads:\n",
    "    response = predictor.predict(payload.body)\n",
    "    print(\"Input:\\n\", payload.body[payload.prompt_key])\n",
    "    print(\"Output:\\n\", response)\n",
    "    print(\"\\n===============\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90d44de0-67c0-4cc6-939f-ee2cd3ef717a",
   "metadata": {},
   "source": [
    "## 3. Fine-tune model\n",
    "\n",
    "You can fine-tune on the dataset with domain adaptation format or fine-tuning with chat / conversational dataset. Please find more details in the section [Dataset formatting instruction for training](#1.-Dataset-formatting-instruction-for-training). In this demo, we will use a subset of OpenAssistant's TOP-1 Conversation Threads as an example dataset for chat fine-tuning. It can be downloaded from [here](https://huggingface.co/datasets/OpenAssistant/oasst_top1_2023-08-25). It contains roughly 13,000 samples of conversations between the Assistant and the user. \n",
    "\n",
    "\n",
    "Training data should be formatted in JSON lines (.jsonl) format, where each line is a dictionary representing a set of conversations. Here is an example of a line in the training file. The key has to be one of `dialog`, `messages`, and `conversations` to be compatible with `chatml` format in HuggingFace SFTTrainer.\n",
    "\n",
    "```\n",
    "{\"dialog\": [{\"content\":\"what is the height of the empire state building\",\"role\":\"user\"},{\"content\":\"381 meters, or 1,250 feet, is the height of the Empire State Building. If you also account for the antenna, it brings up the total height to 443 meters, or 1,454 feet\",\"role\":\"assistant\"},{\"content\":\"Some people need to pilot an aircraft above it and need to know.\\nSo what is the answer in feet?\",\"role\":\"user\"},{\"content\":\"1454 feet\",\"role\":\"assistant\"}]}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58414e0d-4b8a-43ba-ae4e-6914758fec62",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import re\n",
    "\n",
    "# Load the dataset\n",
    "dataset = load_dataset(\"OpenAssistant/oasst_top1_2023-08-25\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d165680d-77b8-4563-80ce-f232db9c04cd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset[\"train\"][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8412760b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function to transform the data\n",
    "def transform_conversation(example):\n",
    "    conversation_text = example[\"text\"]\n",
    "\n",
    "    segments = re.split(\"<\\|im_start\\|>|<\\|im_end\\|>\", conversation_text)\n",
    "    reformatted_segments = []\n",
    "    dialog_list = []\n",
    "\n",
    "    # Iterate over pairs of segments\n",
    "    for i in range(1, len(segments) - 1, 4):\n",
    "        human_text = segments[i].strip().replace(\"user\", \"\").strip()\n",
    "\n",
    "        # Check if there is a corresponding assistant segment before processing\n",
    "        if i + 1 < len(segments):\n",
    "            assistant_text = segments[i + 2].strip().replace(\"assistant\", \"\").strip()\n",
    "            dialog_list.append({\"role\": \"user\", \"content\": human_text})\n",
    "            dialog_list.append({\"role\": \"assistant\", \"content\": assistant_text})\n",
    "\n",
    "        else:\n",
    "            dialog_list.append({\"role\": \"user\", \"content\": human_text})\n",
    "    return {\"dialog\": dialog_list}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e14716f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed_dataset = dataset.map(transform_conversation).remove_columns(\"text\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15213bc3-d3c5-4326-83ce-5c68d292b507",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "transformed_dataset[\"train\"].select(range(5000)).to_json(\"train.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f414ec2e-adf3-449f-af36-9f2ce649c106",
   "metadata": {},
   "source": [
    "### Upload dataset to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "729440e7-6a9c-4234-851e-0487f5f7b107",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"/{default_bucket_prefix}\"\n",
    "\n",
    "local_data_file = \"train.jsonl\"\n",
    "train_data_location = f\"s3://{output_bucket}{default_bucket_prefix_path}/oasst_top1\"\n",
    "S3Uploader.upload(local_data_file, train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd38cde-f6f9-48e4-afa4-991772339331",
   "metadata": {},
   "source": [
    "Retrieve and customize hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5db89ec-a1c7-4b1c-a64b-f348c3f193cb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "my_hyperparameters = hyperparameters.retrieve_default(\n",
    "    model_id=model_id, model_version=model_version\n",
    ")\n",
    "\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "725dec62-06bf-46a4-83cb-258b55695be3",
   "metadata": {},
   "source": [
    "Underlying the training scripts, JumpStart leverages [HuggingFace SFTTrainer](https://huggingface.co/docs/trl/sft_trainer) with [QLoRA](https://arxiv.org/abs/2305.14314).\n",
    "\n",
    "* For chat fine-tuning, specify `chat_dataset` to be `True` \n",
    "* For domain adaptation fine-tuning, specify `chat_dataset` to be `False`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Note. The fine-tuning model artifacts for 405B fine-tuning are in its original precision bf16. After QLoRA fine-tuning, we conducted fp8 quantization ([code source](https://github.com/huggingface/transformers/blob/main/docs/source/en/quantization/fbgemm_fp8.md)) on the trained model artifacts in bf16 to make it deployable on single ml.p5.48xlarge instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2829738-eca5-4bc9-944a-83f35ce55470",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "my_hyperparameters[\"epoch\"] = \"1\"\n",
    "print(my_hyperparameters)\n",
    "\n",
    "hyperparameters.validate(\n",
    "    model_id=model_id, model_version=model_version, hyperparameters=my_hyperparameters\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4774e360-ed5e-4b59-b648-79312575fa49",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "."
     ]
    }
   ],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    hyperparameters=my_hyperparameters,\n",
    "    environment={\n",
    "        \"accept_eula\": \"false\"\n",
    "    },  # please change `accept_eula` to be `true` to accept EULA.\n",
    ")\n",
    "\n",
    "estimator.fit({\"training\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081e71de-0eb8-44f4-ad47-82fdb0730662",
   "metadata": {},
   "source": [
    "### Deploy the fine-tuned model\n",
    "Next, we deploy the fine-tuned model. We will compare the performance of fine-tuned and pre-trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "419ba339-bbea-43f2-baec-0654ba1fdf03",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "finetuned_predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "707f93c8-39bf-4f0e-b589-d930e8e58de8",
   "metadata": {},
   "source": [
    "## 4. Evaluate the pre-trained and fine-tuned model\n",
    "Next, we use the test data to evaluate the performance of the fine-tuned model and compare it with the pre-trained model. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc4cd893",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a gated model, so you will need an HF token to access the model's tokenizer, which can then be applied to the given input.\n",
    "# Once you have the HF token, run this command\n",
    "\n",
    "!huggingface-cli login --token <hf_token>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a172fb54",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-405B-Instruct\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "283d7820",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "\n",
    "def printmd(string):\n",
    "    display(Markdown(string))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2c10f87-e98e-4c09-9361-9b086e76fbb1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_dialog(payload, response, pretrained_response=None):\n",
    "    dialog = payload[0]\n",
    "    for msg in dialog:\n",
    "        printmd(f\"**{msg['role'].upper()}**: {msg['content']}\\n\")\n",
    "    printmd(f\"---**FINE-TUNED RESPONSE**: {response['generated_text']}\\n\")\n",
    "    if pretrained_response is not None:\n",
    "        printmd(f\"---**PRE-TRAINED RESPONSE**: {pretrained_response['generated_text']}\")\n",
    "    printmd(\"\\n==================================\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d2272c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dataset = transformed_dataset[\"test\"]\n",
    "\n",
    "try:\n",
    "    for i, datapoint in enumerate(test_dataset.select(range(10))):\n",
    "        printmd(f\"\\n**============Begin of Example {i+1} ======================**\\n\")\n",
    "        payload = {\n",
    "            \"inputs\": tokenizer.apply_chat_template(\n",
    "                datapoint[\"dialog\"][:-1], tokenize=False, add_generation_prompt=True\n",
    "            ),\n",
    "            \"parameters\": {\"max_new_tokens\": 256, \"top_p\": 0.96, \"temperature\": 0.2},\n",
    "        }\n",
    "        response = finetuned_predictor.predict(payload)\n",
    "        pretrained_response = predictor.predict(payload)\n",
    "        print_dialog([datapoint[\"dialog\"][:-1]], response, pretrained_response)\n",
    "        printmd(\"---**Ground Truth Response:**\")\n",
    "        printmd(\n",
    "            f\"{datapoint['dialog'][-1]['role'].capitalize()}: {datapoint['dialog'][-1]['content']}\"\n",
    "        )\n",
    "        printmd(f\"\\n**============End of Example {i+1} ======================**\\n\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ceef70b-23c4-4040-a508-640f445726af",
   "metadata": {},
   "source": [
    "### Clean up the endpoint\n",
    "Don't forget to clean up resources when finished to avoid unnecessary charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25d2d2f0-9971-4634-8273-1ec556085412",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_predictor()\n",
    "finetuned_predictor.delete_predictor()\n",
    "# Please remove the hf_token from the notebook once completed, as leaving it could result in potential leakage if you upload or share the notebook publicly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92fdaf8f-eec0-4922-9cbd-d95437b2ffda",
   "metadata": {},
   "source": [
    "## Appendix\n",
    "\n",
    "### 1. Dataset formatting instruction for training\n",
    "\n",
    "####  Fine-tune the Model on a New Dataset\n",
    "We currently offer two types of fine-tuning: chat fine-tuning and domain adaption fine-tuning. You can easily switch to one of the training \n",
    "methods by specifying parameter `chat_dataset` being 'True' or 'False'.\n",
    "\n",
    "#### 1.1. Chat fine-tuning\n",
    "\n",
    "\n",
    "The Text generation model can be fine-tuned on the chat dataset, provided that the data is in the expected format. The resulting chat model can be further deployed for inference. Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** A train and an optional validation directory. Train and validation directories should contain one or multiple JSON lines (.jsonl) formatted files. All training data must be in a single folder, however it can be saved in multiple jsonl files. The .jsonl file extension is mandatory.\n",
    "    - The training data must be formatted in a JSON lines (.jsonl) format, where each line is a dictionary representing a single data sample. Each line in the file is a list of conversations between the user and the assistant model. This model only supports 'system', 'user' and 'assistant' roles, starting with 'system', then 'user' and alternating (u/a/u/a/u...).\n",
    "- **Output:**  A trained model that can be deployed for inference.\n",
    "\n",
    "The best model is selected according to the validation loss, calculated at the end of each epoch. If a validation set is not given, an (adjustable) percentage of the training data is automatically split and used for validation.The training data must be formatted in a JSON lines (.jsonl) format, where each line is a dictionary representing a single data sample.   \n",
    "\n",
    "Here is an example of a line in the training file:\n",
    "```\n",
    "{\"dialog\": [{\"content\":\"what is the height of the empire state building\",\"role\":\"user\"},{\"content\":\"381 meters, or 1,250 feet, is the height of the Empire State Building. If you also account for the antenna, it brings up the total height to 443 meters, or 1,454 feet\",\"role\":\"assistant\"},{\"content\":\"Some people need to pilot an aircraft above it and need to know.\\nSo what is the answer in feet?\",\"role\":\"user\"},{\"content\":\"1454 feet\",\"role\":\"assistant\"}]}\n",
    "```\n",
    "\n",
    "\n",
    "#### 1.2. Domain adaptation fine-tuning\n",
    "The Text Generation model can also be fine-tuned on any domain specific dataset. After being fine-tuned on the domain specific dataset, the model\n",
    "is expected to generate domain specific text and solve various NLP tasks in that specific domain with **few shot prompting**.\n",
    "\n",
    "Below are the instructions for how the training data should be formatted for input to the model.\n",
    "\n",
    "- **Input:** A train and an optional validation directory. Each directory contains a CSV/JSON/TXT file. \n",
    "  - For CSV/JSON files, the train or validation data is used from the column called 'text' or the first column if no column called 'text' is found.\n",
    "  - The number of files under train and validation (if provided) should equal to one, respectively. \n",
    "- **Output:** A trained model that can be deployed for inference. \n",
    "\n",
    "Below is an example of a TXT file for fine-tuning the Text Generation model. The TXT file is SEC filings of Amazon from year 2021 to 2022.\n",
    "\n",
    "```Note About Forward-Looking Statements\n",
    "This report includes estimates, projections, statements relating to our\n",
    "business plans, objectives, and expected operating results that are “forward-\n",
    "looking statements” within the meaning of the Private Securities Litigation\n",
    "Reform Act of 1995, Section 27A of the Securities Act of 1933, and Section 21E\n",
    "of the Securities Exchange Act of 1934. Forward-looking statements may appear\n",
    "throughout this report, including the following sections: “Business” (Part I,\n",
    "Item 1 of this Form 10-K), “Risk Factors” (Part I, Item 1A of this Form 10-K),\n",
    "and “Management’s Discussion and Analysis of Financial Condition and Results\n",
    "of Operations” (Part II, Item 7 of this Form 10-K). These forward-looking\n",
    "statements generally are identified by the words “believe,” “project,”\n",
    "“expect,” “anticipate,” “estimate,” “intend,” “strategy,” “future,”\n",
    "“opportunity,” “plan,” “may,” “should,” “will,” “would,” “will be,” “will\n",
    "continue,” “will likely result,” and similar expressions. Forward-looking\n",
    "statements are based on current expectations and assumptions that are subject\n",
    "to risks and uncertainties that may cause actual results to differ materially.\n",
    "We describe risks and uncertainties that could cause actual results and events\n",
    "to differ materially in “Risk Factors,” “Management’s Discussion and Analysis\n",
    "of Financial Condition and Results of Operations,” and “Quantitative and\n",
    "Qualitative Disclosures about Market Risk” (Part II, Item 7A of this Form\n",
    "10-K). Readers are cautioned not to place undue reliance on forward-looking\n",
    "statements, which speak only as of the date they are made. We undertake no\n",
    "obligation to update or revise publicly any forward-looking statements,\n",
    "whether because of new information, future events, or otherwise.\n",
    "GENERAL\n",
    "Embracing Our Future ...\n",
    "```\n",
    "License: [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4998b86",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/generative_ai|sm-jumpstart_foundation_llama_3_1_405b_finetuning.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "env1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}