introduction_to_amazon_algorithms/jumpstart-foundation-models/code-llama-fine-tuning-evaluate-human-eval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e78dc97c",
   "metadata": {},
   "source": [
    "# Fine-tune Code LIama, Deploy and Evaluate the Fine-tuning with [Human-eval Repository](https://github.com/openai/human-eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e567a4",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4e36b9",
   "metadata": {},
   "source": [
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to fine-tune Code LIama, deploy, and evaluate the fine-tuning performance with [human-eval repository](https://github.com/openai/human-eval).\n",
    "\n",
    "Below is the content of the notebook.\n",
    "\n",
    "1. [Setup](#1.-Setup)\n",
    "2. [Deploy model](#2.-Deploy-model)\n",
    "3. [Fine-tune model with LoRA](#3.-Fine-tune-model)\n",
    "4. [Qualitatively evaluate the pre-trained and fine-tuned model](#4.1-Qualitatively-evaluate-the-pre-trained-and-fine-tuned-model)\n",
    "5. [Quantitatively evaluate the pre-trained and fine-tuned model using Human-Eval repository](#4.2-Quantitatively-evaluation-using-Human-Eval-repository)\n",
    "\n",
    "The notebook requires users to specify following variables to start with.\n",
    "* Specify `model_id` (default value: `meta-textgeneration-llama-codellama-7b`)\n",
    "* Specify `accept_eula` argument to be True in `model.deploy()` to accept the end-user license agreement (EULA) before deployment the model in an endpoint, given Code LIama model is gated.\n",
    "* Sepcify `\"accept_eula\": \"true\"` in argument `environment` to accept the end-user license agreement (EULA) before fine-tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af77efb",
   "metadata": {},
   "source": [
    "## 1. Setup\n",
    "First, upgrade to the latest sagemaker SDK to ensure all available models are deployable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b05b931-992e-4526-978d-f03196874a3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --quiet --upgrade sagemaker jmespath datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dbb6b35",
   "metadata": {},
   "source": [
    "Select the desired model to deploy. The provided dropdown filters all text generation models available in SageMaker JumpStart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f625a488",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "\n",
    "\n",
    "try:\n",
    "    dropdown = Dropdown(\n",
    "        options=list_jumpstart_models(\"search_keywords includes Text Generation\"),\n",
    "        value=\"meta-textgeneration-llama-codellama-7b\",\n",
    "        description=\"Select a JumpStart text generation model:\",\n",
    "        style={\"description_width\": \"initial\"},\n",
    "        layout={\"width\": \"max-content\"},\n",
    "    )\n",
    "    display(dropdown)\n",
    "except:\n",
    "    dropdown = None\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a40df34",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdOnly"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "if dropdown:\n",
    "    model_id = dropdown.value\n",
    "else:\n",
    "    model_id = \"meta-textgeneration-llama-codellama-7b\"\n",
    "model_version = \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0a0388",
   "metadata": {},
   "source": [
    "## 2. Deploy model\n",
    "\n",
    "Create a `JumpStartModel` object, which initializes default model configurations conditioned on the selected instance type. JumpStart already sets a default instance type, but you can deploy the model on other instance types by passing `instance_type` to the `JumpStartModel` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85a2a8e5-789f-4041-9927-221257126653",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "\n",
    "model = JumpStartModel(model_id=model_id, model_version=model_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1259ad4f",
   "metadata": {},
   "source": [
    "You can now deploy the model using SageMaker JumpStart. If the selected model is gated, you will need to accept the end-user license agreement (EULA) prior to deployment. This is accomplished by providing the `accept_eula=True` argument to the `deploy` method. The deployment might take few minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f3b42ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = model.deploy(\n",
    "    accept_eula=False\n",
    ")  # please change `accept_eula` to be True to accept EULA."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10cf06c",
   "metadata": {},
   "source": [
    "### Invoke the endpoint\n",
    "\n",
    "This section demonstrates how to invoke the endpoint using example payloads that are retrieved programmatically from the `JumpStartModel` object. You can replace these example payloads with your own payloads."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6728120a-0f1f-41d1-998b-172ce1f4e594",
   "metadata": {},
   "source": [
    "JumpStart stores model-specific default example payloads in its SDK. You can retrieve and view them using following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbb364d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_payloads = model.retrieve_all_examples()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf5899c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import jmespath\n",
    "\n",
    "\n",
    "for payload in example_payloads:\n",
    "    response = predictor.predict(payload.body)\n",
    "    generated_text = jmespath.search(payload.raw_payload[\"output_keys\"][\"generated_text\"], response)\n",
    "    print(\"Input:\\n\", payload.body[payload.prompt_key])\n",
    "    print(\"Output:\\n\", generated_text.strip())\n",
    "    print(\"\\n===============\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90d44de0-67c0-4cc6-939f-ee2cd3ef717a",
   "metadata": {},
   "source": [
    "## 3. Fine-tune model with LoRA\n",
    "\n",
    "### Dataset preparation for instruction fine-tuning\n",
    "\n",
    "The training data must be formatted in a JSON lines (`.jsonl`) format, where each line is a dictionary representing a single data sample. All training data must be in a single folder, however it can be saved in multiple jsonl files. The `.jsonl` file extension is mandatory. The training\n",
    "folder can also contain a `template.json` file describing the input and output formats. If no template file is given, the following template will be used:\n",
    "  ```json\n",
    "  {\n",
    "    \"prompt\": \"{prompt}\",\n",
    "    \"completion\": \"{completion}\"\n",
    "  }\n",
    "  ```\n",
    "\n",
    "In this case, the data in the JSON lines entries must include `prompt` and `completion` fields. If a custom template is provided it must also use `prompt` and `completion` keys to define the input and output templates. Below is a sample custom template:\n",
    "  \n",
    "  ```json\n",
    "{\n",
    "    \"prompt\": \"{system_prompt} \\n\\n### Input: {question}\",\n",
    "    \"completion\": \" {response}\",\n",
    "}\n",
    "  ```\n",
    "Here, each example in the JSON lines must include `system_prompt`, `question` and `response` fields.\n",
    "\n",
    "In this demo, we will use a subset of [Dolphin-coder dataset](https://huggingface.co/datasets/cognitivecomputations/dolphin-coder) in an instruction tuning format. The dataset is available under Apache 2.0 license."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58414e0d-4b8a-43ba-ae4e-6914758fec62",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "\n",
    "dolphin = load_dataset(\"cognitivecomputations/dolphin-coder\", split=\"train\")\n",
    "\n",
    "# We split the dataset into two where test data is used to evaluate at the end.\n",
    "train_and_test_dataset = dolphin.train_test_split(test_size=0.9, seed=0)\n",
    "\n",
    "# Dumping the training data to a local file to be used for training.\n",
    "train_and_test_dataset[\"train\"].to_json(\"train.jsonl\")\n",
    "train_and_test_dataset[\"test\"].select(range(10)).to_json(\"test.jsonl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d165680d-77b8-4563-80ce-f232db9c04cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_and_test_dataset[\"train\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b4bd800-a9b8-4ffb-a043-267ac63afa26",
   "metadata": {},
   "source": [
    "Next, we prepare prompt template used for processing the data in an instruction format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41d30585-5972-488f-a4e6-63b7177a523b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "template = {\n",
    "    \"prompt\": \"\"\"{system_prompt}\n",
    "\n",
    "### Input:\n",
    "{question}\n",
    "\"\"\",\n",
    "    \"completion\": \" {response}\",\n",
    "}\n",
    "with open(\"template.json\", \"w\") as f:\n",
    "    json.dump(template, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f414ec2e-adf3-449f-af36-9f2ce649c106",
   "metadata": {},
   "source": [
    "### Upload dataset to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "729440e7-6a9c-4234-851e-0487f5f7b107",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Uploader\n",
    "import sagemaker\n",
    "import random\n",
    "\n",
    "output_bucket = sagemaker.Session().default_bucket()\n",
    "local_data_file = \"train.jsonl\"\n",
    "train_data_location = f\"s3://{output_bucket}/dolphin_coder_dataset\"\n",
    "S3Uploader.upload(local_data_file, train_data_location)\n",
    "S3Uploader.upload(\"template.json\", train_data_location)\n",
    "print(f\"Training data: {train_data_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd38cde-f6f9-48e4-afa4-991772339331",
   "metadata": {},
   "source": [
    "Retrieve and customize hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5db89ec-a1c7-4b1c-a64b-f348c3f193cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "my_hyperparameters = hyperparameters.retrieve_default(\n",
    "    model_id=model_id, model_version=model_version\n",
    ")\n",
    "\n",
    "print(my_hyperparameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2829738-eca5-4bc9-944a-83f35ce55470",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_hyperparameters[\"epoch\"] = \"1\"\n",
    "print(my_hyperparameters)\n",
    "\n",
    "hyperparameters.validate(\n",
    "    model_id=model_id, model_version=model_version, hyperparameters=my_hyperparameters\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4774e360-ed5e-4b59-b648-79312575fa49",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    hyperparameters=my_hyperparameters,\n",
    "    environment={\n",
    "        \"accept_eula\": \"false\"\n",
    "    },  # please change `accept_eula` to be `true` to accept EULA.\n",
    ")\n",
    "\n",
    "estimator.fit({\"training\": train_data_location})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "081e71de-0eb8-44f4-ad47-82fdb0730662",
   "metadata": {},
   "source": [
    "### Deploy the fine-tuned model\n",
    "Next, we deploy the fine-tuned model. We will compare the performance of fine-tuned and pre-trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "419ba339-bbea-43f2-baec-0654ba1fdf03",
   "metadata": {},
   "outputs": [],
   "source": [
    "finetuned_predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "707f93c8-39bf-4f0e-b589-d930e8e58de8",
   "metadata": {},
   "source": [
    "## 4.1 Qualitatively evaluate the pre-trained and fine-tuned model\n",
    "Next, we use the test data to evaluate the performance of the fine-tuned model and compare it with the pre-trained model. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93236ac6-884a-4eee-87eb-50df164b800c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "test_dataset = load_dataset(\"json\", data_files=\"test.jsonl\")[\"train\"]\n",
    "prompt_inference = template[\"prompt\"]\n",
    "inputs, ground_truth_responses, responses_before_finetuning, responses_after_finetuning = (\n",
    "    [],\n",
    "    [],\n",
    "    [],\n",
    "    [],\n",
    ")\n",
    "\n",
    "\n",
    "def predict_and_print(datapoint):\n",
    "    # For instruction fine-tuning, we insert a special key between input and output\n",
    "    input_output_demarkation_key = \"\\n\\n### Response:\\n\"\n",
    "\n",
    "    payload = {\n",
    "        \"inputs\": prompt_inference.format(\n",
    "            system_prompt=datapoint[\"system_prompt\"], question=datapoint[\"question\"]\n",
    "        )\n",
    "        + input_output_demarkation_key,\n",
    "        \"parameters\": {\"max_new_tokens\": 100},\n",
    "    }\n",
    "    inputs.append(payload[\"inputs\"])\n",
    "    ground_truth_responses.append(datapoint[\"response\"])\n",
    "    pretrained_response = predictor.predict(payload)\n",
    "    responses_before_finetuning.append(pretrained_response[0][\"generated_text\"])\n",
    "    finetuned_response = finetuned_predictor.predict(payload)\n",
    "    responses_after_finetuning.append(finetuned_response[0][\"generated_text\"])\n",
    "\n",
    "\n",
    "try:\n",
    "    for i, datapoint in enumerate(test_dataset.select(range(5))):\n",
    "        predict_and_print(datapoint)\n",
    "\n",
    "    df = pd.DataFrame(\n",
    "        {\n",
    "            \"Inputs\": inputs,\n",
    "            \"Ground Truth\": ground_truth_responses,\n",
    "            \"Response from non-finetuned model\": responses_before_finetuning,\n",
    "            \"Response from fine-tuned model\": responses_after_finetuning,\n",
    "        }\n",
    "    )\n",
    "    display(HTML(df.to_html()))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3b5ab2-9c58-44bb-a993-925dde778df9",
   "metadata": {},
   "source": [
    "## 4.2 Quantitatively evaluate the pre-trained and fine-tuned models using [Human-Eval repository](https://github.com/openai/human-eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "177cf805",
   "metadata": {},
   "source": [
    "Lets now evaluate if our model has improved on the HumanEval metric from OpenAI. HumanEval is a standard benchmark for code generation models that was created using hand written python problems. This version of HumanEval is using python for its language of choice. We will generate solutions to 164 python related questions and then run a test suite on the solutions to generate a score. If you want to read more [here is the official paper.](https://arxiv.org/abs/2107.03374)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9122407-aaf6-4184-9e34-5fb2d05afcae",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip3 install human_eval --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97065e3e-8fcc-48c1-8192-7778d312a281",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from human_eval.evaluation import evaluate_functional_correctness\n",
    "from human_eval.data import write_jsonl, read_problems\n",
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "def generate_one_completion(prompt, predictor):\n",
    "    body = {\"inputs\": prompt, \"parameters\": {\"max_new_tokens\": 384, \"temperature\": 0.2}}\n",
    "\n",
    "    response = predictor.predict(body)\n",
    "\n",
    "    completion = (response[0][\"generated_text\"]).replace(prompt, \"\").split(\"\\n\\n\\n\")[0]\n",
    "    # if prompt is returned from response\n",
    "    completion = completion.replace(\"```\", \"\")\n",
    "    # if markdown code block is created\n",
    "    print(f\"payload: {prompt}\")\n",
    "    print(f\"completion: {completion}\")\n",
    "    return completion\n",
    "\n",
    "\n",
    "# perform HumanEval\n",
    "problems = read_problems()\n",
    "\n",
    "num_samples_per_task = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fad620e6-4a06-4c12-8a7a-90ac84c1bfdc",
   "metadata": {},
   "source": [
    "Generate responses from pre-trained and fine-tuned models for 164 python related questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9de4da2-f359-4bdb-8a7c-159c86747411",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = [\n",
    "    dict(\n",
    "        task_id=task_id, completion=generate_one_completion(problems[task_id][\"prompt\"], predictor)\n",
    "    )\n",
    "    for task_id in tqdm(problems)\n",
    "    for _ in range(num_samples_per_task)\n",
    "]\n",
    "write_jsonl(\"pretrained.jsonl\", samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d67f656-2790-4031-811b-e1c738153c6c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "evaluate_functional_correctness(\"./pretrained.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7e47914",
   "metadata": {},
   "source": [
    "Now lets compare the previous pretrained model to our new fine-tuned model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d3444f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = [\n",
    "    dict(\n",
    "        task_id=task_id,\n",
    "        completion=generate_one_completion(problems[task_id][\"prompt\"], finetuned_predictor),\n",
    "    )\n",
    "    for task_id in tqdm(problems)\n",
    "    for _ in range(num_samples_per_task)\n",
    "]\n",
    "write_jsonl(\"fine-tuned.jsonl\", samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7c75c40",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_functional_correctness(\"./fine-tuned.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ceef70b-23c4-4040-a508-640f445726af",
   "metadata": {},
   "source": [
    "### Clean up the endpoint\n",
    "Don't forget to clean up resources when finished to avoid unnecessary charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25d2d2f0-9971-4634-8273-1ec556085412",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_predictor()\n",
    "finetuned_predictor.delete_predictor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4998b86",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|code-llama-fine-tuning-evaluate-human-eval.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}