 archived/granite-code-instruct.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b2204cd2-3ddd-4fb6-8d0b-a94fa0df92fe",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Deploying Granite Code models in Amazon SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8e46afb",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "559e8037-005b-4130-916c-ae7b5dd12120",
   "metadata": {},
   "source": [
    "The IBM Granite Code models are a family of high-performance, foundational language models pre-trained on over 3 trillion tokens of code and natural language data across 116 programming languages. These models range from 3 billion to 34 billion parameters and come in base and instruction-following variants.\n",
    "\n",
    "What sets the Granite Code models apart is their strong performance on a wide range of code intelligence tasks like code generation, translation, analysis, and refactoring - often outperforming larger open-source models.\n",
    "\n",
    "IBM has released the Granite Code models to open source under the permissive Apache 2.0 license, enabling their use for both research and commercial purposes with no restrictions. The models are available on [Hugging Face](https://huggingface.co/ibm-granite).\n",
    "\n",
    "[Hugging Face](https://huggingface.co/) is a popular open source hub for machine learning (ML) models. AWS and Hugging Face have a partnership that allows a seamless integration through SageMaker with a set of AWS Deep Learning Containers (DLCs) for training and inference in PyTorch or TensorFlow, and Hugging Face estimators and predictors for the SageMaker Python SDK. SageMaker features and capabilities help developers and data scientists get started with natural language processing (NLP) on AWS with ease.\n",
    "\n",
    "In this notebook, we will deploy the Granite models on Amazon SageMaker for accelerating legacy code conversion and modernisation use cases. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9e559f9-e514-4aae-9b15-d4b626f9a94c",
   "metadata": {},
   "source": [
    "## Deploying Granite Code models in Amazon SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5942175-4354-4bec-8e27-6ab4f2c0d541",
   "metadata": {},
   "source": [
    "### Prepare the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8ca3851-308a-4144-a762-ea9f024796f6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install -U sagemaker -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cae3d8a-f29f-4c98-a6bf-b8c386f11ffe",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import sagemaker\n",
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.huggingface import get_huggingface_llm_image_uri\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "account_id = sagemaker_session.account_id()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = sagemaker_session.boto_region_name\n",
    "\n",
    "# Use latest container image (2.0.3) for the Granite models\n",
    "image_uri = \"763104351884.dkr.ecr.{}.amazonaws.com/huggingface-pytorch-tgi-inference:2.3.0-tgi2.0.3-gpu-py310-cu121-ubuntu22.04-v2.0\".format(\n",
    "    region\n",
    ")\n",
    "\n",
    "# print ecr image uri\n",
    "print(f\"llm image uri: {image_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62a91709-b934-4854-bdbc-1337a436f4fc",
   "metadata": {},
   "source": [
    "### Create SageMaker Estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1187dfc-67ab-4bbc-87e5-dfb795cf3d12",
   "metadata": {},
   "source": [
    "Link to the Granite Code models in HuggingFace: https://huggingface.co/ibm-granite"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cff018b9-22d7-4161-9f68-7c03fe3af04f",
   "metadata": {},
   "source": [
    "Next we configure the model object by specifying a unique name, the image_uri for the managed TGI container, and the execution role for the endpoint. Additionally, we specify a number of environment variables including the HF_MODEL_ID which corresponds to the model from the HuggingFace Hub that will be deployed, and the HF_TASK which configures the inference task to be performed by the model.\n",
    "\n",
    "You should also define SM_NUM_GPUS, which specifies the tensor parallelism degree of the model. Tensor parallelism can be used to split the model across multiple GPUs, which is necessary when working with LLMs that are too big for a single GPU. Here, you should set SM_NUM_GPUS to the number of available GPUs on your selected instance type. For example, in this tutorial, we set SM_NUM_GPUS to 4 because our selected instance type ml.g5.12xlarge has 4 available GPUs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca6dd06-16eb-4f89-8418-1343769aee8c",
   "metadata": {},
   "source": [
    "The HuggingFaceModel handles downloading the Granite model and dependencies, packaging them into a Docker container, and deploying to a SageMaker inference endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "975dab7f-88f9-4190-9ea3-69623079ddd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# sagemaker config\n",
    "instance_type = \"ml.g5.12xlarge\"\n",
    "number_of_gpu = 4\n",
    "health_check_timeout = 600\n",
    "\n",
    "# Hub model configuration\n",
    "hub = {\n",
    "    \"HF_MODEL_ID\": \"ibm-granite/granite-20b-code-instruct\",  # since g5.12xlarge has 4 GPU, we are sharding model weights accross 4 GPU's. If you are testing it on g5.2xlarge set this to 1 as it has only 1 GPU\n",
    "    \"SM_NUM_GPUS\": json.dumps(\n",
    "        number_of_gpu\n",
    "    ),  # no effect this varible is only for SM provided TGI container\n",
    "}\n",
    "\n",
    "# create Hugging Face Model Class\n",
    "huggingface_model = HuggingFaceModel(\n",
    "    image_uri=image_uri,\n",
    "    env=hub,\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dad514f6-82e5-458b-8ba8-a5eb8df75caa",
   "metadata": {},
   "source": [
    "After we have created the HuggingFaceModel we deploy it to Amazon SageMaker using the deploy method. We deploy the model with the ml.g5.12xlarge instance type, which has 4 NVIDIA A10G GPUs and 96GB of GPU memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17859d59-5db0-4a70-939e-fbefb54f4a32",
   "metadata": {},
   "outputs": [],
   "source": [
    "# deploy model to SageMaker Inference\n",
    "predictor = huggingface_model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=instance_type,\n",
    "    container_startup_health_check_timeout=health_check_timeout,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed622c3d-11d4-4d74-9338-2f4fc51fd45e",
   "metadata": {},
   "source": [
    "SageMaker will now create our endpoint and deploy the model to it. This can takes a 10-15 minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "390a2f45-e903-497e-81c1-8901d7141a20",
   "metadata": {},
   "source": [
    "### Run inference with the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc503c3-2375-447c-a7c6-19cf2fff1a2f",
   "metadata": {},
   "source": [
    "Now that we have the Granite Code model loaded and deployed to a SageMaker endpoint, we can start generating or converting code. We use the predict method from the predictor to run inference on our endpoint. We can inference with different parameters to impact the generation. Parameters can be defined as in the parameters attribute of the payload."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cd2d9fa-3e47-4493-81b4-2cbea616f7e2",
   "metadata": {},
   "source": [
    "#### Example 1: Code Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d445db8e-3a53-4d9b-8f0a-08f41983d134",
   "metadata": {},
   "source": [
    "In this example, we want to write a function in the Python programming language that reverses a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88c430ec-3bdf-433a-aa15-df336c093cd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_1 = \"\"\"Using the directions below, generate Python code for the specified task.\n",
    "\n",
    "Question:\n",
    "# Write a Python function that prints 'Hello World!' string 'n' times.\n",
    "\n",
    "Answer:\n",
    "def print_n_times(n):\n",
    "    for i in range(n):\n",
    "        print(\"Hello World!\")\n",
    "\n",
    "<end of code>\n",
    "\n",
    "Question:\n",
    "# Write a Python function that reverses the order of letters in a string.\n",
    "# The function named 'reversed' takes the argument 'my_string', which is a string. It returns the string in reverse order.\n",
    "\n",
    "Answer:\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40ce613d-c8da-47a4-9108-b38dc5c30c87",
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters for llm\n",
    "payload = {\n",
    "    \"inputs\": prompt_1,\n",
    "    \"parameters\": {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.6,\n",
    "        \"temperature\": 0.1,\n",
    "        \"top_k\": 50,\n",
    "        \"max_new_tokens\": 300,\n",
    "        \"repetition_penalty\": 1.03,\n",
    "        \"stop\": [\"<end of code>\"],\n",
    "    },\n",
    "}\n",
    "\n",
    "# send request to endpoint\n",
    "response = predictor.predict(payload)\n",
    "\n",
    "print(response[0][\"generated_text\"][len(prompt_1) :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "189840e9-ad83-4b75-8b88-8374aa982caf",
   "metadata": {},
   "source": [
    "The output contains Python code similar to the following snippet:\n",
    "```\n",
    "def reverse_string(my_string):\n",
    "    return my_string[::-1]\n",
    "```\n",
    "Be sure to test the generated code to verify that it works as you expect.\n",
    "\n",
    "For example, if you run `reversed(\"good morning\")`, the result is `gninrom doog`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d45a8b23-56dc-4ecc-b634-d535ee5a56ee",
   "metadata": {},
   "source": [
    "#### Example 2: Code Conversion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "928d3d56-5b9c-4480-aef5-bc0060babd7d",
   "metadata": {},
   "source": [
    "In this example, we want to convert code from one programming language to another. The prompt below converts a code snippet from C++ to Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ca79e50-4a46-484e-9a8c-4420186c317e",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_2 = \"\"\"\n",
    "Question:\n",
    "Translate the following code from C++ to Python.\n",
    "C++:\n",
    "#include \"bits/stdc++.h\"\n",
    "using namespace std;\n",
    "bool isPerfectSquare(long double x) {\n",
    "  long double sr = sqrt(x);\n",
    "  return ((sr - floor(sr)) == 0);\n",
    "}\n",
    "void checkSunnyNumber(int N) {\n",
    "  if (isPerfectSquare(N + 1)) {\n",
    "    cout << \"Yes\n",
    "\";\n",
    "  } else {\n",
    "    cout << \"No\n",
    "\";\n",
    "  }\n",
    "}\n",
    "int main() {\n",
    "  int N = 8;\n",
    "  checkSunnyNumber(N);\n",
    "  return 0;\n",
    "}\n",
    "\n",
    "Answer:\n",
    "Python:\n",
    "from math import *\n",
    " \n",
    "def isPerfectSquare(x):\n",
    "    sr = sqrt(x)\n",
    "    return ((sr - floor(sr)) == 0)\n",
    " \n",
    "def checkSunnyNumber(N):\n",
    "    if (isPerfectSquare(N + 1)):\n",
    "        print(\"Yes\")\n",
    "    else:\n",
    "        print(\"No\")\n",
    " \n",
    "if __name__ == '__main__':\n",
    "    N = 8\n",
    "    checkSunnyNumber(N)\n",
    "\n",
    "<end of code>\n",
    "\n",
    "Question:\n",
    "Translate the following code from C++ to Python.\n",
    "C++:\n",
    "#include <bits/stdc++.h>\n",
    "using namespace std;\n",
    "int countAPs(int S, int D) {\n",
    "  S = S * 2;\n",
    "  int answer = 0;\n",
    "  for (int i = 1; i <= sqrt(S); i++) {\n",
    "    if (S % i == 0) {\n",
    "      if (((S / i) - D * i + D) % 2 == 0)\n",
    "        answer++;\n",
    "      if ((D * i - (S / i) + D) % 2 == 0)\n",
    "        answer++;\n",
    "    }\n",
    "  }\n",
    "  return answer;\n",
    "}\n",
    "int main() {\n",
    "  int S = 12, D = 1;\n",
    "  cout << countAPs(S, D);\n",
    "  return 0;\n",
    "}\n",
    "\n",
    "Answer:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92834275-bb04-4277-a8ff-9b92df7aa0a3",
   "metadata": {},
   "source": [
    "You can send the prompt to the Granite Code model loaded and deployed to the SageMaker endpoint, and adjust the following hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "784a69cd-33ec-4a6a-a60b-26b83c452b1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters for llm\n",
    "payload_2 = {\n",
    "    \"inputs\": prompt_2,\n",
    "    \"parameters\": {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.6,\n",
    "        \"temperature\": 0.1,\n",
    "        \"top_k\": 50,\n",
    "        \"max_new_tokens\": 1000,\n",
    "        \"repetition_penalty\": 1.03,\n",
    "        \"stop\": [\"<end of code>\"],\n",
    "    },\n",
    "}\n",
    "\n",
    "# send request to endpoint\n",
    "response_2 = predictor.predict(payload_2)\n",
    "\n",
    "print(response_2[0][\"generated_text\"][len(prompt_2) :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f4db242-915e-4504-bf72-9de2f521f054",
   "metadata": {},
   "source": [
    "The output contains Python code similar to the following snippet:\n",
    "```\n",
    "Python:\n",
    "from math import *\n",
    " \n",
    "def countAPs(S, D):\n",
    "    S = S * 2\n",
    "    answer = 0\n",
    "    for i in range(1, int(sqrt(S)) + 1):\n",
    "        if S % i == 0:\n",
    "            if ((S // i) - D * i + D) % 2 == 0:\n",
    "                answer += 1\n",
    "            if (D * i - (S // i) + D) % 2 == 0:\n",
    "                answer += 1\n",
    "    return answer\n",
    " \n",
    "if __name__ == '__main__':\n",
    "    S = 12\n",
    "    D = 1\n",
    "    print(countAPs(S, D))\n",
    "\n",
    "```\n",
    "Be sure to test the generated code to verify that it works as you expect."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a2777e9",
   "metadata": {},
   "source": [
    "#### Example 3: Code Conversion (C to Java)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f03b00a8",
   "metadata": {},
   "source": [
    "In this example, you want to convert code from one programming language to another. The prompt below converts a code snippet from C to Java."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce109421",
   "metadata": {},
   "source": [
    "Specifically, we cover common programming constructs like linked lists and file I/O operations. The C code is converted to Java while preserving the functionality and logic. In the Java code, we utilize classes, objects, and Java-specific APIs like `FileWriter` and `BufferedReader` to achieve similar results as the C code.\n",
    "\n",
    "* In the first example, the C code implements a singly linked list data structure. It defines a Node struct containing an integer data value and a pointer to the next node. The code provides functions to create a new node, add a node to the end of the list, and print the list.\n",
    "* In the second example, the C code demonstrates how to write data to a file and then read the data back from the file. It opens a file named \"example.txt\" in write mode, writes a string to the file, and closes the file. Then, it opens the same file in read mode, reads the contents into a buffer, and prints the buffer to the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51ebe880",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_3 = \"\"\"\n",
    "Question:\n",
    "Translate the following code from C to Java.\n",
    "\n",
    "C Code:\n",
    "\n",
    "```c\n",
    "#include <stdio.h>\n",
    "#include <stdlib.h>\n",
    "\n",
    "typedef struct Node {\n",
    "    int data;\n",
    "    struct Node* next;\n",
    "} Node;\n",
    "\n",
    "Node* createNode(int data) {\n",
    "    Node* newNode = (Node*)malloc(sizeof(Node));\n",
    "    newNode->data = data;\n",
    "    newNode->next = NULL;\n",
    "    return newNode;\n",
    "}\n",
    "\n",
    "void addNode(Node** head, int data) {\n",
    "    Node* newNode = createNode(data);\n",
    "    if (*head == NULL) {\n",
    "        *head = newNode;\n",
    "        return;\n",
    "    }\n",
    "    Node* temp = *head;\n",
    "    while (temp->next != NULL) {\n",
    "        temp = temp->next;\n",
    "    }\n",
    "    temp->next = newNode;\n",
    "}\n",
    "\n",
    "void printList(Node* head) {\n",
    "    Node* temp = head;\n",
    "    while (temp != NULL) {\n",
    "        printf(\"%d \", temp->data);\n",
    "        temp = temp->next;\n",
    "    }\n",
    "    printf(\"\\n\");\n",
    "}\n",
    "\n",
    "int main() {\n",
    "    Node* head = NULL;\n",
    "    addNode(&head, 1);\n",
    "    addNode(&head, 2);\n",
    "    addNode(&head, 3);\n",
    "    printList(head);\n",
    "    return 0;\n",
    "}\n",
    "```\n",
    "\n",
    "Java Code:\n",
    "\n",
    "```java\n",
    "class Node {\n",
    "    int data;\n",
    "    Node next;\n",
    "\n",
    "    Node(int data) {\n",
    "        this.data = data;\n",
    "        next = null;\n",
    "    }\n",
    "}\n",
    "\n",
    "class LinkedList {\n",
    "    Node head;\n",
    "\n",
    "    void addNode(int data) {\n",
    "        Node newNode = new Node(data);\n",
    "        if (head == null) {\n",
    "            head = newNode;\n",
    "            return;\n",
    "        }\n",
    "        Node temp = head;\n",
    "        while (temp.next != null) {\n",
    "            temp = temp.next;\n",
    "        }\n",
    "        temp.next = newNode;\n",
    "    }\n",
    "\n",
    "    void printList() {\n",
    "        Node temp = head;\n",
    "        while (temp != null) {\n",
    "            System.out.print(temp.data + \" \");\n",
    "            temp = temp.next;\n",
    "        }\n",
    "        System.out.println();\n",
    "    }\n",
    "\n",
    "    public static void main(String[] args) {\n",
    "        LinkedList list = new LinkedList();\n",
    "        list.addNode(1);\n",
    "        list.addNode(2);\n",
    "        list.addNode(3);\n",
    "        list.printList();\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "<end of code>\n",
    "\n",
    "Question:\n",
    "Translate the following code from C to Java.\n",
    "\n",
    "C Code:\n",
    "\n",
    "```c\n",
    "#include <stdio.h>\n",
    "#include <stdlib.h>\n",
    "\n",
    "int main() {\n",
    "    FILE* file = fopen(\"example.txt\", \"w\");\n",
    "    if (file == NULL) {\n",
    "        printf(\"Error opening file!\\n\");\n",
    "        return 1;\n",
    "    }\n",
    "\n",
    "    fprintf(file, \"This is an example of writing to a file.\\n\");\n",
    "    fclose(file);\n",
    "\n",
    "    file = fopen(\"example.txt\", \"r\");\n",
    "    if (file == NULL) {\n",
    "        printf(\"Error opening file!\\n\");\n",
    "        return 1;\n",
    "    }\n",
    "\n",
    "    char buffer[100];\n",
    "    while (fgets(buffer, sizeof(buffer), file) != NULL) {\n",
    "        printf(\"%s\", buffer);\n",
    "    }\n",
    "\n",
    "    fclose(file);\n",
    "    return 0;\n",
    "}\n",
    "```\n",
    "Answer:\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4970aaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters for llm\n",
    "payload_3 = {\n",
    "    \"inputs\": prompt_3,\n",
    "    \"parameters\": {\n",
    "        \"do_sample\": True,\n",
    "        \"top_p\": 0.6,\n",
    "        \"temperature\": 0.1,\n",
    "        \"top_k\": 50,\n",
    "        \"max_new_tokens\": 1000,\n",
    "        \"repetition_penalty\": 1.03,\n",
    "        \"stop\": [\"<end of code>\"],\n",
    "    },\n",
    "}\n",
    "\n",
    "# send request to endpoint\n",
    "response_3 = predictor.predict(payload_3)\n",
    "\n",
    "print(response_3[0][\"generated_text\"][len(prompt_3) :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "047e205e",
   "metadata": {},
   "source": [
    "The output contains Java code similar to the following snippet:\n",
    "```\n",
    "import java.io.*;\n",
    "\n",
    "public class FileExample {\n",
    "    public static void main(String[] args) {\n",
    "        try (FileWriter writer = new FileWriter(\"example.txt\");\n",
    "             BufferedReader reader = new BufferedReader(new FileReader(\"example.txt\"))) {\n",
    "\n",
    "            writer.write(\"This is an example of writing to a file.\");\n",
    "\n",
    "            String line;\n",
    "            while ((line = reader.readLine())!= null) {\n",
    "                System.out.println(line);\n",
    "            }\n",
    "        } catch (IOException e) {\n",
    "            System.out.println(\"An error occurred: \" + e.getMessage());\n",
    "        }\n",
    "    }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c943e029-2633-4f74-8440-fe48d1348bb7",
   "metadata": {},
   "source": [
    "### Clean up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1b7ddde-e511-40e2-8313-55c3aa6c85e7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa86cd3",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/inference|generativeai|huggingfacetgi|ibm-granite|granite-code-instruct.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0f2d13e-9f8e-4083-a370-2f99d318a066",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}