 archived/question_answering_pinecone_llama-2_jumpstart.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/aws/sagemaker/sagemaker-pinecone-rag.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieval-Augmented Generation: Question Answering using LLama-2, Pinecone & Custom Dataset\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we will demonstrate how to use [**Llama-2-7b**](https://ai.meta.com/llama/) to answer questions using a library of documents as a reference, by using document embeddings and retrieval. The embeddings are generated from **MiniLM** embedding model and retrieved from [**Pinecone Vector Database**](https://www.pinecone.io/). \n",
    "Access to a Pinecone environment is a prerequisite to run this notebook fully. \n",
    "\n",
    "**You can start by using the [Free Tier on Pinecone](https://www.pinecone.io/pricing/). This notebook serves a template such that you can easily replace the example dataset by your own to build a custom question and asnwering application.**\n",
    "\n",
    "To perform inference on the [Llama models](https://ai.meta.com/llama/), you need to pass custom_attributes='accept_eula=true' as part of header. This means you have read and accept the end-user-license-agreement (EULA) of the model. EULA can be found in model card description or from this [webpage](https://ai.meta.com/resources/models-and-libraries/llama-downloads/). By default, this notebook sets custom_attributes='accept_eula=false', so all inference requests will fail until you explicitly change this custom attribute.\n",
    "\n",
    "Note: Custom_attributes used to pass EULA are key/value pairs. The key and value are separated by '=' and pairs are separated by ';'. If the user passes the same key more than once, the last value is kept and passed to the script handler (i.e., in this case, used for conditional logic). For example, if 'accept_eula=false; accept_eula=true' is passed to the server, then 'accept_eula=true' is kept and passed to the script handler."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Deploy Llama-2 7 Billion Chat Model in SageMaker JumpStart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install -qU \\\n",
    "    sagemaker \\\n",
    "    pinecone-client==2.2.1 \\\n",
    "    ipywidgets==7.0.0"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin, we will initialize all of the SageMaker session variables we'll need to use throughout the walkthrough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "my_model = JumpStartModel(model_id = \"meta-textgeneration-llama-2-7b-f\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a `ml.g5.4xlarge` instance to deploy our Llama-2-7 billion model. We can find pricing for all instances [here](https://aws.amazon.com/sagemaker/pricing/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!"
     ]
    }
   ],
   "source": [
    "predictor = my_model.deploy(\n",
    "    initial_instance_count=1, instance_type=\"ml.g5.4xlarge\", endpoint_name=\"llama-2-generator\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Ask a question to LLM without providing the context\n",
    "\n",
    "To better illustrate why we need retrieval-augmented generation (RAG) based approach to solve the question and anwering problem. Let's directly ask the model a question and see how they respond."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"Which instances can I use with Managed Spot Training in SageMaker?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' Based on the context provided, Managed Spot Training in SageMaker allows you to use the following instances:\\n\\n* m5.xlarge\\n* m5.2xlarge\\n* m5.4xlarge\\n* m5.8xlarge\\n* m5.16x'"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/\n",
    "\n",
    "prompt = \"\"\"Answer the following QUESTION based on the CONTEXT\n",
    "given. If you do not know the answer and the CONTEXT doesn't\n",
    "contain the answer truthfully say \"I don't know\n",
    "\n",
    "ANSWER:\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "payload = {\n",
    "    \"inputs\":  \n",
    "      [\n",
    "        [\n",
    "         {\"role\": \"system\", \"content\": prompt},\n",
    "         {\"role\": \"user\", \"content\": question},\n",
    "        ]   \n",
    "      ],\n",
    "   \"parameters\":{\"max_new_tokens\": 64, \"top_p\": 0.9, \"temperature\": 0.6, \"return_full_text\": False}\n",
    "}\n",
    "\n",
    "out = predictor.predict(payload, custom_attributes='accept_eula=true')\n",
    "out[0]['generation']['content']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the generated answer is wrong or doesn't make much sense. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3. Improve the answer to the same question using **prompt engineering** with insightful context\n",
    "\n",
    "\n",
    "To better answer the question well, we provide extra contextual information, combine it with a prompt, and send it to model together with the question. Below is an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "context = \"\"\"Managed Spot Training can be used with all instances\n",
    "supported in Amazon SageMaker. Managed Spot Training is supported\n",
    "in all AWS Regions where Amazon SageMaker is currently available.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Input]: Which instances can I use with Managed Spot Training in SageMaker?\n",
      "[Output]:  Based on the given context, you can use Managed Spot Training with all instances supported in Amazon SageMaker. Therefore, the answer is:\n",
      "\n",
      "All instances supported in Amazon SageMaker.\n"
     ]
    }
   ],
   "source": [
    "prompt_template = \"\"\"Answer the following QUESTION based on the CONTEXT\n",
    "given. If you do not know the answer and the CONTEXT doesn't\n",
    "contain the answer truthfully say \"I don't know\".\n",
    "\n",
    "CONTEXT:\n",
    "{context}\n",
    "\n",
    "\n",
    "ANSWER:\n",
    "\"\"\"\n",
    "\n",
    "text_input = prompt_template.replace(\"{context}\", context).replace(\"{question}\", question)\n",
    "\n",
    "payload = {\n",
    "    \"inputs\":  \n",
    "      [\n",
    "        [\n",
    "         {\"role\": \"system\", \"content\": text_input},\n",
    "         {\"role\": \"user\", \"content\": question},\n",
    "        ]   \n",
    "      ],\n",
    "   \"parameters\":{\"max_new_tokens\": 64, \"top_p\": 0.9, \"temperature\": 0.6, \"return_full_text\": False}\n",
    "}\n",
    "\n",
    "out = predictor.predict(payload, custom_attributes='accept_eula=true')\n",
    "generated_text = out[0]['generation']['content']\n",
    "print(f\"[Input]: {question}\\n[Output]: {generated_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if our LLM is capable of following our instructions..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Input]: What color is my desk?\n",
      "[Output]:  I don't know the answer to your question about the color of your desk as it is not related to the context provided, which is about Amazon SageMaker and its supported instances and regions.\n"
     ]
    }
   ],
   "source": [
    "unanswerable_question = \"What color is my desk?\"\n",
    "\n",
    "text_input = prompt_template.replace(\"{context}\", context).replace(\"{question}\", question)\n",
    "\n",
    "payload = {\n",
    "    \"inputs\":  \n",
    "      [\n",
    "        [\n",
    "         {\"role\": \"system\", \"content\": text_input},\n",
    "         {\"role\": \"user\", \"content\": unanswerable_question},\n",
    "        ]   \n",
    "      ],\n",
    "   \"parameters\":{\"max_new_tokens\":256, \"top_p\":0.9, \"temperature\":0.6}\n",
    "}\n",
    "\n",
    "\n",
    "out = predictor.predict(payload, custom_attributes='accept_eula=true')\n",
    "generated_text = out[0]['generation']['content']\n",
    "print(f\"[Input]: {unanswerable_question}\\n[Output]: {generated_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks great! The LLM is following instructions and we've also demonstrated how contexts can help our LLM answer questions accurately. However, we're unlikely to be inserting a context directly into a prompt like this unless we already know the answer — and if we already know the answer why would we be asking the question at all?\n",
    "\n",
    "We need a way of extracting _relevant contexts_ from huge bases of information. For that we need **R**etrieval **A**ugmented **G**eneration (RAG)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Use RAG based approach to identify the correct documents, and use them along with prompt and question to query LLM\n",
    "\n",
    "\n",
    "We plan to use document embeddings to fetch the most relevant documents in our document knowledge library and combine them with the prompt that we provide to LLM.\n",
    "\n",
    "To achieve that, we will do following.\n",
    "\n",
    "* Generate embedings for each of document in the knowledge library with the MiniLM embedding model.\n",
    "* Identify top K most relevant documents based on user query.\n",
    "    * For a query of your interest, generate the embedding of the query using the same embedding model.\n",
    "    * Search the indexes of top K most relevant documents in the embedding space using the SageMaker KNN algorithm.\n",
    "    * Use the indexes to retrieve the corresponded documents.\n",
    "* Combine the retrieved documents with prompt and question and send them into LLM.\n",
    "\n",
    "\n",
    "\n",
    "Note: The retrieved document/text should be large enough to contain enough information to answer a question; but small enough to fit into the LLM prompt -- maximum sequence length of 1024 tokens. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Deploying the model endpoint for Sentence Transformer embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hub_config = {\n",
    "    \"HF_MODEL_ID\": \"sentence-transformers/all-MiniLM-L6-v2\",  # model_id from hf.co/models\n",
    "    \"HF_TASK\": \"feature-extraction\",\n",
    "}\n",
    "\n",
    "huggingface_model = HuggingFaceModel(\n",
    "    env=hub_config,\n",
    "    role=role,\n",
    "    transformers_version=\"4.6\",  # transformers version used\n",
    "    pytorch_version=\"1.7\",  # pytorch version used\n",
    "    py_version=\"py36\",  # python version of the DLC\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we deploy the model as we did earlier for our generative LLM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----!"
     ]
    }
   ],
   "source": [
    "encoder = huggingface_model.deploy(\n",
    "    initial_instance_count=1, instance_type=\"ml.t2.large\", endpoint_name=\"minilm-embedding\"\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then create the embeddings like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "out = encoder.predict({\"inputs\": [\"some text here\", \"some more text goes here too\"]})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will see that we have two outputs (one for each of our input sentences):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(out)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But if we look at each of these outputs we see something strange..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8, 8)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(out[0]), len(out[1])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would expect the embeddings to be of dimensionality *384*, but we're seeing two lists containing _eight_ items each? What is happening here?\n",
    "\n",
    "When we output feature embeddings from the MiniLM model we're actually outputting a single 384-dimensional vector for every _token_ contained in the inputs we provided. Our second text `\"some more text goes here too\"` contains _eight_ tokens, and so this is where the value `8` is coming from.\n",
    "\n",
    "So, if we were to take a look at one of these vectors we should find the dimensionality of `384`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(out[0][0])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect! There's just one problem, how do we transform these eight vector embeddings into a single _sentence embedding_? For this, we simply take the mean value across each vector dimension, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 384)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "embeddings = np.mean(np.array(out), axis=1)\n",
    "embeddings.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have two 384-dimensional vector embeddings, one for each of our input texts. To make our lives easier later, we will wrap this encoding process into a single function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "\n",
    "def embed_docs(docs: List[str]) -> List[List[float]]:\n",
    "    out = encoder.predict({\"inputs\": docs})\n",
    "    embeddings = np.mean(np.array(out), axis=1)\n",
    "    return embeddings.tolist()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Generate embeddings for each of document in the knowledge library with the Sentence Transformer model.\n",
    "\n",
    "For the purpose of the demo we will use [Amazon SageMaker FAQs](https://aws.amazon.com/sagemaker/faqs/) as knowledge library. The data are formatted in a CSV file with two columns Question and Answer. We use **only** the Answer column as the documents of knowledge library, from which relevant documents are retrieved based on a query. \n",
    "\n",
    "**Each row in the CSV format dataset corresponds to a textual document. \n",
    "We will iterate each document to get its embedding vector via the MiniLM embedding model. \n",
    "For your purpose, you can replace the example dataset of your own to build a custom question and answering application.**\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we download the dataset from our S3 bucket to the local."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_path = f\"s3://jumpstart-cache-prod-us-east-2/training-datasets/Amazon_SageMaker_FAQs/Amazon_SageMaker_FAQs.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://jumpstart-cache-prod-us-east-2/training-datasets/Amazon_SageMaker_FAQs/Amazon_SageMaker_FAQs.csv to ./Amazon_SageMaker_FAQs.csv\n"
     ]
    }
   ],
   "source": [
    "# Downloading the Database\n",
    "!aws s3 cp $s3_path Amazon_SageMaker_FAQs.csv"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the dataset with Pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Question</th>\n",
       "      <th>Answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is Amazon SageMaker?</td>\n",
       "      <td>Amazon SageMaker is a fully managed service to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>In which Regions is Amazon SageMaker available...</td>\n",
       "      <td>For a list of the supported Amazon SageMaker A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is the service availability of Amazon Sag...</td>\n",
       "      <td>Amazon SageMaker is designed for high availabi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How does Amazon SageMaker secure my code?</td>\n",
       "      <td>Amazon SageMaker stores code in ML storage vol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What security measures does Amazon SageMaker h...</td>\n",
       "      <td>Amazon SageMaker ensures that ML model artifac...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            Question  \\\n",
       "0                          What is Amazon SageMaker?   \n",
       "1  In which Regions is Amazon SageMaker available...   \n",
       "2  What is the service availability of Amazon Sag...   \n",
       "3          How does Amazon SageMaker secure my code?   \n",
       "4  What security measures does Amazon SageMaker h...   \n",
       "\n",
       "                                              Answer  \n",
       "0  Amazon SageMaker is a fully managed service to...  \n",
       "1  For a list of the supported Amazon SageMaker A...  \n",
       "2  Amazon SageMaker is designed for high availabi...  \n",
       "3  Amazon SageMaker stores code in ML storage vol...  \n",
       "4  Amazon SageMaker ensures that ML model artifac...  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df_knowledge = pd.read_csv(\"Amazon_SageMaker_FAQs.csv\", header=None, names=[\"Question\", \"Answer\"])\n",
    "df_knowledge.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop the `Question` column since it is not used in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Answer</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Amazon SageMaker is a fully managed service to...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>For a list of the supported Amazon SageMaker A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Amazon SageMaker is designed for high availabi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amazon SageMaker stores code in ML storage vol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Amazon SageMaker ensures that ML model artifac...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              Answer\n",
       "0  Amazon SageMaker is a fully managed service to...\n",
       "1  For a list of the supported Amazon SageMaker A...\n",
       "2  Amazon SageMaker is designed for high availabi...\n",
       "3  Amazon SageMaker stores code in ML storage vol...\n",
       "4  Amazon SageMaker ensures that ML model artifac..."
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_knowledge.drop([\"Question\"], axis=1, inplace=True)\n",
    "df_knowledge.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Next we can initialize our connection to **Pinecone**. To do this we need a [free API key](https://app.pinecone.io)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    }
   ],
   "source": [
    "import pinecone\n",
    "import os\n",
    "\n",
    "# add Pinecone API key from app.pinecone.io\n",
    "api_key = os.environ.get(\"PINECONE_API_KEY\") or \"YOUR_API_KEY\"\n",
    "# set Pinecone environment - find next to API key in console\n",
    "env = os.environ.get(\"PINECONE_ENVIRONMENT\") or \"YOUR_ENV\"\n",
    "\n",
    "pinecone.init(api_key=api_key, environment=env)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List all present indexes associated with your key, should be empty on the first run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['jumpstart-minilm-l6',\n",
       " 'retrieval-augmentation-aws-6j',\n",
       " 'retrieval-augmentation-aws']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pinecone.list_indexes()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create a new index called `retrieval-augmentation-aws`. It's important that we align the index `dimension` and `metric` parameters with those required by the MiniLM model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "index_name = \"llama-2-7b-example\"\n",
    "\n",
    "if index_name in pinecone.list_indexes():\n",
    "    pinecone.delete_index(index_name)\n",
    "\n",
    "pinecone.create_index(name=index_name, dimension=embeddings.shape[1], metric=\"cosine\")\n",
    "# wait for index to finish initialization\n",
    "while not pinecone.describe_index(index_name).status[\"ready\"]:\n",
    "    time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['jumpstart-minilm-l6',\n",
       " 'llama-2-7b-example',\n",
       " 'retrieval-augmentation-aws-6j',\n",
       " 'retrieval-augmentation-aws']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pinecone.list_indexes()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we upsert the data, we will do this in batches of `128`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aa50e0ad54eb4b35b685f27b27874c5d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "A Jupyter Widget"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from tqdm.auto import tqdm\n",
    "\n",
    "batch_size = 2  # can increase but needs larger instance size otherwise instance runs out of memory\n",
    "vector_limit = 1000\n",
    "\n",
    "answers = df_knowledge[:vector_limit]\n",
    "index = pinecone.Index(index_name)\n",
    "\n",
    "for i in tqdm(range(0, len(answers), batch_size)):\n",
    "    # find end of batch\n",
    "    i_end = min(i + batch_size, len(answers))\n",
    "    # create IDs batch\n",
    "    ids = [str(x) for x in range(i, i_end)]\n",
    "    # create metadata batch\n",
    "    metadatas = [{\"text\": text} for text in answers[\"Answer\"][i:i_end]]\n",
    "    # create embeddings\n",
    "    texts = answers[\"Answer\"][i:i_end].tolist()\n",
    "    embeddings = embed_docs(texts)\n",
    "    # create records list for upsert\n",
    "    records = zip(ids, embeddings, metadatas)\n",
    "    # upsert to Pinecone\n",
    "    index.upsert(vectors=records)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 384,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {'': {'vector_count': 154}},\n",
       " 'total_vector_count': 154}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check number of records in the index\n",
    "index.describe_index_stats()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.3 Combine the retrieved documents, prompt, and question to query the LLM"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're ready begin querying our LLM with a **R**etrieval **A**ugmented **G**eneration (RAG) pipeline. Let's see how this will work step-by-step first."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create our _query embedding_ and use it to query Pinecone:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'matches': [{'id': '90',\n",
       "              'metadata': {'text': 'Managed Spot Training can be used with all '\n",
       "                                   'instances supported in Amazon '\n",
       "                                   'SageMaker.\\r\\n'},\n",
       "              'score': 0.881181657,\n",
       "              'values': []}],\n",
       " 'namespace': ''}"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract embeddings for the questions\n",
    "query_vec = embed_docs(question)[0]\n",
    "\n",
    "# query pinecone\n",
    "res = index.query(query_vec, top_k=1, include_metadata=True)\n",
    "\n",
    "# show the results\n",
    "res"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get multiple relevant contexts here. We can use these to contruct a single `context` to feed into our LLM prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "contexts = [match.metadata[\"text\"] for match in res.matches]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "max_section_len = 1000\n",
    "separator = \"\\n\"\n",
    "\n",
    "\n",
    "def construct_context(contexts: List[str]) -> str:\n",
    "    chosen_sections = []\n",
    "    chosen_sections_len = 0\n",
    "\n",
    "    for text in contexts:\n",
    "        text = text.strip()\n",
    "        # Add contexts until we run out of space.\n",
    "        chosen_sections_len += len(text) + 2\n",
    "        if chosen_sections_len > max_section_len:\n",
    "            break\n",
    "        chosen_sections.append(text)\n",
    "    concatenated_doc = separator.join(chosen_sections)\n",
    "    print(\n",
    "        f\"With maximum sequence length {max_section_len}, selected top {len(chosen_sections)} document sections: \\n{concatenated_doc}\"\n",
    "    )\n",
    "    return concatenated_doc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With maximum sequence length 1000, selected top 1 document sections: \n",
      "Managed Spot Training can be used with all instances supported in Amazon SageMaker.\n"
     ]
    }
   ],
   "source": [
    "context_str = construct_context(contexts=contexts)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would then feed this `context_str` into our LLama-2 prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def create_payload(question, context_str) -> dict:\n",
    "    prompt_template = \"\"\"Answer the following QUESTION based on the CONTEXT\n",
    "    given. If you do not know the answer and the CONTEXT doesn't\n",
    "    contain the answer truthfully say \"I don't know\".\n",
    "\n",
    "    CONTEXT:\n",
    "    {context}\n",
    "\n",
    "\n",
    "    ANSWER:\n",
    "    \"\"\"\n",
    "\n",
    "    text_input = prompt_template.replace(\"{context}\", context_str).replace(\"{question}\", question)\n",
    "\n",
    "    payload = {\n",
    "        \"inputs\":  \n",
    "          [\n",
    "            [\n",
    "             {\"role\": \"system\", \"content\": text_input},\n",
    "             {\"role\": \"user\", \"content\": question},\n",
    "            ]   \n",
    "          ],\n",
    "       \"parameters\":{\"max_new_tokens\": 256, \"top_p\": 0.9, \"temperature\": 0.6, \"return_full_text\": False}\n",
    "    }\n",
    "    return(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Input]: Which instances can I use with Managed Spot Training in SageMaker?\n",
      "[Output]:  Based on the context provided, you can use Managed Spot Training with all instances supported in Amazon SageMaker. Therefore, the answer is:\n",
      "\n",
      "All instances supported in Amazon SageMaker.\n"
     ]
    }
   ],
   "source": [
    "payload = create_payload(question, context_str)\n",
    "out = predictor.predict(payload, custom_attributes='accept_eula=true')\n",
    "generated_text = out[0]['generation']['content']\n",
    "print(f\"[Input]: {question}\\n[Output]: {generated_text}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's place all of this logic into a single RAG query function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def rag_query(question: str) -> str:\n",
    "    # create query vec\n",
    "    query_vec = embed_docs(question)[0]\n",
    "    # query pinecone\n",
    "    res = index.query(query_vec, top_k=5, include_metadata=True)\n",
    "    # get contexts\n",
    "    contexts = [match.metadata[\"text\"] for match in res.matches]\n",
    "    # build the multiple contexts string\n",
    "    context_str = construct_context(contexts=contexts)\n",
    "    # create our retrieval augmented prompt\n",
    "    payload = create_payload(question, context_str)\n",
    "    # make prediction\n",
    "    out = predictor.predict(payload, custom_attributes='accept_eula=true')\n",
    "    return out[0][\"generation\"][\"content\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now ask the question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With maximum sequence length 1000, selected top 5 document sections: \n",
      "Managed Spot Training can be used with all instances supported in Amazon SageMaker.\n",
      "Managed Spot Training is supported in all AWS Regions where Amazon SageMaker is currently available.\n",
      "Managed Spot Training with Amazon SageMaker lets you train your ML models using Amazon EC2 Spot instances, while reducing the cost of training your models by up to 90%.\n",
      "For a list of the supported Amazon SageMaker AWS Regions, please visit the AWS Regional Services page. Also, for more information, see Regional endpoints in the AWS general reference guide.\n",
      "At launch, we will support all Regions supported by Amazon SageMaker, except the AWS China Regions.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "' Yes, Amazon SageMaker supports spot instances for managed spot training. According to the provided context, Managed Spot Training can be used with all instances supported in Amazon SageMaker, and Managed Spot Training is supported in all AWS Regions where Amazon SageMaker is currently available.\\n\\nTherefore, the answer to your question is:\\n\\nYes, SageMaker supports spot instances in all regions where Amazon SageMaker is available.'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rag_query(\"Does SageMaker support spot instances?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also ask questions about things that are out of context (not contained within our dataset). From this we expect the model to *not* hallucinate and honestly tell us that it does not know the answer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "With maximum sequence length 1000, selected top 2 document sections: \n",
      "No. Amazon SageMaker operates the compute infrastructure on your behalf, allowing it to perform health checks, apply security patches, and do other routine maintenance. You can also deploy the model artifacts from training with custom inference code in your own hosting environment.\n",
      "Amazon SageMaker Data Wrangler provides a unified experience enabling you to prepare data and seamlessly train a machine learning model in Amazon SageMaker Autopilot. SageMaker Autopilot automatically builds, trains, and tunes the best ML models based on your data. With SageMaker Autopilot, you still maintain full control and visibility of your data and model. You can also use features prepared in SageMaker Data Wrangler with your existing models. You can configure Amazon SageMaker Data Wrangler processing jobs to run as part of your SageMaker training pipeline either by configuring the job in the user interface (UI) or exporting a notebook with the orchestration code.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "' Based on the context provided, the answer is \"Yes, you can deploy a model trained outside of Amazon SageMaker.\"\\n\\nAccording to the text, Amazon SageMaker Data Wrangler provides a unified experience for preparing data and training machine learning models, including models trained outside of SageMaker. This suggests that SageMaker Autopilot can be used to deploy models trained outside of the platform, as long as they are in a format that can be processed by SageMaker.\\n\\nAdditionally, the text states that you can configure Amazon SageMaker Data Wrangler processing jobs to run as part of your SageMaker training pipeline, either through the user interface or by exporting a notebook with the orchestration code. This suggests that you can integrate models trained outside of SageMaker into your SageMaker workflows and deploy them using the platform\\'s infrastructure.\\n\\nTherefore, based on the context provided, the answer is \"Yes, you can deploy a model trained outside of Amazon SageMaker.\"'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rag_query(\"Can I deploy a model trained outside of SageMaker?\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}