 archived/text-embedding-sentence-similarity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9fb44f6d",
   "metadata": {},
   "source": [
    "# Amazon SageMaker Jumpstart - Text Embedding & Sentence Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3be16ff",
   "metadata": {},
   "source": [
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c2bc55e",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "Welcome to [Amazon SageMaker Jumpstart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use Amazon SageMaker Jumpstart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html#use-prebuilt-models-with-sagemaker-jumpstart). \n",
    "\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK for Text Embedding and Sentence Similarity. Sentence similarity involves assessing the likeness between two pieces of text. Models designed for sentence similarity transform input texts into vectors or embeddings, capturing semantic details, and then compute their proximity or similarity. We demonstrate the following here:\n",
    "\n",
    "- How to run inference on a Text Embedding model.\n",
    "- How to find the nearest neighbors for an input sentence with your own dataset\n",
    "- How to run the batch Transform\n",
    "\n",
    "The following text embedding models are available currently in the SageMaker Jumpstart-\n",
    "\n",
    "| Model Name                                                    | JumpStart Model ID                                   |\n",
    "|---------------------------------------------------------------|------------------------------------------------------|\n",
    "| [bge-large-en](https://huggingface.co/BAAI/bge-large-en)      | huggingface-sentencesimilarity-bge-large-en          |\n",
    "| [bge-base-en](https://huggingface.co/BAAI/bge-base-en)        | huggingface-sentencesimilarity-bge-base-en           |\n",
    "| [gte-large](https://huggingface.co/thenlper/gte-large)        | huggingface-sentencesimilarity-gte-large             |\n",
    "| [gte-base](https://huggingface.co/thenlper/gte-base)          | huggingface-sentencesimilaritygte-base               |\n",
    "| [e5-large-v2](https://huggingface.co/intfloat/e5-large-v2)    | huggingface-sentencesimilarity-e5-large-v2           |\n",
    "| [bge-small-en](https://huggingface.co/BAAI/bge-small-en)      | huggingface-sentencesimilarity-bge-small-en          |\n",
    "| [e5-base-v2](https://huggingface.co/intfloat/e5-base-v2)      | huggingface-sentencesimilarity-e5-base-v2            |\n",
    "| [multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | huggingface-sentencesimilarity-multilingual-e5-large |\n",
    "| [e5-large](https://huggingface.co/intfloat/e5-large)          | huggingface-sentencesimilarity-e5-large              |\n",
    "| [gte-small](https://huggingface.co/thenlper/gte-small)        | huggingface-sentencesimilarity-gte-small             |\n",
    "| [e5-base](https://huggingface.co/intfloat/e5-base)            | huggingface-sentencesimilarity-e5-base               |\n",
    "| [e5-small-v2](https://huggingface.co/intfloat/e5-small-v2)    | huggingface-sentencesimilarity-e5-small-v2           |\n",
    "| [multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | huggingface-sentencesimilarity-multilingual-e5-base  |\n",
    "| [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | huggingface-sentencesimilarity-all-MiniLM-L6-v2      |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "408a3d2b",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a model](#2.-Select-a-model)\n",
    "3. [Deploy an endpoint & Query endpoint](#3.-Deploy-an-Endpoint-&-Query-endpointt)\n",
    "4. [Getting Nearest Neighbor On Your Own Dataset](#4.-Getting-Nearest-Neighbor-On-Your-Own-Dataset)\n",
    "5. [Getting the Accuracy of deployed model on the Amazon_SageMaker_FAQs dataset](#5.-Getting-the-Accuracy-of-deployed-model-on-the-Amazon_SageMaker_FAQs-dataset)\n",
    "6. [Run Batch Transform](#6.-Run-Batch-Transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763839e5",
   "metadata": {},
   "source": [
    "### 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a7c2a7",
   "metadata": {},
   "source": [
    "---\n",
    "Before executing the notebook, there are some initial steps required for set up\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a357fa6",
   "metadata": {
    "jumpStartAlterations": [
     "dropModelSelection"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade sagemaker --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c2132f-7a63-4ad8-958c-4028648540c6",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "To train and host on Amazon Sagemaker, we need to setup and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook instance as the AWS account role with SageMaker access. It has necessary permissions, including access to your data in S3. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c906d7db-9c5f-4f28-91bf-c0a346044a41",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker.session import Session\n",
    "\n",
    "sagemaker_session = Session()\n",
    "aws_role = sagemaker_session.get_caller_identity_arn()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89965762",
   "metadata": {},
   "source": [
    "## 2. Select a pre-trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb9d688c",
   "metadata": {
    "jumpStartAlterations": [
     "dropModelSelection"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id = \"huggingface-sentencesimilarity-gte-small\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cbb7a89-8dc6-422a-afa6-4026ebf0c73c",
   "metadata": {
    "jumpStartAlterations": [
     "dropModelSelection"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "import IPython\n",
    "from ipywidgets import Dropdown\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "from sagemaker.jumpstart.filters import And\n",
    "\n",
    "\n",
    "filter_value = And(\"task == sentencesimilarity\", \"framework == huggingface\")\n",
    "ss_models = list_jumpstart_models(filter=filter_value)\n",
    "\n",
    "dropdown = Dropdown(\n",
    "    value=model_id,\n",
    "    options=ss_models,\n",
    "    description=\"Sagemaker Pre-Trained Sentence Similarity Models:\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")\n",
    "display(IPython.display.Markdown(\"## Select a pre-trained model from the dropdown below\"))\n",
    "display(dropdown)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "867badfa",
   "metadata": {},
   "source": [
    "### 3. Deploy an Endpoint & Query Endpoint\n",
    "\n",
    "***\n",
    "\n",
    "Using SageMaker, we can perform inference on the pre-trained model.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cf4668c-3dd5-4cc6-93c8-029701fc8b8a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Deploying the model\n",
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "from sagemaker.serializers import JSONSerializer\n",
    "\n",
    "# The model is deployed on the ml.g5.2xlarge instance. To see all the supported parameters by the JumpStartModel\n",
    "# class use this link - https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.jumpstart.model.JumpStartModel\n",
    "my_model = JumpStartModel(model_id=dropdown.value)\n",
    "predictor = my_model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d22b024f-5433-44a3-8ea2-5ecd4a0296b2",
   "metadata": {},
   "source": [
    "### 3.1 Query Endpoint to Get Embeddings\n",
    "You can query the endpoint with a batch of input texts within a json payload. Here, we send a single request to the endpoint and the parsed response is a list of the embedding vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2549032-3d8d-4368-bd3a-42c20f139d2e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "text1 = \"How cute your dog is!\"\n",
    "text2 = \"Your dog is so cute.\"\n",
    "text3 = \"The mitochondria is the powerhouse of the cell.\"\n",
    "\n",
    "payload = [text1, text2, text3]\n",
    "\n",
    "predictor.predict(json.dumps(payload).encode(\"utf-8\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4b098a5",
   "metadata": {},
   "source": [
    "### 3.2 Query endpoint for Getting Nearest Neighbor\n",
    "The deployed model facilitates the process of identifying the nearest neighbors to input queries within the corpus. When provided with queries and a corpus, the model will produce a list. For each query, the output will provide both the corpus_id, which denotes the position of the relevant corpus entry in the input corpus list, and a score indicating the degree of proximity to the query. Please keep in mind that when making requests to the SageMaker invoke endpoint, payloads are restricted to approximately 5MB, and the request timeout is set to 1 minute. If your corpus size exceeds these limits, please utilize the approach outlined in the \"4. Getting Nearest Neighbor On Your Own Dataset\" section.\n",
    "\n",
    "* **corpus:** Provide the list of inputs from which to find the nearest neighbour\n",
    "* **queries:** Provide the list of inputs for which to find the nearest neighbour from the corpus\n",
    "* **top_k:** The number of nearest neighbour to find from the corpus\n",
    "* **mode:** Supply it as \"nn_corpus\" for getting the nearest neighbors to input queries within the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b113929b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "\n",
    "predictor.serializer = JSONSerializer()\n",
    "predictor.content_type = \"application/json\"\n",
    "\n",
    "corpus = [\n",
    "    \"Amazon SageMaker is a fully managed service to prepare data and build, train, and deploy machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows.\",\n",
    "    \"Amazon SageMaker stores code in ML storage volumes, secured by security groups and optionally encrypted at rest.\",\n",
    "    \"Amazon SageMaker provides a full end-to-end workflow, but you can continue to use your existing tools with SageMaker. You can easily transfer the results of each stage in and out of SageMaker as your business requirements dictate.\",\n",
    "]\n",
    "queries = [\n",
    "    \"What is Amazon SageMaker?\",\n",
    "    \"How does Amazon SageMaker secure my code?\",\n",
    "    \"What if I have my own notebook, training, or hosting environment?\",\n",
    "]\n",
    "\n",
    "payload_nearest_neighbour = {\"corpus\": corpus, \"queries\": queries, \"top_k\": 3, \"mode\": \"nn_corpus\"}\n",
    "\n",
    "query_response = predictor.predict(payload_nearest_neighbour)\n",
    "print(query_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04d99e2",
   "metadata": {},
   "source": [
    "### Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c3b60c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8dd37af-3b10-45ff-b8a6-ad28ab28c016",
   "metadata": {},
   "source": [
    "### 4. Getting Nearest Neighbor On Your Own Dataset\n",
    "\n",
    "***\n",
    "\n",
    "To find the nearest neighbor from your own dataset, you must provide it in the specified format during the training process. The training job will then generate \n",
    "embeddings for your dataset and save them along with the model. These embeddings will be utilized during inference to find the nearest neighbors \n",
    "for an input sentence. The process of finding the nearest neighbors once we have the embeddings is carried out using the [Sentence Transformer](https://www.sbert.net/) \n",
    "and its util function. The nearest neighbor is based on the cosine similarity between the input sentence embedding and already computed sentence embeddings \n",
    "during the training job.\n",
    "\n",
    "***\n",
    "\n",
    "### Required Data Format for the training job\n",
    "- **Input:** A directory containing a 'data.csv' file. \n",
    "    - Each row of the first column of 'data.csv' should have unique id\n",
    "    - Each row of the second column should have the corresponding text. \n",
    "- **Output:** A model prepackaged with input data embeddings that can be deployed for inference to get the nearest neighbor embedding id for an input sentence\n",
    " \n",
    "Below is an example of 'data.csv' file showing values in its first two columns. Note that the file should not have any header.\n",
    "\n",
    "|   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
    "|---|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
    "|1 | \"Amazon SageMaker is a fully managed service to prepare data and build, train, and deploy machine learning (ML) models for any use case with fully managed infrastructure, tools, and workflows.\"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |\n",
    "|2 | \"For a list of the supported Amazon SageMaker AWS Regions, please visit the AWS Regional Services page. Also, for more information, see Regional endpoints in the AWS general reference guide.\"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b321689-1bd1-4428-bab8-e31845667e2d",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.1. Getting Dataset \n",
    "***\n",
    "In this section, we'll be fetching and prepping the Amazon_SageMaker_FAQs dataset to utilize it in finding the nearest neighbour to an input question.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3777d713-8d36-4254-925d-a126935abe50",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Getting the Data for Training\n",
    "!aws s3 cp s3://jumpstart-cache-prod-us-west-2/training-datasets/Amazon_SageMaker_FAQs/Amazon_SageMaker_FAQs.csv Amazon_SageMaker_FAQs.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf87da76-2d17-4c3e-9250-fac5a3c0c48a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Preparing the Data in the required format\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(\"Amazon_SageMaker_FAQs.csv\", names=[\"Questions\", \"Answers\"])\n",
    "data[\"id\"] = data.index\n",
    "\n",
    "data_req = data[[\"id\", \"Answers\"]]\n",
    "\n",
    "data_req.to_csv(\"data.csv\", index=False, header=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78303f6b-9a0d-47c2-a5a4-7f0b573e16c9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Uploading the data in required format to s3 Bucket\n",
    "output_bucket = sess.default_bucket()\n",
    "output_prefix = \"jumpstart-example-ss-training\"\n",
    "\n",
    "s3_output_location = f\"s3://{output_bucket}/{output_prefix}/output\"\n",
    "training_dataset_s3_path = f\"s3://{output_bucket}/{output_prefix}/data/data.csv\"\n",
    "\n",
    "!aws s3 cp data.csv {training_dataset_s3_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "745a8f63-ceb1-423d-aa31-604dfa64eb3e",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.2. Set Training parameters\n",
    "***\n",
    "\n",
    "There are two kinds of parameters that need to be set for training. \n",
    "\n",
    "The first one are the parameters for the training job. These include: (i) Training data path. This is S3 folder in which the input data is stored, (ii) Output path: This the s3 folder in which the training output is stored. (iii) Training instance type: This indicates the type of machine on which to run the training. Typically, we use GPU instances for these training. \n",
    "***\n",
    "The second set of parameters are algorithm specific training hyper-parameters.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6688b70-bfe0-4b88-bf04-bacdb4cb26ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "# Retrieve the default hyper-parameters for fine-tuning the model\n",
    "hyperparameters = hyperparameters.retrieve_default(model_id=dropdown.value, model_version=\"*\")\n",
    "\n",
    "# [Optional] Override default hyperparameters with custom values\n",
    "# max_seq_length parameter is the max sequence length of the input to process by the embedding model. The default None value results in using the default max_seq_length for the model.\n",
    "hyperparameters[\"batch_size\"] = \"64\"\n",
    "print(hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "218bedd1-1dd2-4d76-bd7b-39b789682fd6",
   "metadata": {},
   "source": [
    "### 4.3. Getting the Embeddings for the Input Data\n",
    "***\n",
    "We start by creating the estimator object with all the required assets and then launch the training job.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "112495e6-6f42-46c7-8ccb-d0e47e20c6db",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.estimator import JumpStartEstimator\n",
    "\n",
    "estimator = JumpStartEstimator(\n",
    "    model_id=dropdown.value, hyperparameters=hyperparameters, output_path=s3_output_location\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd1487eb-4b19-4fd2-b80b-50bc26622917",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Launch a SageMaker Training job by passing s3 path of the data\n",
    "estimator.fit({\"training\": f\"s3://{output_bucket}/{output_prefix}/data\"}, logs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece0a299-492f-4602-8fcf-dd111ff98f0d",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 4.4. Deploy & run Inference on the model\n",
    "***\n",
    "The deployed model can be used for running inference. We support two types of the inference methods on the model. We follow the same steps as in 3. Deploy an Endpoint & Query Endpoint\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c27ab86-c9a7-4914-9ee9-1555cf58bb3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Use the estimator from the previous step to deploy to a SageMaker endpoint\n",
    "predictor = estimator.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8de6f424-6728-441f-8b3d-49d1ea711636",
   "metadata": {},
   "source": [
    "### 4.5 Query endpoint\n",
    "#### Query Endpoint to Get Embeddings\n",
    "You can query the endpoint with a batch of input texts within a json payload. Here, we send a single request to the endpoint and the parsed response is a list of the embedding vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caa4bf97-1bc4-44f5-bcf3-8393cea477a1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "payload = [\"Is R supported with Amazon SageMaker?\"]\n",
    "\n",
    "response = predictor.predict(json.dumps(payload).encode(\"utf-8\"))\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f4235ad-2699-48f3-a1de-02066e0e8655",
   "metadata": {},
   "source": [
    "### Query Endpoint to Get Nearest Neighbor\n",
    "You also have the option to make queries to the endpoint using a JSON payload containing a batch of input texts, to find the nearest neighbors of the input text from the dataset which is provided during the training job.\n",
    "\n",
    "* **queries:** Provide the list of inputs for which to find the closest match from the training data\n",
    "* **top_k:** The number of closest match to find from the training data\n",
    "* **mode:** Supply it as \"nn_train_data\" for getting the nearest neighbors to input queries within the dataset provided"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7469b4b-be0c-4844-bacf-c0d8ce96e095",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import JSONSerializer\n",
    "\n",
    "newline = \"\\n\"\n",
    "predictor.serializer = JSONSerializer()\n",
    "predictor.content_type = \"application/json\"\n",
    "\n",
    "payload_nearest_neighbour = {\n",
    "    \"queries\": [\"Is R supported with Amazon SageMaker?\"],\n",
    "    \"top_k\": 1,\n",
    "    \"mode\": \"nn_train_data\",\n",
    "}\n",
    "\n",
    "response = predictor.predict(payload_nearest_neighbour)\n",
    "\n",
    "print(\"The nearest neighbour for the input question is - \", response)\n",
    "\n",
    "question = payload_nearest_neighbour[\"queries\"][0]\n",
    "answer = data[\"Answers\"].iloc[int(response[0][0][\"id\"])]\n",
    "# Relating the Input Question with the Answer\n",
    "print(f\"The input Question is: {question}{newline}\" f\"The Corresponding Answer is: {answer}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4210cf46-0bd2-4f84-b747-325372226959",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5. Getting the Accuracy of deployed model on the Amazon_SageMaker_FAQs dataset\n",
    "\n",
    "***\n",
    "We will Query the endpoint for the questions in our Amazon_SageMaker_FAQs dataset and will compare if we get the correct corresponding answer using our sentence similarity model.\n",
    "***\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e253eb20-41a8-4f13-8a62-0612e4015054",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "total_correct_answers = 0\n",
    "for i in range(len(data)):\n",
    "    question = data[\"Questions\"].iloc[i]\n",
    "    payload_nearest_neighbour = {\"queries\": [question], \"top_k\": 1, \"mode\": \"nn_train_data\"}\n",
    "    response = predictor.predict(payload_nearest_neighbour)\n",
    "\n",
    "    response_id = response[0][0][\"id\"]\n",
    "\n",
    "    if int(response_id) == i:\n",
    "        total_correct_answers += 1\n",
    "\n",
    "print(\n",
    "    f\"The accuracy of the model on the Amazon_SageMaker_FAQs dataset is: {total_correct_answers*100/len(data)}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9080d5e-f4b5-4302-b7eb-bb7c85c2d9fe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint and the attached resources\n",
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db02a101",
   "metadata": {},
   "source": [
    "### 6. Run Batch Transform to Get Embeddings On Large Datasets\n",
    "***\n",
    "Using SageMaker, we can perform batch inference on the model for large datasets. For this example, that means on an input sentence providing the embedding. When you start a batch transform job, Amazon SageMaker launches the necessary compute resources to process the data, including CPU or GPU instances depending on the selected instance type. During the batch transform job, Amazon SageMaker automatically provisions and manages the compute resources required to process the data, including instances, storage, and networking resources. Once the batch transform job has completed, the compute resources are automatically cleaned up by Amazon SageMaker. This means that the instances and storage used during the job are terminated and removed, freeing up resources and minimizing costs\n",
    "\n",
    "- Batch Transform is useful in the following scenarios:\n",
    "    - Preprocess datasets to remove noise or bias that interferes with training or inference from your dataset.\n",
    "    - Get inferences from large datasets.\n",
    "    - Run inference when you don't need a persistent endpoint.\n",
    "    - Associate input records with inferences to assist the interpretation of results.\n",
    "\n",
    "The input format for the batch transform job is a jsonl file with entries as -> \n",
    "- {\"id\":1,\"text_inputs\":\"How cute your dog is!\"}\n",
    "- {\"id\":2,\"text_inputs\":\"The mitochondria is the powerhouse of the cell.\"}\n",
    "\n",
    "While the output format is -> \n",
    "- {\"id\":1, \"embedding\":[0.025507507845759392, 0.009654928930103779, -0.01139055471867323, .........]}\n",
    "- {\"id\":2, \"embedding\":[-0.018594933673739433, -0.011756304651498795, -0.006888044998049736,.....]}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13660588",
   "metadata": {},
   "source": [
    "### 6.1. Prepare data for Batch Transform\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cb7a16d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_input_data_path = f\"s3://{output_bucket}/{output_prefix}/batch_input/\"\n",
    "s3_output_data_path = f\"s3://{output_bucket}/{output_prefix}/batch_output/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26192804",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import boto3\n",
    "import os\n",
    "\n",
    "data = pd.read_csv(\"Amazon_SageMaker_FAQs.csv\", names=[\"Questions\", \"Answers\"])\n",
    "\n",
    "# Provide the test data and the ground truth file name\n",
    "test_data_file_name = \"test.jsonl\"\n",
    "\n",
    "test_data = []\n",
    "\n",
    "# We will go over each data entry and create the data in the input required format as described above\n",
    "for i in range(len(data)):\n",
    "    answer = data.loc[i, \"Answers\"]\n",
    "    payload = {\"id\": i, \"text_inputs\": answer}\n",
    "    test_data.append(payload)\n",
    "\n",
    "with open(test_data_file_name, \"w\") as outfile:\n",
    "    for entry in test_data:\n",
    "        outfile.write(f\"{json.dumps(entry)}\\n\")\n",
    "\n",
    "# Uploading the data\n",
    "s3 = boto3.client(\"s3\")\n",
    "s3.upload_file(test_data_file_name, output_bucket, f\"{output_prefix}/batch_input/test.jsonl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d6e6f2",
   "metadata": {},
   "source": [
    "### 6.2. Run Batch Transform\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3aa95589",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Creating the batch transformer object. If you have a large dataset you can\n",
    "# divide it into smaller chunks and use more instances for faster inference\n",
    "my_model = JumpStartModel(model_id=dropdown.value, model_version=\"1.*\")\n",
    "\n",
    "batch_transformer = my_model.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.p3.2xlarge\",\n",
    "    output_path=s3_output_data_path,\n",
    "    assemble_with=\"Line\",\n",
    "    accept=\"text/csv\",\n",
    "    max_payload=1,\n",
    ")\n",
    "\n",
    "# Making the predictions on the input data\n",
    "batch_transformer.transform(\n",
    "    s3_input_data_path, content_type=\"application/jsonlines\", split_type=\"Line\"\n",
    ")\n",
    "\n",
    "batch_transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b455e77",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Downloading the Generated Embeddings\n",
    "\n",
    "# Downloading the predictions\n",
    "s3.download_file(\n",
    "    output_bucket, output_prefix + \"/batch_output/\" + \"test.jsonl.out\", \"predict.jsonl\"\n",
    ")\n",
    "\n",
    "with open(\"predict.jsonl\", \"r\") as json_file:\n",
    "    json_list = list(json_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8533da3c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Creating the predictions list which can be used to extract the embeddings given the id\n",
    "import ast\n",
    "\n",
    "predict_dict_list = []\n",
    "for predict in json_list:\n",
    "    if len(predict) > 1:\n",
    "        predict_dict = ast.literal_eval(predict)\n",
    "        predict_dict_req = {\n",
    "            \"id\": predict_dict[\"id\"],\n",
    "            \"embedding\": predict_dict[\"embedding\"],\n",
    "        }\n",
    "        predict_dict_list.append(predict_dict_req)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcf8206c",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|question_answering_retrieval_augmented_generation|text-embedding-sentence-similarity.ipynb)"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}