 archived/text2text-generation-bloomz.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4a1a52b6",
   "metadata": {},
   "source": [
    "# SageMaker JumpStart Foundation Models - BloomZ: Multilingual Text Classification, Question and Answering, Code Generation, Paragraph rephrase, and More"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "71a7d870",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5acea92d",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to Amazon [SageMaker JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use Sagemaker JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html#use-prebuilt-models-with-sagemaker-jumpstart).\n",
    "\n",
    "\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK for deploying Foundation Models as an endpoint and use them for various NLP tasks. The Foundation models perform **Text2Text Generation**. It takes a prompting text as an input, and returns the text generated by the model according to the prompt.\n",
    "\n",
    "Here, we show how to use the state-of-the-art pre-trained **BloomZ 7b1** from [Hugging Face](https://huggingface.co/bigscience/bloomz-7b1) for Text2Text Generation in the following tasks. You can directly use **BloomZ 7b1** for many NLP tasks, without finetuning the model.\n",
    "\n",
    "\n",
    "* **Multilingual text / sentiment classification**\n",
    "* **Multilingual question and answering**\n",
    "* **Code generation**\n",
    "* **Paragraph rephrase**\n",
    "* **Summarization**\n",
    "* **Common sense reasoning / natural language inference**\n",
    "* **Question and answering**\n",
    "* **Sentence / sentiment classification**\n",
    "* **Translation**\n",
    "* **Pronoun resolution**\n",
    "* **Imaginary article generation based on title**\n",
    "* **Summarize a title based on a article**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815c0bc7",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a model](#2.-Select-a-model)\n",
    "3. [Retrieve Artifacts & Deploy an Endpoint](#3.-Retrieve-Artifacts-&-Deploy-an-Endpoint)\n",
    "4. [Query endpoint and parse response](#4.-Query-endpoint-and-parse-response)\n",
    "5. [Advanced features: How to use varisous parameters to control the generated text](#5.-Advanced-features:-How-to-use-various-advanced-parameters-to-control-the-generated-text)\n",
    "6. [Advanced features: How to use prompts engineering to solve different tasks](#6.-Advacned-features:-How-to-use-prompts-engineering-to-solve-different-tasks)\n",
    "5. [Clean up the endpoint](#5.-Clean-up-the-endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7e35194",
   "metadata": {},
   "source": [
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with **conda_pytorch_p39** kernel and in Amazon SageMaker Notebook instance with **conda_python3** kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f8dfad",
   "metadata": {},
   "source": [
    "### 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f31be0",
   "metadata": {},
   "source": [
    "---\n",
    "Before executing the notebook, there are some initial steps required for set up. This notebook requires ipywidgets.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb67d497",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install ipywidgets --quiet\n",
    "!pip install --upgrade sagemaker --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769f5d81",
   "metadata": {},
   "source": [
    "#### Permissions and environment variables\n",
    "\n",
    "---\n",
    "To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67131eee",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker.session import Session\n",
    "\n",
    "sagemaker_session = Session()\n",
    "aws_role = sagemaker_session.get_caller_identity_arn()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69849d02",
   "metadata": {},
   "source": [
    "## 2. Select a pre-trained model\n",
    "***\n",
    "You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of SageMaker pre-trained models can also be accessed at [Sagemaker pre-trained Models](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#).\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "652b2d4f",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, model_version, = (\n",
    "    \"huggingface-textgeneration1-bloomz-7b1-fp16\",\n",
    "    \"1.*\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "170e1228",
   "metadata": {},
   "source": [
    "***\n",
    "[Optional] Select a different Sagemaker pre-trained model. Here, we download the model_manifest file from the Built-In Algorithms s3 bucket, filter-out all the Text Generation models and select a model for inference.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d8a1f7e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "\n",
    "# Retrieves all Text Generation models available by SageMaker Built-In Algorithms.\n",
    "filter_value = \"task == textgeneration1\"\n",
    "text_generation_models = list_jumpstart_models(filter=filter_value)\n",
    "\n",
    "# display the model-ids in a dropdown to select a model for inference.\n",
    "model_dropdown = widgets.Dropdown(\n",
    "    options=text_generation_models,\n",
    "    value=model_id,\n",
    "    description=\"Select a model\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28d45e5",
   "metadata": {},
   "source": [
    "#### Choose a model for Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52b7a67a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(model_dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "271642dc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# model_version=\"*\" fetches the latest version of the model\n",
    "model_id, model_version = model_dropdown.value, \"1.*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b08aa4a",
   "metadata": {},
   "source": [
    "### 3. Retrieve Artifacts & Deploy an Endpoint\n",
    "\n",
    "***\n",
    "\n",
    "Using SageMaker, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. We start by retrieving the `deploy_image_uri`, `deploy_source_uri`, and `model_uri` for the pre-trained model. To host the pre-trained model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it. This may take a few minutes.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c122ee7e-4180-4f4b-8cc9-cf60d3e2b6f2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_sagemaker_session(local_download_dir) -> sagemaker.Session:\n",
    "    \"\"\"Return the SageMaker session.\"\"\"\n",
    "\n",
    "    sagemaker_client = boto3.client(\n",
    "        service_name=\"sagemaker\", region_name=boto3.Session().region_name\n",
    "    )\n",
    "\n",
    "    session_settings = sagemaker.session_settings.SessionSettings(\n",
    "        local_download_dir=local_download_dir\n",
    "    )\n",
    "\n",
    "    # the unit test will ensure you do not commit this change\n",
    "    session = sagemaker.session.Session(\n",
    "        sagemaker_client=sagemaker_client, settings=session_settings\n",
    "    )\n",
    "\n",
    "    return session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44051043-7405-457e-809f-a7c004646943",
   "metadata": {},
   "source": [
    "We need to create a directory to host the downloaded model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fcfc73e-00b1-4672-b327-99a477abfbc8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!mkdir -p download_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "631ae768",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris, hyperparameters\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-{model_id}\")\n",
    "\n",
    "inference_instance_type = \"ml.g4dn.12xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri. This is the base HuggingFace container image for the default model above.\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,  # automatically inferred from model_id\n",
    "    image_scope=\"inference\",\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    framework=\"djl-deepspeed\", region=sagemaker_session.boto_session.region_name, version=\"0.20.0\"\n",
    ")\n",
    "\n",
    "# Retrieve the model uri.\n",
    "model_uri = model_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, model_scope=\"inference\"\n",
    ")\n",
    "\n",
    "model = Model(\n",
    "    image_uri=deploy_image_uri,\n",
    "    model_data=model_uri,  # \"s3://testgptj/test/infer-huggingface-textgeneration1-bloomz-7b.tar.gz\",#model_uri,\n",
    "    role=aws_role,\n",
    "    predictor_cls=Predictor,\n",
    "    name=endpoint_name,\n",
    ")\n",
    "\n",
    "# deploy the Model. Note that we need to pass Predictor class when we deploy model through Model class,\n",
    "# for being able to run inference through the sagemaker API.\n",
    "model_predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    predictor_cls=Predictor,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9c254b",
   "metadata": {},
   "source": [
    "### 4. Query endpoint and parse response\n",
    "\n",
    "---\n",
    "Input to the endpoint is any string of text formatted as json and encoded in `utf-8` format. Output of the endpoint is a `json` with generated text.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "439998c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n",
    "\n",
    "\n",
    "def query_endpoint(encoded_text, endpoint_name):\n",
    "    client = boto3.client(\"runtime.sagemaker\")\n",
    "    response = client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name, ContentType=\"application/x-text\", Body=encoded_text\n",
    "    )\n",
    "    return response\n",
    "\n",
    "\n",
    "def parse_response(query_response):\n",
    "    model_predictions = json.loads(query_response[\"Body\"].read())\n",
    "    generated_text = model_predictions[0][\"generated_text\"]\n",
    "    return generated_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc5d644d",
   "metadata": {},
   "source": [
    "---\n",
    "Below, we  put in some example input text. You can put in any text and the model predicts next words in the sequence. Longer sequences of text can be generated by calling the model repeatedly.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8262dfcc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n",
    "\n",
    "text1 = \"Translate to German:  My name is Arthur\"\n",
    "text2 = \"A step by step recipe to make bolognese pasta:\"\n",
    "\n",
    "\n",
    "for text in [text1, text2]:\n",
    "    query_response = query_endpoint(json.dumps(text).encode(\"utf-8\"), endpoint_name=endpoint_name)\n",
    "    generated_text = parse_response(query_response)\n",
    "    print(\n",
    "        f\"Inference:{newline}\"\n",
    "        f\"input text: {text}{newline}\"\n",
    "        f\"generated text: {bold}{generated_text}{unbold}{newline}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2554851-cbcc-4ef9-864e-776a3550ceca",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5. Advanced features: How to use various advanced parameters to control the generated text\n",
    "\n",
    "***\n",
    "This model also supports many advanced parameters while performing inference. They include:\n",
    "\n",
    "* **max_length:** Model generates text until the output length (which includes the input context length) reaches `max_length`. If specified, it must be a positive integer.\n",
    "* **num_return_sequences:** Number of output sequences returned. If specified, it must be a positive integer.\n",
    "* **num_beams:** Number of beams used in the greedy search. If specified, it must be integer greater than or equal to `num_return_sequences`.\n",
    "* **no_repeat_ngram_size:** Model ensures that a sequence of words of `no_repeat_ngram_size` is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **early_stopping:** If True, text generation is finished when all beam hypotheses reach the end of stence token. If specified, it must be boolean.\n",
    "* **do_sample:** If True, sample the next word as per the likelyhood. If specified, it must be boolean.\n",
    "* **top_k:** In each step of text generation, sample from only the `top_k` most likely words. If specified, it must be a positive integer.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "* **seed:** Fix the randomized state for reproducibility. If specified, it must be an integer.\n",
    "\n",
    "We may specify any subset of the parameters mentioned above while invoking an endpoint. Next, we show an example of how to invoke endpoint with these arguments\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1af6f7b0-4093-48c9-acdb-54b05886b2dc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Input must be a json\n",
    "payload = {\n",
    "    \"text_inputs\": [\"Tell me the steps to make a pizza\", \"i work at new york\"],\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 3,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "def query_endpoint_with_json_payload(encoded_json, endpoint_name):\n",
    "    client = boto3.client(\"runtime.sagemaker\")\n",
    "    response = client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name, ContentType=\"application/json\", Body=encoded_json\n",
    "    )\n",
    "    return response\n",
    "\n",
    "\n",
    "query_response = query_endpoint_with_json_payload(\n",
    "    json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    ")\n",
    "\n",
    "\n",
    "def parse_response_multiple_texts(query_response):\n",
    "    model_predictions = json.loads(query_response[\"Body\"].read())\n",
    "    # generated_text = model_predictions[0][\"generated_texts\"]\n",
    "    generated_text = [x[\"generated_text\"] for x in model_predictions[0]]\n",
    "\n",
    "    return generated_text\n",
    "\n",
    "\n",
    "generated_texts = parse_response_multiple_texts(query_response)\n",
    "print(generated_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54977c4-d91a-4489-b137-26fa1d5f1f2d",
   "metadata": {},
   "source": [
    "### 6. Advanced features: How to use prompts engineering to solve different tasks\n",
    "\n",
    "Below we domenstrate solving 12 key tasks with **BloomZ 7B1** model. The tasks are following.\n",
    "\n",
    "\n",
    "* **Multilingual text / sentiment classification**\n",
    "* **Multilingual question and answering**\n",
    "* **Code generation**\n",
    "* **Paragraph rephrase**\n",
    "* **Summarization**\n",
    "* **Common sense reasoning / natural language inference**\n",
    "* **Question and answering**\n",
    "* **Sentence / sentiment classification**\n",
    "* **Translation**\n",
    "* **Pronoun resolution**\n",
    "* **Imaginary article generation based on title**\n",
    "* **Summarize a title based on a article**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12706eb8-67d6-4e9d-ae9d-5faaa9709106",
   "metadata": {},
   "source": [
    "### 6.1. Multilingual text / sentiment classification (Chinese to English)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23266852-fb89-4db2-a9c3-3afd3cb68151",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"一个传奇的开端，一个不灭的神话，这不仅仅是一部电影，而是作为一个走进新时代的标签，永远彪炳史册。\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "287d1734-15e4-4566-b071-3536cabeef8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\"\"\"{text} Would you rate the previous review as positive, neutral or negative?\"\"\"]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 500,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{text}\", text)\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"For input with prompt: {payload['text_inputs']}\")\n",
    "    print(f\"{bold}Result{unbold}: {generated_texts}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5547820c-efe6-4346-b197-2bc64e353b0f",
   "metadata": {},
   "source": [
    "### 6.2. Multilingual question and answering (English to Chinese)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2446dfe-85cd-4be3-8534-856b313bce8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"what is the backpropagation.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5453765-b869-4e7e-b0ce-504077c15d58",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\"\"\"Explain to me in Traditional Chinese  {text}\"\"\"]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 500,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{text}\", text)\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"For input with prompt: {payload['text_inputs']}\")\n",
    "    print(f\"{bold}Result{unbold}: {generated_texts}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8ec3e43-d056-4f6b-a4a8-4edc1870e8d8",
   "metadata": {},
   "source": [
    "### 6.3. Code generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c2c7116-c872-4ef3-89af-1ebae1fb743a",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"binary search tree\"\n",
    "code_start = \"\"\"\n",
    "def binary_search(a, x):\n",
    "    low = 0\n",
    "    high = len(a) - 1\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f038829-d850-4537-8ed7-dd1f22a65e0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Write a {text} with O(log(n)) computational complexity. \n",
    "{code_start}\"\"\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d2258d6-260f-407a-b56b-c9370d4839de",
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    \"max_length\": 500,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{text}\", text)\n",
    "    input_text = input_text.replace(\"{code_start}\", code_start)\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"For input with prompt: {payload['text_inputs']}\")\n",
    "    print(f\"{bold} For prompt: '{each_prompt}'{unbold}{newline}\")\n",
    "    print(f\"{bold}Result{unbold}: {generated_texts}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c74bd0-04f3-431f-81c1-11561cb7182b",
   "metadata": {},
   "source": [
    "### 6.4. Paragraph rephrase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "878faee0-c1e3-480d-b1ee-b6c1796f984b",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"\"\"Amazon Web Services (AWS) has announced nine major new updates for its cloud-based machine learning platform, SageMaker.\n",
    "SageMaker aims to provide a machine learning service which can be used to build, train, and deploy ML models for virtually any use case.\n",
    "During this year’s re:Invent conference, AWS made several announcements to further improve SageMaker’s capabilities.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c17df95-f941-489c-9b94-2ee7c64efde8",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"\\ {sentence}\\n\\nHow would you rephrase that briefly using English?\"\"\",\n",
    "    \"\"\"\"{sentence}\\nThe above sentence is very complicated. Please provide me a simplified synonymous version consisting of multiple sentences:\"\"\",\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 50000,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.01,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ec5b958-1f20-4899-a02f-e3854e804c27",
   "metadata": {},
   "source": [
    "### 6.5. Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68a1b142-bf45-4e4e-9535-57184ad986df",
   "metadata": {},
   "source": [
    "Define the text article you want to summarize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84c4893d-5673-428d-90b8-09583987151a",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents. For example, using Amazon Comprehend you can search social networking feeds for mentions of products or scan an entire document repository for key phrases. \n",
    "You can access Amazon Comprehend document analysis capabilities using the Amazon Comprehend console or using the Amazon Comprehend APIs. You can run real-time analysis for small workloads or you can start asynchronous analysis jobs for large document sets. You can use the pre-trained models that Amazon Comprehend provides, or you can train your own custom models for classification and entity recognition. \n",
    "All of the Amazon Comprehend features accept UTF-8 text documents as the input. In addition, custom classification and custom entity recognition accept image files, PDF files, and Word files as input. \n",
    "Amazon Comprehend can examine and analyze documents in a variety of languages, depending on the specific feature. For more information, see Languages supported in Amazon Comprehend. Amazon Comprehend's Dominant language capability can examine documents and determine the dominant language for a far wider selection of languages.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e746d6f-317b-4cb4-a334-11b9b3eb55d5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\"\"\"{text}\\n\\n===\\nWrite a summary of the previous text in English:\"\"\"]\n",
    "\n",
    "num_return_sequences = 3\n",
    "parameters = {\n",
    "    \"max_length\": 500,\n",
    "    \"num_return_sequences\": num_return_sequences,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "print(f\"{bold}Number of return sequences are set as {num_return_sequences}{unbold}{newline}\")\n",
    "for each_prompt in prompts:\n",
    "    payload = {\"text_inputs\": each_prompt.replace(\"{text}\", text), **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} For prompt: '{each_prompt}'{unbold}{newline}\")\n",
    "    print(f\"{bold} The {num_return_sequences} summarized results are{unbold}:{newline}\")\n",
    "    for idx, each_generated_text in enumerate(generated_texts):\n",
    "        print(f\"{bold}Result {idx}{unbold}: {each_generated_text}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1925b70a-548f-4a11-8912-c69004380583",
   "metadata": {},
   "source": [
    "### 6.6. Common sense reasoning / natural language inference\n",
    "\n",
    "In the common sense reasoning, you can design a prompt and combine it with the premise, hypothesis, and options, send the combined text into the endpoint to get an answer. Examples are demonstrated as below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55757d1d-a9e9-4307-8c1c-ff919474979e",
   "metadata": {},
   "source": [
    "Define the premise, hypothesis, and options that you hope the model to reason."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51d4f714-2484-4901-ada3-7c9cc6360881",
   "metadata": {},
   "outputs": [],
   "source": [
    "premise = \"The world cup has kicked off in Los Angeles, United States.\"\n",
    "hypothesis = \"The world cup takes place in United States.\"\n",
    "options = \"\"\"[\"yes\", \"no\"]\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1322b063-7c56-45c3-9ab0-5cd69e3aa50a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Given that {premise} Does it follow that {hypothesis} Yes or no?\"\"\",\n",
    "    \"\"\"Suppose {premise} Can we infer that \\\"{hypothesis}\\\"? Yes or no?\"\"\",\n",
    "    \"\"\"{premise} Based on the previous passage, is it true that \\\"{hypothesis}\\\"? Yes or no?\"\"\",\n",
    "    \"\"\"{premise} Using only the above description and what you know about the world, is \\\"{hypothesis}\\\" definitely correct? Yes or no?\"\"\",\n",
    "    \"\"\"{premise}\\nQuestion: {hypothesis} True or False?\"\"\",\n",
    "    \"\"\"\"{premise} Using only the above description and what you know about the world, is \\\"{hypothesis}\\\" definitely correct? Yes or no?\"\"\",\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{premise}\", premise)\n",
    "    input_text = input_text.replace(\"{hypothesis}\", hypothesis)\n",
    "    # input_text = input_text.replace(\"{options_}\", options)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00febfb-9f98-4f45-8204-a79f134ed17e",
   "metadata": {},
   "source": [
    "### 6.7. Question and Answering\n",
    "\n",
    "Now, let's try another reasoning task with a different type of prompt template. You can simply provide context and question as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0a08432-60ef-430f-b564-1c59c84bd2ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"\"\"The newest and most innovative Kindle yet lets you take notes on millions of books and documents, write lists and journals, and more. \n",
    "\n",
    "For readers who have always wished they could write in their eBooks, Amazon’s new Kindle lets them do just that. The Kindle Scribe is the first Kindle for reading and writing and allows users to supplement their books and documents with notes, lists, and more.\n",
    "\n",
    "Here’s everything you need to know about the Kindle Scribe, including frequently asked questions.\n",
    "\n",
    "The Kindle Scribe makes it easy to read and write like you would on paper \n",
    "\n",
    "The Kindle Scribe features a 10.2-inch, glare-free screen (the largest of all Kindle devices), crisp 300 ppi resolution, and 35 LED front lights that automatically adjust to your environment. Further personalize your experience with the adjustable warm light, font sizes, line spacing, and more.\n",
    "\n",
    "It comes with your choice of the Basic Pen or the Premium Pen, which you use to write on the screen like you would on paper. They also attach magnetically to your Kindle and never need to be charged. The Premium Pen includes a dedicated eraser and a customizable shortcut button.\n",
    "\n",
    "The Kindle Scribe has the most storage options of all Kindle devices: choose from 8 GB, 16 GB, or 32 GB to suit your level of reading and writing.\n",
    "\"\"\"\n",
    "question = \"what are the key features of new Kindle?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a70c89f-42d5-4e2c-b3c1-4cfaf4e3f655",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\"\"\"question: \\\"{question}\"\\\\n\\nContext: \\\"{context}\"\\\\n\\nAnswer:\"\"\"]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 5000,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{context}\", context)\n",
    "    input_text = input_text.replace(\"{question}\", question)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13ab5c92-5210-4ffe-842a-78e8ea0a6521",
   "metadata": {},
   "source": [
    "### 6.8. Sentence / Sentiment Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f140ae5-54c0-4738-b2c2-d715f8ae8050",
   "metadata": {},
   "source": [
    "Define the sentence you want to classifiy and the corresponded options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd29e5fb-d7f9-43bd-ad03-4887c1381169",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence1 = \"This moive is so great and once again dazzles and delights us\"\n",
    "options_ = \"\"\"OPTIONS:\\n-positive \\n-negative \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edb65e09-8c2f-4a8c-87bd-64ab49a067b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Review:\\n{sentence}\\nIs this movie review sentence negative or positive?\\n{options_}\"\"\",\n",
    "    \"\"\"Short movie review: {sentence}\\nDid the critic think positively or negatively of the movie?\\n{options_}\"\"\",\n",
    "    \"\"\"Sentence from a movie review: {sentence}\\nWas the movie seen positively or negatively based on the preceding review? \\n\\n{options_}\"\"\",\n",
    "    \"\"\"\\\"{sentence}\\\"\\nHow would the sentiment of this sentence be perceived?\\n\\n{options_}\"\"\",\n",
    "    \"\"\"Is the sentiment of the following sentence positive or negative?\\n{sentence}\\n{options_}\"\"\",\n",
    "    \"\"\"What is the sentiment of the following movie review sentence?\\n{sentence}\\n{options_}\"\"\",\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence1)\n",
    "    input_text = input_text.replace(\"{options_}\", options_)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc215766-a55a-44a1-8479-0012947d9c29",
   "metadata": {},
   "source": [
    "Now let's try if the model can identify the sentence is grammatically correct or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a6ad0c2-3f27-49e4-8db4-89dab1fe8307",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"\"\"study I mathematics.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19de59e2-bd03-4fe1-b187-d0959e260448",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"The following sentence is either \\\"{{\\\"acceptable\\\"}}\\\", meaning it is grammatically correct and makes sense, or \\\"{{\\\"unacceptable\\\"}}\\\". Which is it?\\n{sentence}\\n|\"\"\",\n",
    "    \"\"\"{sentence}\\nI'm worried that sentence didn't make any sense, or was grammatically incorrect. Was it correct?\\n\"\"\",\n",
    "    \"\"\"\"{sentence}\\nIs this example grammatically correct and sensible?\\n\"\"\",\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3046a9ef-d64e-42ec-a1ca-107a15f7164f",
   "metadata": {},
   "source": [
    "### 6.9. Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f151f47-a656-42aa-9b6b-8cc6e4ced876",
   "metadata": {},
   "source": [
    "Define the sentence and the language you want to translate the sentence to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f8885f-76bd-4b42-92ac-cdeb905c97a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sent1 = \"My name is Arthur\"\n",
    "lang2 = \"\"\"German\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7259e6a-45c1-49af-a7c8-f176f3552ee8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"{sent1}\\n\\nTranslate to {lang2}\"\"\",\n",
    "    \"\"\"{sent1}\\n\\nCould you please translate this to {lang2}?\"\"\",\n",
    "    \"\"\"Translate to {lang2}:\\n\\n{sent1}\"\"\",\n",
    "    \"\"\"Translate the following sentence to {lang2}:\\n{sent1}\"\"\",\n",
    "    \"\"\"How is \\\"{sent1}\\\" said in {lang2}?\"\"\",\n",
    "    \"\"\"Translate \\\"{sent1}\\\" to {lang2}?\"\"\",\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sent1}\", sent1)\n",
    "    input_text = input_text.replace(\"{lang2}\", lang2)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The translated result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdeede3e-dc5d-4e57-af30-ef381b694a22",
   "metadata": {},
   "source": [
    "### 6.10. Imaginary article generation based on title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908fe937-3b2c-4e20-b6d0-e3db02dbcee8",
   "metadata": {},
   "outputs": [],
   "source": [
    "title = \"University has new facility coming up\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6408ef62-03d0-4505-b5ab-1459d5558dc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Title: \\\"{title}\\\"\\\\nGiven the above title of an imaginary article, imagine the article.\\\\n\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 50000,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{title}\", title)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfa5259a-c45d-4309-b536-f7ecd74bf8da",
   "metadata": {},
   "source": [
    "### 6.11. Summarize a title based on the article"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2604e4b-be03-4d0c-9106-1e43e26e7511",
   "metadata": {},
   "outputs": [],
   "source": [
    "article = \"\"\"The newest and most innovative Kindle yet lets you take notes on millions of books and documents, write lists and journals, and more. \n",
    "\n",
    "For readers who have always wished they could write in their eBooks, Amazon’s new Kindle lets them do just that. The Kindle Scribe is the first Kindle for reading and writing and allows users to supplement their books and documents with notes, lists, and more.\n",
    "\n",
    "Here’s everything you need to know about the Kindle Scribe, including frequently asked questions.\n",
    "\n",
    "The Kindle Scribe makes it easy to read and write like you would on paper \n",
    "\n",
    "The Kindle Scribe features a 10.2-inch, glare-free screen (the largest of all Kindle devices), crisp 300 ppi resolution, and 35 LED front lights that automatically adjust to your environment. Further personalize your experience with the adjustable warm light, font sizes, line spacing, and more.\n",
    "\n",
    "It comes with your choice of the Basic Pen or the Premium Pen, which you use to write on the screen like you would on paper. They also attach magnetically to your Kindle and never need to be charged. The Premium Pen includes a dedicated eraser and a customizable shortcut button.\n",
    "\n",
    "The Kindle Scribe has the most storage options of all Kindle devices: choose from 8 GB, 16 GB, or 32 GB to suit your level of reading and writing.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f27c7d62-cce0-4279-a082-7bd6bd7a86e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts = [\"\"\"'\\'{article} \\n\\n \\\\n\\\\nGive me a good title for the article above.\"\"\"]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 5000,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{article}\", article)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa5de21f",
   "metadata": {},
   "source": [
    "### 7. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69b588d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "model_predictor.delete_model()\n",
    "model_predictor.delete_endpoint()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3c8760a6",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text2text-generation-bloomz.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_pytorch_p39",
   "language": "python",
   "name": "conda_pytorch_p39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}