 archived/text-generation-few-shot-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4a1a52b6",
   "metadata": {},
   "source": [
    "# SageMaker JumpStart Foundation Models - GPT-J, GPT-Neo Few-shot learning"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "14d6b75f",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5acea92d",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to Amazon [SageMaker JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use Sagemaker JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html#use-prebuilt-models-with-sagemaker-jumpstart).\n",
    "\n",
    "\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK for deploying Foundation Models as an endpoint and use them for various NLP tasks. The Foundation models perform **Text Generation**. It takes a text string as input and predicts a sequence of next few words. These models can, for example, fill in incomplete text or paraphrase.\n",
    "\n",
    "Here, we show how to use the state-of-the-art pre-trained **GPT-J-6B models** from [Hugging Face](https://huggingface.co/EleutherAI/gpt-j-6B) for **Text Generation** in the following tasks. You can use **GPT-J-6B** model with **few shot learning** for many NLP tasks, without fine-tuning the model. There are other text generation models like [**Bloom 7b1**](https://huggingface.co/bigscience/bloom-7b1), [**Bloom 3b**](https://huggingface.co/bigscience/bloom-3b), [**GPT-NEO-2b7**](https://huggingface.co/EleutherAI/gpt-neo-2.7B), and etc.\n",
    "\n",
    "The **few shot leanring** allows a machine learning model to make predictions with only a couple of examples. There is no need to train a new model as models like **GPT-J**, **Bloom** have a significant amount of parameters such that they can easily adapt to many contexts without being re-trained. Below are tasks that text generation can solve using few shot learning method.\n",
    "\n",
    "\n",
    "* **Text summarization**\n",
    "* **Code generation**\n",
    "* **Name entity recognition (NER)**\n",
    "* **Question answering**\n",
    "* **Grammar and spelling correction**\n",
    "* **Product description and generalization**\n",
    "* **Sentence / sentiment classification**\n",
    "* **Chatbot and conversational AI**\n",
    "* **Tweet generation**\n",
    "* **Machine translation**\n",
    "* **Pharapharsing**\n",
    "* **Intent classification**\n",
    "\n",
    "**For domain adaption finetuning of text generation models such as GPT-J 6B, please check the notebook `domain-adaption-finetuning-gpt-j-6b.ipynb` in the same directory.**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815c0bc7",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a model](#2.-Select-a-model)\n",
    "3. [Retrieve Artifacts & Deploy an Endpoint](#3.-Retrieve-Artifacts-&-Deploy-an-Endpoint)\n",
    "4. [Query endpoint and parse response](#4.-Query-endpoint-and-parse-response)\n",
    "5. [Advanced features: How to use varisous parameters to control the generated text](#5.-Advanced-features:-How-to-use-various-advanced-parameters-to-control-the-generated-text)\n",
    "6. [Advanced features: How to use prompts engineering to solve different tasks](#6.-Advacned-features:-How-to-use-prompts-engineering-to-solve-different-tasks)\n",
    "5. [Clean up the endpoint](#5.-Clean-up-the-endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7e35194",
   "metadata": {},
   "source": [
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f8dfad",
   "metadata": {},
   "source": [
    "### 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f31be0",
   "metadata": {},
   "source": [
    "---\n",
    "Before executing the notebook, there are some initial steps required for set up. This notebook requires ipywidgets.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb67d497",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install ipywidgets==7.0.0 --quiet\n",
    "!pip install --upgrade sagemaker --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "769f5d81",
   "metadata": {},
   "source": [
    "#### Permissions and environment variables\n",
    "\n",
    "---\n",
    "To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67131eee",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker.session import Session\n",
    "\n",
    "sagemaker_session = Session()\n",
    "aws_role = sagemaker_session.get_caller_identity_arn()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69849d02",
   "metadata": {},
   "source": [
    "## 2. Select a pre-trained model\n",
    "***\n",
    "You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of SageMaker pre-trained models can also be accessed at [Sagemaker pre-trained Models](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#).\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "652b2d4f",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id, model_version, = (\n",
    "    \"huggingface-textgeneration1-gpt-j-6b\",\n",
    "    \"1.*\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "170e1228",
   "metadata": {},
   "source": [
    "***\n",
    "[Optional] Select a different Sagemaker pre-trained model. Here, we download the model_manifest file from the Built-In Algorithms s3 bucket, filter-out all the Text Generation models and select a model for inference.\n",
    "\n",
    "The notebook in the following sections uses [**GPT-J-6B**](https://huggingface.co/EleutherAI/gpt-j-6B) as an example. You are welcome to try [**Bloom 7b1**](https://huggingface.co/bigscience/bloom-7b1), [**Bloom 3b**](https://huggingface.co/bigscience/bloom-3b), [**GPT-NEO-2b7**](https://huggingface.co/EleutherAI/gpt-neo-2.7B) and many others yourself. Please modify `model_id` based on the dropdown list shown as below.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d8a1f7e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "\n",
    "# Retrieves all Text Generation models available by SageMaker Built-In Algorithms.\n",
    "filter_value = \"task == textgeneration1\"\n",
    "text_generation_models = list_jumpstart_models(filter=filter_value)\n",
    "\n",
    "# display the model-ids in a dropdown to select a model for inference.\n",
    "model_dropdown = Dropdown(\n",
    "    options=text_generation_models,\n",
    "    value=model_id,\n",
    "    description=\"Select a model\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28d45e5",
   "metadata": {},
   "source": [
    "#### Choose a model for Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52b7a67a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display(model_dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "271642dc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# model_version=\"*\" fetches the latest version of the model\n",
    "model_id, model_version = model_dropdown.value, \"1.*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b08aa4a",
   "metadata": {},
   "source": [
    "### 3. Retrieve Artifacts & Deploy an Endpoint\n",
    "\n",
    "***\n",
    "\n",
    "Using SageMaker, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. We start by retrieving the `deploy_image_uri`, `deploy_source_uri`, and `model_uri` for the pre-trained model. To host the pre-trained model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it. This may take a few minutes.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acff6c2a",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "631ae768",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris, hyperparameters\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-{model_id}\")\n",
    "\n",
    "inference_instance_type = \"ml.g5.12xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri. This is the base HuggingFace container image for the default model above.\n",
    "\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,  # automatically inferred from model_id\n",
    "    image_scope=\"inference\",\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "\n",
    "# Retrieve the model uri.\n",
    "model_uri = model_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, model_scope=\"inference\"\n",
    ")\n",
    "\n",
    "\n",
    "model = Model(\n",
    "    image_uri=deploy_image_uri,\n",
    "    model_data=model_uri,\n",
    "    role=aws_role,\n",
    "    predictor_cls=Predictor,\n",
    "    name=endpoint_name,\n",
    ")\n",
    "\n",
    "# deploy the Model. Note that we need to pass Predictor class when we deploy model through Model class,\n",
    "# for being able to run inference through the sagemaker API.\n",
    "model_predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    predictor_cls=Predictor,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9c254b",
   "metadata": {},
   "source": [
    "### 4. Query endpoint and parse response\n",
    "\n",
    "---\n",
    "Input to the endpoint is any string of text formatted as json and encoded in `utf-8` format. Output of the endpoint is a `json` with generated text.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "439998c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n",
    "\n",
    "\n",
    "def query_endpoint(encoded_text, endpoint_name):\n",
    "    client = boto3.client(\"runtime.sagemaker\")\n",
    "    response = client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name, ContentType=\"application/x-text\", Body=encoded_text\n",
    "    )\n",
    "    return response\n",
    "\n",
    "\n",
    "def parse_response(query_response):\n",
    "    model_predictions = json.loads(query_response[\"Body\"].read())\n",
    "    generated_text = model_predictions[0][\"generated_text\"]\n",
    "    return generated_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc5d644d",
   "metadata": {},
   "source": [
    "---\n",
    "Below, we  put in some example input text. You can put in any text and the model predicts next words in the sequence. Longer sequences of text can be generated by calling the model repeatedly.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8262dfcc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n",
    "\n",
    "text1 = \"I will\"\n",
    "text2 = \"The movie is so funny\"\n",
    "\n",
    "\n",
    "for text in [text1, text2]:\n",
    "    query_response = query_endpoint(json.dumps(text).encode(\"utf-8\"), endpoint_name=endpoint_name)\n",
    "    generated_text = parse_response(query_response)\n",
    "    print(\n",
    "        f\"Inference:{newline}\"\n",
    "        f\"input text: {text}{newline}\"\n",
    "        f\"generated text: {bold}{generated_text}{unbold}{newline}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2554851-cbcc-4ef9-864e-776a3550ceca",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5. Advanced features: How to use various advanced parameters to control the generated text\n",
    "\n",
    "***\n",
    "This model also supports many advanced parameters while performing inference. They include:\n",
    "\n",
    "* **max_length:** Model generates text until the output length (which includes the input context length) reaches `max_length`. If specified, it must be a positive integer.\n",
    "* **num_return_sequences:** Number of output sequences returned. If specified, it must be a positive integer.\n",
    "* **num_beams:** Number of beams used in the greedy search. If specified, it must be integer greater than or equal to `num_return_sequences`.\n",
    "* **no_repeat_ngram_size:** Model ensures that a sequence of words of `no_repeat_ngram_size` is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **early_stopping:** If True, text generation is finished when all beam hypotheses reach the end of stence token. If specified, it must be boolean.\n",
    "* **do_sample:** If True, sample the next word as per the likelyhood. If specified, it must be boolean.\n",
    "* **top_k:** In each step of text generation, sample from only the `top_k` most likely words. If specified, it must be a positive integer.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "* **seed:** Fix the randomized state for reproducibility. If specified, it must be an integer.\n",
    "\n",
    "We may specify any subset of the parameters mentioned above while invoking an endpoint. Next, we show an example of how to invoke endpoint with these arguments\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1af6f7b0-4093-48c9-acdb-54b05886b2dc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Input must be a json\n",
    "payload = {\n",
    "    \"text_inputs\": [\"I like living in New York\"],\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "    \"num_beams\": 3,\n",
    "}\n",
    "\n",
    "\n",
    "def query_endpoint_with_json_payload(encoded_json, endpoint_name):\n",
    "    client = boto3.client(\"runtime.sagemaker\")\n",
    "    response = client.invoke_endpoint(\n",
    "        EndpointName=endpoint_name, ContentType=\"application/json\", Body=encoded_json\n",
    "    )\n",
    "    return response\n",
    "\n",
    "\n",
    "def parse_response_multiple_texts(query_response):\n",
    "    model_predictions = json.loads(query_response[\"Body\"].read())\n",
    "    # generated_text = model_predictions[0][\"generated_texts\"]\n",
    "    generated_text = []\n",
    "    for x in model_predictions[0]:\n",
    "        generated_text.append(x[\"generated_text\"])\n",
    "    return generated_text\n",
    "\n",
    "\n",
    "query_response = query_endpoint_with_json_payload(\n",
    "    json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    ")\n",
    "\n",
    "generated_texts = parse_response_multiple_texts(query_response)\n",
    "print(generated_texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c54977c4-d91a-4489-b137-26fa1d5f1f2d",
   "metadata": {},
   "source": [
    "### 6. Advanced features: How to use prompts engineering to solve different few shot learning NLP tasks\n",
    "\n",
    "<font color='red'>Note </font>.  The notebook in the following sections uses [**GPT-J-6B**](https://huggingface.co/EleutherAI/gpt-j-6B) as an example. You are welcome to try [**Bloom 7b1**](https://huggingface.co/bigscience/bloom-7b1), [**Bloom 3b**](https://huggingface.co/bigscience/bloom-3b), [**GPT-NEO-2b7**](https://huggingface.co/EleutherAI/gpt-neo-2.7B) and many others yourself. Please modify `model_id` in above section."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ec5b958-1f20-4899-a02f-e3854e804c27",
   "metadata": {},
   "source": [
    "### 6.1. Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68a1b142-bf45-4e4e-9535-57184ad986df",
   "metadata": {},
   "source": [
    "Define the text article you want to summarize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84c4893d-5673-428d-90b8-09583987151a",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"Amazon Comprehend uses natural language processing (NLP) to extract insights about the content of documents. It develops insights by recognizing the entities, key phrases, language, sentiments, and other common elements in a document. Use Amazon Comprehend to create new products based on understanding the structure of documents. For example, using Amazon Comprehend you can search social networking feeds for mentions of products or scan an entire document repository for key phrases. \n",
    "You can access Amazon Comprehend document analysis capabilities using the Amazon Comprehend console or using the Amazon Comprehend APIs. You can run real-time analysis for small workloads or you can start asynchronous analysis jobs for large document sets. You can use the pre-trained models that Amazon Comprehend provides, or you can train your own custom models for classification and entity recognition. \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e746d6f-317b-4cb4-a334-11b9b3eb55d5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"[Original]: Amazon scientists, in collaboration with researchers from the University of Sheffield, are making a large-scale fact extraction and verification dataset publicly available for the first time. The dataset, comprising more than 185,000 evidence-backed claims, is being made available to hopefully catalyze research and development that addresses the problems of fact extraction and verification in software applications or cloud-based services that perform automatic information extraction.\n",
    "    [Summary]: Amazon and University researchers make fact extraction and verification dataset publicly available.\n",
    "    ###\n",
    "    [Original]: Prime members in the U.S. can get even more delivered to their door with a Prime membership. Members can now enjoy one year of Grubhub+ valued at $9.99 per month for free—at no added cost to their Prime membership. To activate this deal, visit amazon.com/grubhub. This new offer includes unlimited, $0 food delivery fees on orders over $12 as well as exclusive perks for Grubhub+ members and rewards like free food and order discounts. Plus, diners can “eat good while doing good” by opting into Grubhub’s Donate the Change program, a donation-matching initiative that raised more than $25 million in 2021 alone, benefiting more than 20 charitable organizations across the country.\n",
    "    [Summary]: Prime members in the U.S. can enjoy one year of Grubhub+ for free, with no food-delivery fees on eligible orders.\n",
    "    ###\n",
    "    [Original]: {text}\n",
    "    [Summary]:\"\"\"\n",
    "]\n",
    "\n",
    "num_return_sequences = 1\n",
    "parameters = {\n",
    "    \"max_length\": 600,\n",
    "    \"num_return_sequences\": num_return_sequences,\n",
    "    \"top_p\": 0.01,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "print(f\"{bold}Number of return sequences are set as {num_return_sequences}{unbold}{newline}\")\n",
    "for each_prompt in prompts:\n",
    "    payload = {\"text_inputs\": each_prompt.replace(\"{text}\", text), **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} For prompt: '{each_prompt}'{unbold}{newline}\")\n",
    "    print(f\"{bold} The {num_return_sequences} summarized results are{unbold}:{newline}\")\n",
    "    for idx, each_generated_text in enumerate(generated_texts):\n",
    "        print(f\"{bold}Result {idx}{unbold}: {each_generated_text}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c357c0dc-2644-4ce6-b579-f61c60a0777e",
   "metadata": {},
   "source": [
    "### 6.2. Code generation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f33632b8-b054-47ba-a3f0-f59fadc7bb4f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "description = \"a Headline saying Welcome to AI\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "633e0c14-19c5-4bfe-9a0a-584dffadc5f3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"description: a orange button that says stop\n",
    "    code: <button style=color:white; background-color:orange;>Stop</button>\n",
    "    ###\n",
    "    description: a blue box that contains yellow circles with red borders\n",
    "    code: <div style=background-color: blue; padding: 20px;><div style=background-color: yellow; border: 5px solid red; border-radius: 50%; padding: 20px; width: 100px; height: 100px;>\n",
    "    ###\n",
    "    description: {description}\n",
    "    code:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 200,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.3,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{description}\", description)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8340134-5938-40c4-8aab-ed05b84f8beb",
   "metadata": {},
   "source": [
    "How about SQL code?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29d6d889-5ea6-4d55-abd1-33d875f7df9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"Fetch three employees from the Employee table\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d937ec4-3987-4f40-a839-b82177c060e2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Question: Fetch the companies that have less than five people in it.\n",
    "    Answer: SELECT COMPANY, COUNT(EMPLOYEE_ID) FROM Employee GROUP BY COMPANY HAVING COUNT(EMPLOYEE_ID) < 5;\n",
    "    ###\n",
    "    Question: Show all companies along with the number of employees in each department\n",
    "    Answer: SELECT COMPANY, COUNT(COMPANY) FROM Employee GROUP BY COMPANY;\n",
    "    ###\n",
    "    Question: Show the last record of the Employee table\n",
    "    Answer: SELECT * FROM Employee ORDER BY LAST_NAME DESC LIMIT 1;\n",
    "    ###\n",
    "    Question: {question};\n",
    "    Answer:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 200,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_p\": 0.01,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{question}\", question)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8f7ac9-695d-4707-b857-4244236ec101",
   "metadata": {},
   "source": [
    "### 6.3. Name entity recognition (NER)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13666249-8961-4c94-a702-4af4d2b38314",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentence = \"David Melvin is an investment and financial services professional at CITIC CLSA with over 30 years’ experience in investment banking and private equity. He is currently a Senior Adviser of CITIC CLSA.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f072e98-2489-4404-a766-9ac9e5cf1d62",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"[Text]: Fred is a serial entrepreneur. Co-founder and CEO of Platform.sh, he previously co-founded Commerce Guys, a leading Drupal ecommerce provider. His mission is to guarantee that as we continue on an ambitious journey to profoundly transform how cloud computing is used and perceived, we keep our feet well on the ground continuing the rapid growth we have enjoyed up until now. \n",
    "    [Name]: Fred\n",
    "    [Position]: Co-founder and CEO\n",
    "    [Company]: Platform.sh\n",
    "    ###\n",
    "    [Text]: Microsoft (the word being a portmanteau of \"microcomputer software\") was founded by Bill Gates on April 4, 1975, to develop and sell BASIC interpreters for the Altair 8800. Steve Ballmer replaced Gates as CEO in 2000, and later envisioned a \"devices and services\" strategy.\n",
    "    [Name]:  Steve Ballmer\n",
    "    [Position]: CEO\n",
    "    [Company]: Microsoft\n",
    "    ###\n",
    "    [Text]: Franck Riboud was born on 7 November 1955 in Lyon. He is the son of Antoine Riboud, the previous CEO, who transformed the former European glassmaker BSN Group into a leading player in the food industry. He is the CEO at Danone.\n",
    "    [Name]:  Franck Riboud\n",
    "    [Position]: CEO\n",
    "    [Company]: Danone\n",
    "    ###\n",
    "    [Text]: {sentence}\n",
    "    \"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 550,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_p\": 0.1,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "310ef8b5-b913-47a5-979e-79debb651a0b",
   "metadata": {},
   "source": [
    "### 6.4. Question answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2df4f7ee-5067-4033-9075-99521965a0f2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"Which plan is recommended for GPT-J?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b14ff933-2238-47b8-82b0-8b6d345e7241",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Context: NLP Cloud was founded in 2021 when the team realized there was no easy way to reliably leverage Natural Language Processing in production.\n",
    "    Question: When was NLP Cloud founded?\n",
    "    Answer: 2021\n",
    "    ###\n",
    "    Context: NLP Cloud developed their API by mid-2020 and they added many pre-trained open-source models since then.\n",
    "    Question: What did NLP Cloud develop?\n",
    "    Answer: API\n",
    "    ###\n",
    "    Context: All plans can be stopped anytime. You only pay for the time you used the service. In case of a downgrade, you will get a discount on your next invoice.\n",
    "    Question: When can plans be stopped?\n",
    "    Answer: Anytime\n",
    "    ###\n",
    "    Context: The main challenge with GPT-J is memory consumption. Using a GPU plan is recommended.\n",
    "    Question: {question}\n",
    "    Answer:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 350,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_p\": 0.1,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{question}\", question)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "368d9c05-ffc9-4d2b-a53b-d7d5ef5481ba",
   "metadata": {},
   "source": [
    "### 6.5. Grammar and spelling correction "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2727c519-ba16-460c-a601-c72a16051f35",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentence = \"I do not wan to go\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52c87de5-63e7-4ae3-af1a-f2cbf1f7bf08",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"I love goin to the beach.\n",
    "    Correction: I love going to the beach.\n",
    "    ###\n",
    "    Let me hav it!\n",
    "    Correction: Let me have it!\n",
    "    ###\n",
    "    It have too many drawbacks.\n",
    "    Correction: It has too many drawbacks.\n",
    "    ###\n",
    "    {sentence}\n",
    "    Correction:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 250,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_p\": 0.1,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c00febfb-9f98-4f45-8204-a79f134ed17e",
   "metadata": {},
   "source": [
    "### 6.6. Product description and generalization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0a08432-60ef-430f-b564-1c59c84bd2ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"\"\"t-shirt, men, $39\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a70c89f-42d5-4e2c-b3c1-4cfaf4e3f655",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Generate a product description out of keywords.\n",
    "\n",
    "    Keywords: shoes, women, $59\n",
    "    Sentence: Beautiful shoes for women at the price of $59.\n",
    "    ###\n",
    "    Keywords: trousers, men, $69\n",
    "    Sentence: Modern trousers for men, for $69 only.\n",
    "    ###\n",
    "    Keywords: gloves, winter, $19\n",
    "    Sentence: Amazingly hot gloves for cold winters, at $19.\n",
    "    ###\n",
    "    Keywords: {sentence}\n",
    "    Sentence:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 150,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{each_prompt}'{newline}\")\n",
    "\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13ab5c92-5210-4ffe-842a-78e8ea0a6521",
   "metadata": {},
   "source": [
    "### 6.7. Sentence / Sentiment Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f140ae5-54c0-4738-b2c2-d715f8ae8050",
   "metadata": {},
   "source": [
    "Define the sentence you want to classifiy and the corresponded options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd29e5fb-d7f9-43bd-ad03-4887c1381169",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"I am trying to cook chicken with tomatoes.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edb65e09-8c2f-4a8c-87bd-64ab49a067b0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Message: When the spaceship landed on Mars, the whole humanity was excited\n",
    "    Topic: space\n",
    "    ###\n",
    "    Message: I love playing tennis and golf. I'm practicing twice a week.\n",
    "    Topic: sport\n",
    "    ###\n",
    "    Message: Managing a team of sales people is a tough but rewarding job.\n",
    "    Topic: business\n",
    "    ###\n",
    "    Message: {sentence}\n",
    "    Topic:\"\"\"\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 20,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3046a9ef-d64e-42ec-a1ca-107a15f7164f",
   "metadata": {},
   "source": [
    "### 6.8. Chatbot and Conversational AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f151f47-a656-42aa-9b6b-8cc6e4ced876",
   "metadata": {},
   "source": [
    "Define the sentence and the language you want to translate the sentence to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f8885f-76bd-4b42-92ac-cdeb905c97a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"I caught flu\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7259e6a-45c1-49af-a7c8-f176f3552ee8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"This is a discussion between a [human] and a [robot]. \n",
    "    The [robot] is very nice and empathetic.\n",
    "\n",
    "    [human]: Hello nice to meet you.\n",
    "    [robot]: Nice to meet you too.\n",
    "    ###\n",
    "    [human]: How is it going today?\n",
    "    [robot]: Not so bad, thank you! How about you?\n",
    "    ###\n",
    "    [human]: I am ok, but I am a bit sad...\n",
    "    [robot]: Oh? Why that?\n",
    "    ###\n",
    "    [human]: {sentence}\n",
    "    [robot]:\"\"\"\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 250,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The translated result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca7602e2-27f0-4fe7-86dd-806081810265",
   "metadata": {},
   "source": [
    "### 6.9. Tweet generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74c0b210-1c03-4af9-a2aa-a6a7225b738d",
   "metadata": {},
   "source": [
    "Define the sentence, pronoun, and options you want to reason."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d099455d-7275-46cb-a73b-75ffaf918796",
   "metadata": {},
   "outputs": [],
   "source": [
    "keyword = \"nlp\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "974aca77-2154-4162-af03-49a4f62cda9b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"keyword: markets\n",
    "    tweet: Take feedback from nature and markets, not from people\n",
    "    ###\n",
    "    keyword: children\n",
    "    tweet: Maybe we die so we can come back as children.\n",
    "    ###\n",
    "    keyword: startups\n",
    "    tweet: Startups should not worry about how to put out fires, they should worry about how to start them.\n",
    "    ###\n",
    "    keyword: {keyword}\n",
    "    tweet:\"\"\"\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 300,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{keyword}\", keyword)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efb120a1-a66d-4312-ad71-4ec632111d49",
   "metadata": {},
   "source": [
    "### 6.10. Machine translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e4a91a4-16d2-4bb7-a6bc-fde2e0603ecf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentence = \"NLP Cloud permet de deployer le NLP en production facilement.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d51584bc-1004-455d-92bd-5f24cb162309",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"Hugging Face a révolutionné le NLP.\n",
    "    Translation: Hugging Face revolutionized NLP.\n",
    "    ###\n",
    "    Cela est incroyable!\n",
    "    Translation: This is unbelievable!\n",
    "    ###\n",
    "    Désolé je ne peux pas.\n",
    "    Translation: Sorry but I cannot.\n",
    "    ###\n",
    "    {sentence}\n",
    "    Translation:\"\"\"\n",
    "]\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 150,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"do_sample\": False,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38b9a36-588f-43c2-8235-3d0d143b1196",
   "metadata": {},
   "source": [
    "### 6.11. Paraphrasing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2286ca20-8695-4fa9-9f27-a3f41f426f36",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentence = \"What is the best way to learn english?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8f4ee79-f414-4a43-b8ba-31a3a2253111",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"[Original]: Can you recommed some nice restaurants in New York?  \n",
    "[Paraphrase]: list some excellent restaurants to visit in new york city?\n",
    "###\n",
    "[Original]: Which course should I take to get started in data science?\n",
    "[Paraphrase]: What should I learn to become a data scientist?\n",
    "###\n",
    "[Original]: What are the famous places we should not miss in United States?\n",
    "[Paraphrase]: Recommend some of the best places to visit in United States?\n",
    "###\n",
    "[Original]: {sentence}\n",
    "[Paraphrase]:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 150,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_p\": 0.5,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61f0b08a-9605-4075-ad53-66766ddc24b9",
   "metadata": {},
   "source": [
    "### 6.12. Intent classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ac7e00b-cb6f-406f-be6f-2256caf2ec1f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentence = \"Can you please teach me Chinese next week?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cee0a58-a91a-47cb-93db-1844894d1e7e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"\"\"I want to start coding tomorrow because it seems to be so fun!\n",
    "    Intent: start coding\n",
    "    ###\n",
    "    Show me the last pictures you have please.\n",
    "    Intent: show pictures\n",
    "    ###\n",
    "    Search all these files as fast as possible.\n",
    "    Intent: search files\n",
    "    ###\n",
    "    {sentence}\n",
    "    Intent:\"\"\"\n",
    "]\n",
    "\n",
    "\n",
    "parameters = {\n",
    "    \"max_length\": 150,\n",
    "    \"num_return_sequences\": 1,\n",
    "    \"top_p\": 0.5,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "for each_prompt in prompts:\n",
    "    input_text = each_prompt.replace(\"{sentence}\", sentence)\n",
    "    print(f\"{bold} For prompt{unbold}: '{input_text}'{newline}\")\n",
    "    payload = {\"text_inputs\": input_text, **parameters}\n",
    "    query_response = query_endpoint_with_json_payload(\n",
    "        json.dumps(payload).encode(\"utf-8\"), endpoint_name=endpoint_name\n",
    "    )\n",
    "    generated_texts = parse_response_multiple_texts(query_response)\n",
    "    print(f\"{bold} The reasoning result is{unbold}: '{generated_texts}'{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa5de21f",
   "metadata": {},
   "source": [
    "### 7. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69b588d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "model_predictor.delete_model()\n",
    "model_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3de5d96",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**For domain adaption finetuning of text generation models such as GPT-J 6B, please check the notebook `domain-adaption-finetuning-gpt-j-6b.ipynb` in the same directory.**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4f08c710",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart-foundation-models|text-generation-few-shot-learning.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}