 archived/Amazon_JumpStart_Text_Generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "82b0442d",
   "metadata": {},
   "source": [
    "# Introduction to SageMaker Built-In Algorithms - Text Generation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "983f8b47",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "656ad3c2",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to Amazon [SageMaker Built-In Algorithms](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use Sagemaker Built-In Algorithms to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html#use-prebuilt-models-with-sagemaker-jumpstart). \n",
    "\n",
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK for Text Generation. Text generation is the task of generating text which appears indistinguishable from the human-written text. It is also sometimes known as \"natural language generation\". Here, we show how to use state-of-the-art pre-trained GPT models for Text Generation. We also demonstrate running inference on any [Text Generation](https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads) model available on HugginFace\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ff5ed8e",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a model](#2.-Select-a-model)\n",
    "3. [Retrieve Artifacts & Deploy an Endpoint](#3.-Retrieve-Artifacts-&-Deploy-an-Endpoint)\n",
    "4. [Query endpoint and parse response](#4.-Query-endpoint-and-parse-response)\n",
    "5. [Advanced features](#5.-Advanced-features)\n",
    "6. [Clean up the endpoint](#5.-Clean-up-the-endpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc221c3",
   "metadata": {},
   "source": [
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_pytorch_p39 kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4890e5e2",
   "metadata": {},
   "source": [
    "### 1. Set Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2f2a30b",
   "metadata": {},
   "source": [
    "---\n",
    "Before executing the notebook, there are some initial steps required for set up. This notebook requires ipywidgets.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adcd3f0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sagemaker ipywidgets --upgrade --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d51f601",
   "metadata": {},
   "source": [
    "#### Permissions and environment variables\n",
    "\n",
    "---\n",
    "To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13621a75",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker.session import Session\n",
    "\n",
    "sagemaker_session = Session()\n",
    "aws_role = sagemaker_session.get_caller_identity_arn()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d23ec4",
   "metadata": {},
   "source": [
    "## 2. Select a pre-trained model\n",
    "***\n",
    "You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of SageMaker pre-trained models can also be accessed at [JumpStart pre-trained Models](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#).\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7b89b4c",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ]
   },
   "outputs": [],
   "source": [
    "# Please use model_version as 2.* if you're using the open-llama model\n",
    "model_id, model_version, = (\n",
    "    \"huggingface-textgeneration-gpt2\",\n",
    "    \"1.*\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7c4f52",
   "metadata": {},
   "source": [
    "***\n",
    "[Optional] Select a different Sagemaker pre-trained model. Here, we download the model_manifest file from the Built-In Algorithms s3 bucket, filter-out all the Text Generation models and select a model for inference.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7003aa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "from sagemaker.jumpstart.filters import And\n",
    "\n",
    "# Retrieves all Text Generation models available by SageMaker Built-In Algorithms.\n",
    "filter_value = And(\"task == textgeneration\", \"framework == huggingface\")\n",
    "text_generation_models = list_jumpstart_models(filter=filter_value)\n",
    "\n",
    "# display the model-ids in a dropdown to select a model for inference.\n",
    "model_dropdown = widgets.Dropdown(\n",
    "    options=text_generation_models,\n",
    "    value=model_id,\n",
    "    description=\"Select a model\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b8c17d5",
   "metadata": {},
   "source": [
    "#### Chose a model for Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcc10a2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(model_dropdown)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2269025",
   "metadata": {},
   "source": [
    "### Using Models not Present in the Dropdown\n",
    "***\n",
    "If you want to choose any other model which is not present in the dropdown and is available at HugginFace [Text Generation](https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads) please choose huggingface-textgeneration-models in the dropdown and pass the model_id in the HF_MODEL_ID variable. Inference on the models listed in the dropdown menu can be run in [network isolation](https://docs.aws.amazon.com/sagemaker/latest/dg/mkt-algo-model-internet-free.html). In such a case, no inbound or outbound network calls can be made to or from the model container. The models listed in the dropdown can also be deployed with custom VPC settings, which can provide your model container with a network connection within your VPC that is not connected to the internet. Refer to [AWS documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/host-vpc.html) for more details.\n",
    "\n",
    "However, when running inference on a model specified through HF_MODEL_ID, the model container will download the model artifact from HuggingFace. Therefore, the model container cannot run in network isolation. Furthermore, if you want to use custom VPC settings, you must provide access the HuggingFace portal in your VPC.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56f322fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model_version=\"*\" fetches the latest version of the model\n",
    "model_id, model_version = model_dropdown.value, \"1.*\"\n",
    "\n",
    "hub = {}\n",
    "HF_MODEL_ID = \"xlnet-base-cased\"  # Pass any other HF_MODEL_ID from - https://huggingface.co/models?pipeline_tag=text-classification&sort=downloads\n",
    "if model_id == \"huggingface-textgeneration-models\":\n",
    "    hub[\"HF_MODEL_ID\"] = HF_MODEL_ID\n",
    "    hub[\"HF_TASK\"] = \"text-generation\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9619c64",
   "metadata": {},
   "source": [
    "### 3. Retrieve Artifacts & Deploy an Endpoint\n",
    "\n",
    "***\n",
    "\n",
    "Using SageMaker, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. We start by retrieving the `deploy_image_uri`, `deploy_source_uri`, and `model_uri` for the pre-trained model. To host the pre-trained model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it. This may take a few minutes.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "067fa2fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris, hyperparameters\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-{model_id}\")\n",
    "\n",
    "inference_instance_type = \"ml.p3.2xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri. This is the base HuggingFace container image for the default model above.\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,  # automatically inferred from model_id\n",
    "    image_scope=\"inference\",\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "\n",
    "\n",
    "# Retrieve the model uri. This includes the pre-trained nvidia-ssd model and parameters.\n",
    "model_uri = model_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, model_scope=\"inference\"\n",
    ")\n",
    "\n",
    "\n",
    "# Create the SageMaker model instance\n",
    "model = Model(\n",
    "    image_uri=deploy_image_uri,\n",
    "    model_data=model_uri,\n",
    "    role=aws_role,\n",
    "    predictor_cls=Predictor,\n",
    "    name=endpoint_name,\n",
    "    env=hub,\n",
    ")\n",
    "\n",
    "# deploy the Model. Note that we need to pass Predictor class when we deploy model through Model class,\n",
    "# for being able to run inference through the sagemaker API.\n",
    "model_predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    predictor_cls=Predictor,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d0c349",
   "metadata": {},
   "source": [
    "### 4. Query endpoint and parse response\n",
    "\n",
    "---\n",
    "Input to the endpoint is any string of text dumped in json and encoded in `utf-8` format. Output of the endpoint is a `json` with generated text.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b0166be",
   "metadata": {},
   "outputs": [],
   "source": [
    "def query(model_predictor, text):\n",
    "    \"\"\"Query the model predictor.\"\"\"\n",
    "\n",
    "    encoded_text = text.encode(\"utf-8\")\n",
    "\n",
    "    query_response = model_predictor.predict(\n",
    "        encoded_text,\n",
    "        {\n",
    "            \"ContentType\": \"application/x-text\",\n",
    "            \"Accept\": \"application/json\",\n",
    "        },\n",
    "    )\n",
    "    return query_response\n",
    "\n",
    "\n",
    "def parse_response(query_response):\n",
    "    \"\"\"Parse response and return the generated text.\"\"\"\n",
    "\n",
    "    model_predictions = json.loads(query_response)\n",
    "    generated_text = model_predictions[\"generated_text\"]\n",
    "    return generated_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e876cf29",
   "metadata": {},
   "source": [
    "---\n",
    "Below, we  put in some example input text. You can put in any text and the model predicts next words in the sequence. Longer sequences of text can be generated by calling the model repeatedly.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0287ba94",
   "metadata": {},
   "outputs": [],
   "source": [
    "newline, bold, unbold = \"\\n\", \"\\033[1m\", \"\\033[0m\"\n",
    "\n",
    "text1 = \"As far as I am concerned, I will\"\n",
    "text2 = \"The movie is\"\n",
    "\n",
    "for text in [text1, text2]:\n",
    "    query_response = query(model_predictor, text)\n",
    "    generated_text = parse_response(query_response)\n",
    "    print(f\"Input text: {text}{newline}\" f\"Generated text: {bold}{generated_text}{unbold}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81bba04a",
   "metadata": {},
   "source": [
    "### 5. Advanced features\n",
    "\n",
    "***\n",
    "This model also supports many advanced parameters while performing inference. They include:\n",
    "\n",
    "* **max_length:** Model generates text until the output length (which includes the input context length) reaches `max_length`. If specified, it must be a positive integer.\n",
    "* **num_return_sequences:** Number of output sequences returned. If specified, it must be a positive integer.\n",
    "* **num_beams:** Number of beams used in the greedy search. If specified, it must be integer greater than or equal to `num_return_sequences`.\n",
    "* **no_repeat_ngram_size:** Model ensures that a sequence of words of `no_repeat_ngram_size` is not repeated in the output sequence. If specified, it must be a positive integer greater than 1.\n",
    "* **temperature:** Controls the randomness in the output. Higher temperature results in output sequence with low-probability words and lower temperature results in output sequence with high-probability words. If `temperature` -> 0, it results in greedy decoding. If specified, it must be a positive float.\n",
    "* **early_stopping:** If True, text generation is finished when all beam hypotheses reach the end of stence token. If specified, it must be boolean.\n",
    "* **do_sample:** If True, sample the next word as per the likelyhood. If specified, it must be boolean.\n",
    "* **top_k:** In each step of text generation, sample from only the `top_k` most likely words. If specified, it must be a positive integer.\n",
    "* **top_p:** In each step of text generation, sample from the smallest possible set of words with cumulative probability `top_p`. If specified, it must be a float between 0 and 1.\n",
    "* **seed:** Fix the randomized state for reproducibility. If specified, it must be an integer.\n",
    "* **return_full_text:** If True, input text will be part of the output generated text. If specified, it must be boolean. The default value for it is False.\n",
    "\n",
    "We may specify any subset of the parameters mentioned above while invoking an endpoint. Next, we show an example of how to invoke endpoint with these arguments\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29e64c46",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "payload = {\n",
    "    \"text_inputs\": \"My name is Lewis and I like to\",\n",
    "    \"max_length\": 50,\n",
    "    \"num_return_sequences\": 3,\n",
    "    \"top_k\": 50,\n",
    "    \"top_p\": 0.95,\n",
    "    \"do_sample\": True,\n",
    "}\n",
    "\n",
    "\n",
    "def query_endpoint_with_json_payload(model_predictor, payload):\n",
    "    \"\"\"Query the model predictor with json payload.\"\"\"\n",
    "\n",
    "    encoded_payload = json.dumps(payload).encode(\"utf-8\")\n",
    "\n",
    "    query_response = model_predictor.predict(\n",
    "        encoded_payload,\n",
    "        {\n",
    "            \"ContentType\": \"application/json\",\n",
    "            \"Accept\": \"application/json\",\n",
    "        },\n",
    "    )\n",
    "    return query_response\n",
    "\n",
    "\n",
    "def parse_response_multiple_texts(query_response):\n",
    "    \"\"\"Parse response and return the generated texts.\"\"\"\n",
    "\n",
    "    model_predictions = json.loads(query_response)\n",
    "    generated_texts = model_predictions[\"generated_texts\"]\n",
    "    return generated_texts\n",
    "\n",
    "\n",
    "query_response = query_endpoint_with_json_payload(model_predictor, payload)\n",
    "generated_texts = parse_response_multiple_texts(query_response)\n",
    "print(f\"Input text: {text}{newline}\" f\"Generated text: {bold}{generated_texts}{unbold}{newline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81938a15",
   "metadata": {},
   "source": [
    "### 6. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ce49ebc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint\n",
    "model_predictor.delete_model()\n",
    "model_predictor.delete_endpoint()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7d3c18aa",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart_text_generation|Amazon_JumpStart_Text_Generation.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_pytorch_p39",
   "language": "python",
   "name": "conda_pytorch_p39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}