    generative_ai/sm-jumpstart_foundation_text_generation_inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e78dc97c",
   "metadata": {},
   "source": [
    "# Introduction to SageMaker JumpStart - Text Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e567a4",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4e36b9",
   "metadata": {},
   "source": [
    "In this demo notebook, we demonstrate how to use the SageMaker Python SDK to deploy a SageMaker JumpStart text generation model and invoke the endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af77efb",
   "metadata": {},
   "source": [
    "## Setup\n",
    "First, upgrade to the latest sagemaker SDK to ensure all available models are deployable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b05b931-992e-4526-978d-f03196874a3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --quiet --upgrade sagemaker jmespath"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dbb6b35",
   "metadata": {},
   "source": [
    "Select the desired model to deploy. The provided dropdown filters all text generation models available in SageMaker JumpStart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f625a488",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import Dropdown\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "\n",
    "\n",
    "dropdown = Dropdown(\n",
    "    options=list_jumpstart_models(\"search_keywords includes Text Generation\"),\n",
    "    value=\"huggingface-llm-mistral-7b-instruct\",\n",
    "    description=\"Select a JumpStart text generation model:\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")\n",
    "display(dropdown)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a40df34",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdOnly"
    ],
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id = dropdown.value\n",
    "model_version = \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0a0388",
   "metadata": {},
   "source": [
    "## Deploy model\n",
    "\n",
    "Create a `JumpStartModel` object, which initializes default model configurations conditioned on the selected instance type. JumpStart already sets a default instance type, but you can deploy the model on other instance types by passing `instance_type` to the `JumpStartModel` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85a2a8e5-789f-4041-9927-221257126653",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.jumpstart.model import JumpStartModel\n",
    "\n",
    "\n",
    "model = JumpStartModel(model_id=model_id, model_version=model_version)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1259ad4f",
   "metadata": {},
   "source": [
    "You can now deploy the model using SageMaker JumpStart. If the selected model is gated, you will need to accept the end-user license agreement (EULA) prior to deployment. This is accomplished by providing the `accept_eula=True` argument to the `deploy` method. The deployment might take few minutes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f3b42ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = model.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f10cf06c",
   "metadata": {},
   "source": [
    "## Invoke the endpoint\n",
    "\n",
    "This section demonstrates how to invoke the endpoint using example payloads that are retrieved programmatically from the `JumpStartModel` object. You can replace these example payloads with your own payloads."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbb364d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_payloads = model.retrieve_all_examples()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf5899c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import jmespath\n",
    "\n",
    "\n",
    "for payload in example_payloads:\n",
    "    response = predictor.predict(payload.body)\n",
    "    generated_text = jmespath.search(payload.raw_payload[\"output_keys\"][\"generated_text\"], response)\n",
    "    print(\"Input:\\n\", payload.body[payload.prompt_key])\n",
    "    print(\"Output:\\n\", generated_text.strip())\n",
    "    print(\"\\n===============\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a53e57",
   "metadata": {},
   "source": [
    "## Clean up the endpoint\n",
    "Don't forget to clean up resources when finished to avoid unnecessary charges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e19a2bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.delete_predictor()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4998b86",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/generative_ai|sm-jumpstart_foundation_text_generation_inference.ipynb)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}