 archived/HuggingFace-Async-Inference-Walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Amazon SageMaker Asynchronous Inference with Hugging Face Model\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_**A new near real-time Inference option for generating machine learning model predictions**_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of Contents**\n",
    "\n",
    "* [Background](#background)\n",
    "* [Notebook Scope](#scope)\n",
    "* [Overview and sample end to end flow](#overview)\n",
    "* [Section 1 - Setup](#setup) \n",
    "    * [Create Model](#createmodel)\n",
    "    * [Create EndpointConfig](#endpoint-config)\n",
    "    * [Create Endpoint](#create-endpoint)\n",
    "* [Section 2 - Using the Endpoint](#endpoint) \n",
    "    * [Invoke Endpoint](#invoke-endpoint)\n",
    "    * [Check Output Location](#check-output) \n",
    "* [Section 3 - Clean up](#clean)\n",
    "\n",
    "### Background <a id='background'></a>  \n",
    "Amazon SageMaker Asynchronous Inference is a new capability in SageMaker that queues incoming requests and processes them asynchronously. SageMaker currently offers two inference options for customers to deploy machine learning models: 1) a real-time option for low-latency workloads 2) Batch transform, an offline option to process inference requests on batches of data available upfront. Real-time inference is suited for workloads with payload sizes of less than 6 MB and require inference requests to be processed within 60 seconds. Batch transform is suitable for offline inference on batches of data. \n",
    "\n",
    "Asynchronous inference is a new inference option for near real-time inference needs. Requests can take up to 15 minutes to process and have payload sizes of up to 1 GB. Asynchronous inference is suitable for workloads that do not have sub-second latency requirements and have relaxed latency requirements. For example, you might need to process an inference on a large image of several MBs within 5 minutes. In addition, asynchronous inference endpoints let you control costs by scaling down endpoints instance count to zero when they are idle, so you only pay when your endpoints are processing requests. \n",
    "\n",
    "### Notebook scope <a id='scope'></a>  \n",
    "This notebook provides an introduction on how to use the SageMaker Asynchronous inference capability with Hugging Face models. This notebook will cover the steps required to create an Asynchronous inference endpoint and test it with some sample requests. \n",
    "\n",
    "### Overview <a id='overview'></a>\n",
    "Asynchronous inference endpoints have many similarities (and some key differences) compared to real-time endpoints. The process to create asynchronous endpoints is similar to real-time endpoints. You need to create: a model, an endpoint configuration, and then an endpoint. However, there are specific configuration parameters specific to asynchronous inference endpoints which we will explore below. \n",
    "\n",
    "Invocation of asynchronous endpoints differ from real-time endpoints. Rather than pass request payload inline with the request, you upload the payload to Amazon S3 and pass an Amazon S3 URI as a part of the request. Upon receiving the request, SageMaker provides you with a token with the output location where the result will be placed once processed. Internally, SageMaker maintains a queue with these requests and processes them. During endpoint creation, you can optionally specify an Amazon SNS topic to receive success or error notifications. Once you receive the notification that your inference request has been successfully processed, you can access the result in the output Amazon S3 location. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 1. Setup <a id='setup'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we ensure we have an updated version of Sagemaker, which includes the latest SageMaker features:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the required python libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install --upgrade pip --quiet\n",
    "!pip install -U awscli --quiet\n",
    "!pip install --upgrade sagemaker --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime\n",
    "from sagemaker import image_uris\n",
    "import sagemaker\n",
    "import logging\n",
    "import boto3\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logger = logging.getLogger(\"__name__\")\n",
    "logger.setLevel(logging.DEBUG)\n",
    "logger.addHandler(logging.StreamHandler())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logger.info(f\"Using SageMaker version: {sagemaker.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region = sagemaker.Session().boto_region_name\n",
    "role = sagemaker.get_execution_role()\n",
    "boto3.setup_default_session(region_name=region)\n",
    "boto_session = boto3.Session(region_name=region)\n",
    "sm_session = sagemaker.session.Session()\n",
    "sagemaker_client = boto_session.client(\"sagemaker\")\n",
    "sm_runtime = boto_session.client(\"sagemaker-runtime\")\n",
    "s3_bucket = sm_session.default_bucket()\n",
    "current_timestamp = strftime(\"%m-%d-%H-%M\", gmtime())\n",
    "logger.info(f\"Region = {region}\")\n",
    "logger.info(f\"Role = {role}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify your IAM role. Go the AWS IAM console (https://console.aws.amazon.com/iam/home) and add the following policies to your IAM Role:\n",
    "* SageMakerFullAccessPolicy\n",
    "* Amazon S3 access: Apply this to get and put objects in your Amazon S3 bucket. Replace `bucket_name` with the name of your Amazon S3 bucket:      \n",
    "\n",
    "```json\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Action\": [\n",
    "                \"s3:GetObject\",\n",
    "                \"s3:PutObject\",\n",
    "                \"s3:AbortMultipartUpload\",\n",
    "                \"s3:ListBucket\"\n",
    "            ],\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Resource\": \"arn:aws:s3:::<bucket_name>/*\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "\n",
    "* (Optional) Amazon SNS access: Add `sns:Publish` on the topics you define. Apply this if you plan to use Amazon SNS to receive notifications.\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Action\": [\n",
    "                \"sns:Publish\"\n",
    "            ],\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Resource\": \"arn:aws:sns:us-east-2:123456789012:MyTopic\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "\n",
    "* (Optional) KMS decrypt, encrypt if your Amazon S3 bucket is encrypted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify your SageMaker IAM Role (`role`) and Amazon S3 bucket . You can optionally use a default SageMaker Session IAM Role and Amazon S3 bucket. Make sure the role you use has the necessary permissions for SageMaker, Amazon S3, and optionally Amazon SNS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Create Model <a id='createmodel'></a>\n",
    "Specify the location of the pre-trained model stored in Amazon S3. This example uses a pre-trained Hugging Face model name (https://huggingface.co/finiteautomata/beto-sentiment-analysis) sentimentanalysis.tar.gz. The full Amazon S3 URI is stored in a string variable `MODEL_DATA_URL`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_DATA_URL = \"s3://asyncendpointexperiment/sentimentanalysis.tar.gz\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify a primary container. For the primary container, you specify the Docker image that contains inference code, artifacts (from prior training), and a custom environment map that the inference code uses when you deploy the model for predictions. In this example, we retrieve the appropriate container image by specifying the right framework version and framework details. Here in this case we are downloading container image associated with Hugging Face framework. For further details on right container images to use for your use case please refer to this link https://github.com/awsdocs/amazon-sagemaker-developer-guide/blob/master/doc_source/ and look in to appropriate ecr folder pertaining to the region of your interest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ecr_image = image_uris.retrieve(\n",
    "    framework=\"huggingface\",\n",
    "    region=region,\n",
    "    version=\"4.6.1\",\n",
    "    image_scope=\"inference\",\n",
    "    base_framework_version=\"pytorch1.7.1\",\n",
    "    py_version=\"py36\",\n",
    "    container_version=\"ubuntu18.04\",\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    ")\n",
    "ecr_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = f\"beto-sentiment-analysis-async\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a model by specifying the `ModelName`, the `ExecutionRoleARN` (the ARN of the IAM role that Amazon SageMaker can assume to access model artifacts/ docker images for deployment), and the `PrimaryContainer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sagemaker_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\n",
    "        \"Image\": ecr_image,\n",
    "        \"ModelDataUrl\": MODEL_DATA_URL,\n",
    "        \"Environment\": {\n",
    "            \"HF_MODEL_ID\": \"finiteautomata/beto-sentiment-analysis\",\n",
    "            \"HF_TASK\": \"text-classification\",\n",
    "            \"SAGEMAKER_CONTAINER_LOG_LEVEL\": \"20\",\n",
    "            \"SAGEMAKER_REGION\": region,\n",
    "        },\n",
    "    },\n",
    ")\n",
    "model_arn = response[\"ModelArn\"]\n",
    "\n",
    "logger.info(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_config_name = model_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Create EndpointConfig <a id='endpointconfig'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have a model, create an endpoint configuration with CreateEndpointConfig. Amazon SageMaker hosting services uses this configuration to deploy models. In the configuration, you identify one or more models that were created using with CreateModel API, to deploy the resources that you want Amazon SageMaker to provision. Specify the AsyncInferenceConfig object and provide an output Amazon S3 location for OutputConfig. You can optionally specify Amazon SNS topics on which to send notifications about prediction results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sagemaker_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant-1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.m5.xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "        }\n",
    "    ],\n",
    "    AsyncInferenceConfig={\n",
    "        \"OutputConfig\": {\n",
    "            \"S3OutputPath\": f\"s3://{s3_bucket}/output\",\n",
    "            # Optionally specify Amazon SNS topics\n",
    "            # \"NotificationConfig\": {\n",
    "            #   \"SuccessTopic\": \"arn:aws:sns:us-east-2:123456789012:MyTopic\",\n",
    "            #   \"ErrorTopic\": \"arn:aws:sns:us-east-2:123456789012:MyTopic\",\n",
    "            # }\n",
    "        },\n",
    "    },\n",
    ")\n",
    "endpoint_config_arn = response[\"EndpointConfigArn\"]\n",
    "logger.info(f\"Created EndpointConfig: {endpoint_config_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Create Endpoint <a id='create-endpoint'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have your model and endpoint configuration, use the CreateEndpoint API to create your endpoint. The endpoint name must be unique within an AWS Region in your AWS account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name = model_name\n",
    "response = sagemaker_client.create_endpoint(\n",
    "    EndpointName=\"HuggingFaceAsyncEndpoint\", EndpointConfigName=\"beto-sentiment-analysis-async\"\n",
    ")\n",
    "endpoint_arn = response[\"EndpointArn\"]\n",
    "logger.info(f\"Created Endpoint: {endpoint_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "## 2. Using the Endpoint <a id='endpoint'></a>\n",
    "\n",
    "### 2.1 Uploading the Request Payload <a id='upload'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample input.json placed in the input location\n",
    "\n",
    "{\"inputs\": [\"I like you. I love you\",\"This is sad\",\"am so happy that i want to cry\",\"async endpoints are awesome\"]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_s3_location = f\"s3://{s3_bucket}/input/input.json\"\n",
    "print(input_s3_location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Invoke Endpoint   <a id='invoke-endpoint'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get inferences from the model hosted at your asynchronous endpoint with InvokeEndpointAsync. Specify the location of your inference data in the InputLocation field and the name of your endpoint for EndpointName. The response payload contains the output Amazon S3 location where the result will be placed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = sm_runtime.invoke_endpoint_async(\n",
    "    EndpointName=\"HuggingFaceAsyncEndpoint\",\n",
    "    InputLocation=input_s3_location,\n",
    "    ContentType=\"application/json\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Check Output Location <a id='check-output'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the output location to see if the inference has been processed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample inference output processed and  placed in the output location\n",
    "\n",
    "[{\"label\":\"POS\",\"score\":0.9982852339744568},{\"label\":\"NEG\",\"score\":0.9333241581916809},{\"label\":\"POS\",\"score\":0.595783531665802},{\"label\":\"NEU\",\"score\":0.9964613318443298}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Summary & Clean up <a id='clean'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To Summarize, In this notebook we learned how to use the SageMaker Asynchronous inference capability with pre-trained Hugging Face models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you enabled auto-scaling for your endpoint, ensure you deregister the endpoint as a scalable target before deleting the endpoint. To do this, run the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = client.deregister_scalable_target(\n",
    "    ServiceNamespace=\"sagemaker\",\n",
    "    ResourceId=\"resource_id\",\n",
    "    ScalableDimension=\"sagemaker:variant:DesiredInstanceCount\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember to delete your endpoint after use as you will be charged for the instances used in this Demo. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may also want to delete any other resources you might have created such as SNS topics, S3 objects, etc."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/async-inference|HuggingFace-Async-Inference-Walkthrough.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.g4dn.xlarge",
  "kernelspec": {
   "display_name": "Python 3 (PyTorch 1.6 Python 3.6 GPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:eu-west-1:470317259841:image/pytorch-1.6-gpu-py36-cu110-ubuntu18.04-v3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}