connect-to-aws/Access_AWS_from_Studio_Lab_Deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fa8a43b3-d3ff-46df-8bb5-463c0357f29b",
   "metadata": {},
   "source": [
    "# Deploy A Hugging Face Pretrained Model to Amazon SageMaker Serverless Endpoint - Boto3\n",
    "\n",
    "[![Open In Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github/aws/studio-lab-examples/blob/main/connect-to-aws/Access_AWS_from_Studio_Lab_Deployment.ipynb)\n",
    "\n",
    "Amazon SageMaker Studio Lab provides you a free environment to develop an experimental project, which is suitable for ML learners to start their first ML lesson. The CPU session is avaliable for continously 12 hours, and GPU session is avaliable for continously 4 hours.\n",
    "\n",
    "What if you want to have a think big project for your ML learning, for instances, you want to make the model public avaliable instead of running the model locally, what can be your options? Are you thinking about the same thing? Deploy to an Amazon SageMaker Endpoint!\n",
    "\n",
    "In this tutorial, I will share you how to deploy a Hugging Face pre-trained model to an Amazon SageMaker Serverless Endpoint. By showing you: \n",
    "- (1) introduce the model  \n",
    "- (2) set up the pre-requisition - AWS user, Amazon SageMaker execution role\n",
    "- (3) connect your AWS resources(SageMaker) using boto3 \n",
    "- (4) create a SageMaker model with boto3 SageMaker Client \n",
    "- (5) create a serverless endpoint configuration \n",
    "- (6) test with boto3 SageMaker run time to invoke the endpoint \n",
    "- (7) delete the model, the endpoint config, and the endpoint\n",
    "\n",
    "![image info](./image/02-serverless-endpoint.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2218479-135b-4c5e-b6ba-b2956b3dc03f",
   "metadata": {},
   "source": [
    "# Step 1. Introduce the model  \n",
    "\n",
    "\n",
    "Hugging Face is a community and data science platform. Which provides tools that enable users to build, train and deploy ML models based on open source code and technologies. In their website, you may see open source model and dataset in their [website](https://huggingface.co/).\n",
    "\n",
    "`distilbert-base-uncased-finetuned-sst-2-english` model is an English text classification model based on the DistilBERT. [DistilBERT](https://huggingface.co/distilbert-base-uncased) is a transformers model, which is smaller and faster than BERT. It was pretrained on the same corpus and used the BERT base model as a teacher. \n",
    "\n",
    "You can test the model with python code as bellow. You can also test the model in the Hosted inference API session on the page:\n",
    "https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15ea52f1-6967-4723-8e24-d2368b379db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install huggingface_hub==0.1.0 \n",
    "%pip install transformers==4.12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab23a370-6bd5-4081-b8ae-bb38acc75e0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
    "from transformers import TextClassificationPipeline\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased-finetuned-sst-2-english\")\n",
    "\n",
    "model = AutoModelForSequenceClassification.from_pretrained(\"distilbert-base-uncased-finetuned-sst-2-english\")\n",
    "\n",
    "pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True)\n",
    "pipe(\"I love Amazon SageMaker Studio Lab!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "011a2651-79fe-45a6-a7ad-667b478637c9",
   "metadata": {},
   "source": [
    "![image info](./image/01-hugging-face-model.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75c10a5f-5cf1-4217-aff4-a520e7f10b5c",
   "metadata": {},
   "source": [
    "# Step 2. Set up the pre-requisition\n",
    "\n",
    "## 2.1 AWS user login\n",
    "You will need to install AWS CLI, boto3, and configure with your AWS credentials. You can find the credentials from the IAM > users.\n",
    "\n",
    "If you don't have users, create a new one. If you already have users, select one, and get the credentials in the Security credentials tab.\n",
    " \n",
    "- Boto3\n",
    "\n",
    "Boto3 is the AWS SDK for Python (Boto3). Which you can use it to create, configure, and manage AWS services. The SDK provides an object-oriented API as well as low-level access to AWS services. To learn more about boto3: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html\n",
    "\n",
    "- AWS CLI\n",
    "\n",
    "The AWS Command Line Interface (AWS CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. To learn more about AWS CLI: https://aws.amazon.com/cli/\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5164c77-3696-48ef-9c91-72918b44b29e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e63e9d47-b46c-4669-98c9-425e65b4b785",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install awscli"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da0ef255-285a-4637-b2ea-99f4da3319a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%mkdir ~/.aws"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e9b9094-090e-4465-9ef5-6b70a0bb03c5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile ~/.aws/credentials\n",
    "\n",
    "[default]\n",
    "aws_access_key_id =  < paste your access key here, run this cell, then delete the cell >\n",
    "aws_secret_access_key = < paste your secret key here, run this cell, then delete the cell > "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08a81657-8b85-4d46-9f92-9956079595eb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile ~/.aws/config\n",
    "\n",
    "[default]\n",
    "region=us-east-1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f3bbd43-ca37-432f-b037-53ff83bb7d11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you don't want to setup the key/secret with credentials, you can setup the environment variable with\n",
    "ACCESS_KEY= < paste your access key here, run this cell, then delete the cell >\n",
    "SECRET_KEY= < paste your secret key here, run this cell, then delete the cell > "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48aaed78-950b-458c-a2e0-349efacb016f",
   "metadata": {},
   "source": [
    "## 2.2 Amazon SageMaker execution role\n",
    "\n",
    "Amazon SageMaker execution role is an AWS Identity and Access Management (IAM) role that performs operations on your behalf on the AWS hardware that is managed by SageMaker. A SageMaker user can grant these permissions with an IAM role (referred to as an execution role).\n",
    "\n",
    "You can create an execution role by following the [instruction](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html). If you are already used to using SageMaker within your own AWS account, please copy and paste the arn for your execution role below. \n",
    "\n",
    "Please note, in order to complete this you will need to have already created this SageMaker IAM Execution role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab6826b5-9291-4eb3-911a-891af1964411",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "# create a sagemaker execution role via the AWS SageMaker console, then paste in the arn here\n",
    "role = ' < paste your execution role here > '"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5844163e-40a3-4a79-a206-954cb9cf76b6",
   "metadata": {},
   "source": [
    "# Step3. connect your AWS resources(SageMaker) using boto3\n",
    "\n",
    "Amazon SageMaker Serverless Inference is a purpose-built inference option that makes it easy for you to deploy and scale ML models. Serverless Inference is ideal for workloads which have idle periods between traffic spurts and can tolerate cold starts. Serverless endpoints automatically launch compute resources and scale them in and out depending on traffic, eliminating the need to choose instance types or manage scaling policies. This takes away the undifferentiated heavy lifting of selecting and managing servers. Serverless Inference integrates with AWS Lambda to offer you high availability, built-in fault tolerance and automatic scaling. Here is the pseudo code to show you the process to create a Amazon SageMaker serverless endpoint.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b51184c3-bad4-4c88-8607-57a4ecdc3140",
   "metadata": {},
   "source": [
    "```python\n",
    "container = {\n",
    "    'Image': CONTAINER_URL, \n",
    "    'ModelDataUrl': ‘s3://my-bucket/path/to/artifacts/’, #applies to bring your own model case\n",
    "    'Mode': 'SingleModel' } \n",
    "\n",
    "response = sm_client.create_model( \n",
    "    ModelName = 'model-name',\n",
    "    PrimaryContainer=container_config,\n",
    "    ExecutionRoleArn=role, \n",
    "    EnableNetworkIsolation=False) \n",
    "\n",
    "response = sm_client.create_endpoint_config(    \n",
    "     EndpointConfigName = ‘my-epc’, \n",
    "     ServerlessConfig=[{…,…}])\n",
    "\n",
    "response = sm_client.create_endpoint(EndpointName = ‘my-endpoint’,\n",
    "                                     EndpointConfigName = ‘my-epc’)\n",
    "```\n",
    "\n",
    "## How Amazon SageMaker Serverless Endpoints Works:\n",
    "![image info](./image/03-serverless-endpoints-how-it-works.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d0e0b26-1fc7-4b0b-a26b-440d103558c0",
   "metadata": {},
   "source": [
    "- more reference for serverless\n",
    "    - https://towardsdatascience.com/sagemaker-serverless-inference-is-now-generally-available-e42550a146fe\n",
    "    - https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints.html\n",
    "    - https://aws.amazon.com/about-aws/whats-new/2021/12/amazon-sagemaker-serverless-inference/\n",
    "    - https://github.com/huggingface/notebooks/blob/main/sagemaker/11_deploy_model_from_hf_hub/deploy_transformer_model_from_hf_hub.ipynb\n",
    "    - [Host Hugging Face transformer models using Amazon SageMaker Serverless Inference](https://aws.amazon.com/blogs/machine-learning/host-hugging-face-transformer-models-using-amazon-sagemaker-serverless-inference/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5c2f223-6e6f-4e20-ae87-48945d7ab69a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "sm_client = boto3.client('sagemaker',\n",
    "                        aws_access_key_id=ACCESS_KEY,\n",
    "                        aws_secret_access_key=SECRET_KEY)\n",
    "response = sm_client.list_endpoints()\n",
    "\n",
    "len(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a809006c-464c-4c91-9065-1f00f1b30ea2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "ml_model_name = \"text-classification-hugging-face\"\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "model_name = ml_model_name + '-model' + timestamp\n",
    "endpoint_config_name = ml_model_name + '-epc' + timestamp\n",
    "print(model_name)\n",
    "print(endpoint_config_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc1110b1-fa59-488d-b5b5-6d3927d1fb36",
   "metadata": {},
   "source": [
    "# Step 4. create a SageMaker model with boto3 SageMaker Client\n",
    "- Before create an endpoint, you need to create a model to deploy first. Here shows how to use SageMaker Boto3 client to create a model. The document can be found here: [boto3 sagemaker client create model](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model\n",
    ")\n",
    "\n",
    "- For each model, you will need to specify which Amazon SageMaker container you are going to use in this Amazon SageMaker endpoint. AWS has prebuilt deep learning containers for 5 software frameworks, including TensorFlow, PyTorch, MXNet, Hugging Face, and AutoGloun. We call it: **Deep Learning Containers(DLC)**. It depends on the use case, which can be a Tensorflow based, Pytorch based DLC image. In this example, we use Hugging Face with Pytorch as the container image. All the avalible open source framework is documented [here](https://github.com/aws/deep-learning-containers/blob/master/available_images.md). The URL composed with two part, `repository URL by region + framework version with job type, hardware spec, and python version`. \n",
    "\n",
    "- To learn more about script mode on SageMaker, check out our documentation [here](https://sagemaker.readthedocs.io/en/stable/frameworks/index.html).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b2453bf-9a30-4826-8b9d-f2e44e7e917d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_data_url=\"763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.9-transformers4.12-cpu-py38-ubuntu20.04\"\n",
    "\n",
    "container_config = {'Image': model_data_url,\n",
    "                    'Mode': 'SingleModel',\n",
    "                    'Environment': {\n",
    "                        'HF_MODEL_ID': 'distilbert-base-uncased-finetuned-sst-2-english',\n",
    "                        'HF_TASK' : 'text-classification',\n",
    "                        'SAGEMAKER_CONTAINER_LOG_LEVEL' : '20',\n",
    "                        'SAGEMAKER_REGION' : 'us-east-1'\n",
    "                    }\n",
    "                   }\n",
    "\n",
    "response = sm_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    PrimaryContainer=container_config,\n",
    "    ExecutionRoleArn=role, \n",
    "    EnableNetworkIsolation=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2949f294-2fec-4080-bc04-729b4346486a",
   "metadata": {},
   "source": [
    "- Check model is created by visiting: Console > Amazon SageMaker > Models\n",
    "\n",
    "# step 5. create a serverless endpoint configuration\n",
    "\n",
    "- In this step, we will create an endpoint configuration with ServerlessConfig, which can be used in `sm_client.create_endpoint`. The configuration detail is listed [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint_config).\n",
    "\n",
    "- The endpoint is not created until you run the `sm_client.create_endpoint`. After running this step, you will need to wait about 2 minutes for the model deployment. The detail of create endpoint API is available [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86499162-e830-4c88-86a6-bc8899539270",
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "   EndpointConfigName=endpoint_config_name,\n",
    "   ProductionVariants=[\n",
    "        {\n",
    "            \"ModelName\": model_name,\n",
    "            \"VariantName\": \"AllTraffic\",\n",
    "            \"ServerlessConfig\": {\n",
    "                # Specify MemorySizeInMB and MaxConcurrency in the serverless config object\n",
    "                \"MemorySizeInMB\": 4096,\n",
    "                \"MaxConcurrency\": 10\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "print('Endpoint configuration name: {}'.format(endpoint_config_name))\n",
    "print('Endpoint configuration arn:  {}'.format(endpoint_config_response['EndpointConfigArn']))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bde58726-686b-49f5-bfd2-4d37b002ea94",
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name = \"studio-lab-ep\" + '-epc' + timestamp\n",
    "response = sm_client.create_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    EndpointConfigName=endpoint_config_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0220fb4-8df7-42eb-8a63-6cd5c5708fc6",
   "metadata": {},
   "source": [
    "- check endpoint is created!\n",
    "    - Console > Amazon SageMaker > Endpoints\n",
    "    \n",
    "# step 6. test with boto3 SageMaker run time to invoke the endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be879b1a-2131-4428-ba63-e9ce15538bd3",
   "metadata": {},
   "source": [
    "After checking the endpoint is ready, we now use SageMaker Runtime to interact with the endpoint. According to the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/serverless-endpoints-invoke.html), you will need: \n",
    "- (1) endpoint_name, use the name of the in-service serverless endpoint you want to invoke. \n",
    "- (2) content_type, specify the MIME type of your input data in the request body (for example, application/json). If you want to learn more about what are the other options for content type, you can visit [Common Data Formats for Inference](https://docs.aws.amazon.com/sagemaker/latest/dg/cdf-inference.html) to understand more. \n",
    "- (3) payload, use your request payload for inference. Your payload should be in bytes or a file-like object.\n",
    "\n",
    "Now, let's try out the endpoint!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4376b14c-163b-4b66-90b6-2707de9497f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import boto3\n",
    "runtime = boto3.client(\"sagemaker-runtime\",\n",
    "                       aws_access_key_id=ACCESS_KEY,\n",
    "                       aws_secret_access_key=SECRET_KEY) \n",
    "\n",
    "content_type = \"application/json\"\n",
    "\n",
    "# example request, you always need to define \"inputs\"\n",
    "data = {\n",
    "   \"inputs\": \"Happy Birthday to you!\"\n",
    "}\n",
    "\n",
    "response = runtime.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType=content_type,\n",
    "    Body=json.dumps(data)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4970e596-354b-4dd1-8c99-44c869bcd8ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "response[\"Body\"].read().decode()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9753d0-92fc-47e0-adc3-4788e8f7d903",
   "metadata": {},
   "source": [
    "# step 7. delete the model, the endpoint config, and the endpoint\n",
    "\n",
    "Now, if we don't need the model, or the endpoint anymore. We will need to clean up the artifacts that we created earlier. According to the [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_endpoint): \n",
    "```\n",
    "You must not delete an EndpointConfig that is in use by an endpoint that is live or while the UpdateEndpoint or CreateEndpoint operations are being performed on the endpoint. To update an endpoint, you must create a new EndpointConfig .\n",
    "```\n",
    "\n",
    "Let's delete the artifacts in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87ef2a0a-3405-4993-8ea5-880a513d8c5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_client.delete_endpoint(EndpointName=endpoint_name)\n",
    "sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
    "sm_client.delete_model(ModelName=model_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "default:Python",
   "language": "python",
   "name": "conda-env-default-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}