sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "035cb23f",
   "metadata": {},
   "source": [
    "# Huggingface Sagemaker-sdk - Run a batch transform inference job with 🤗 Transformers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99877aa4",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)  \n",
    "2. [Run Batch Transform after training a model](#Run-Batch-Transform-after-training-a-model)  \n",
    "3. [Run Batch Transform Inference Job with a fine-tuned model using `jsonl`](#Run-Batch-Transform-Inference-Job-with-a-fine-tuned-model-using-jsonl)   \n",
    "\n",
    "Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy two transformer model for inference. \n",
    "In the first example we deploy a trained Hugging Face Transformer model on to SageMaker for inference.\n",
    "In the second example we directly deploy one of the 10 000+ Hugging Face Transformers from the [Hub](https://huggingface.co/models) to Amazon SageMaker for Inference.<"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac2fa2e7",
   "metadata": {},
   "source": [
    "## Run Batch Transform after training a model \n",
    "_not included in the notebook_\n",
    "\n",
    "After you train a model, you can use [Amazon SageMaker Batch Transform](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-batch.html) to perform inferences with the model. In Batch Transform you provide your inference data as a S3 uri and SageMaker will care of downloading it, running the prediction and uploading the results afterwards to S3 again. You can find more documentation for Batch Transform [here](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html)\n",
    "\n",
    "If you trained the model using the **HuggingFace estimator**, you can invoke `transformer()` method to create a transform job for a model based on the training job.\n",
    "\n",
    "```python\n",
    "batch_job = huggingface_estimator.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type='ml.c5.2xlarge',\n",
    "    strategy='SingleRecord')\n",
    "\n",
    "\n",
    "batch_job.transform(\n",
    "    data='s3://s3-uri-to-batch-data',\n",
    "    content_type='application/json',    \n",
    "    split_type='Line')\n",
    "```\n",
    "For more details about what can be specified here, see [API docs](https://sagemaker.readthedocs.io/en/stable/overview.html#sagemaker-batch-transform).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85a59b43",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"sagemaker>=2.48.0\" \"datasets==1.11\" --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ec5df68",
   "metadata": {},
   "source": [
    "# Run Batch Transform Inference Job with a fine-tuned model using `jsonl`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f618f99d",
   "metadata": {},
   "source": [
    "## Data Pre-Processing\n",
    "\n",
    "In this example we are using the provided `tweet_data.csv` as dataset. The `csv` contains ~1800 tweets about different airlines. The `csv` contains 1 column `\"inputs\"` with the tweets. To use this `csv` we need to convert it into a `jsonl` file and upload it to s3. Due to the complex structure of text are only `jsonl` file supported for batch transform. As pre-processing we are removing the `@` in the beginning of the tweet to get the names/identities correct.\n",
    "\n",
    "_**NOTE**: While preprocessing you need to make sure that your `inputs` fit the `max_length`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "939a2ee1-87e0-442e-a060-ba9ef3847716",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker role arn: arn:aws:iam::558105141721:role/sagemaker_execution_role\n",
      "sagemaker bucket: sagemaker-us-east-1-558105141721\n",
      "sagemaker session region: us-east-1\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "sess = sagemaker.Session()\n",
    "# sagemaker session bucket -> used for uploading data, models and logs\n",
    "# sagemaker will automatically create this bucket if it not exists\n",
    "sagemaker_session_bucket=None\n",
    "if sagemaker_session_bucket is None and sess is not None:\n",
    "    # set to default bucket if a bucket name is not given\n",
    "    sagemaker_session_bucket = sess.default_bucket()\n",
    "\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except ValueError:\n",
    "    iam = boto3.client('iam')\n",
    "    role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n",
    "\n",
    "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n",
    "print(f\"sagemaker session region: {sess.boto_region_name}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7d650fc3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tweet_data.jsonl uploaded to s3://sagemaker-us-east-1-558105141721/batch_transform/input/tweet_data.jsonl\n"
     ]
    }
   ],
   "source": [
    "import csv\n",
    "import json\n",
    "from sagemaker.s3 import S3Uploader,s3_path_join\n",
    "\n",
    "# datset files\n",
    "dataset_csv_file=\"tweet_data.csv\"\n",
    "dataset_jsonl_file=\"tweet_data.jsonl\"\n",
    "\n",
    "with open(dataset_csv_file, \"r+\") as infile, open(dataset_jsonl_file, \"w+\") as outfile:\n",
    "    reader = csv.DictReader(infile)\n",
    "    for row in reader:\n",
    "        # remove @\n",
    "        row[\"inputs\"] = row[\"inputs\"].replace(\"@\",\"\")\n",
    "        json.dump(row, outfile)\n",
    "        outfile.write('\\n')\n",
    "\n",
    "                \n",
    "# uploads a given file to S3.\n",
    "input_s3_path = s3_path_join(\"s3://\",sagemaker_session_bucket,\"batch_transform/input\")\n",
    "output_s3_path = s3_path_join(\"s3://\",sagemaker_session_bucket,\"batch_transform/output\")\n",
    "s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)\n",
    "\n",
    "print(f\"{dataset_jsonl_file} uploaded to {s3_file_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e908f14f",
   "metadata": {},
   "source": [
    "The created file looks like this\n",
    "\n",
    "```json\n",
    "{\"inputs\": \"VirginAmerica What dhepburn said.\"}\n",
    "{\"inputs\": \"VirginAmerica plus you've added commercials to the experience... tacky.\"}\n",
    "{\"inputs\": \"VirginAmerica I didn't today... Must mean I need to take another trip!\"}\n",
    "{\"inputs\": \"VirginAmerica it's really aggressive to blast obnoxious \\\"entertainment\\\"....\"}\n",
    "{\"inputs\": \"VirginAmerica and it's a really big bad thing about it\"}\n",
    "{\"inputs\": \"VirginAmerica seriously would pay $30 a flight for seats that didn't h....\"}\n",
    "{\"inputs\": \"VirginAmerica yes, nearly every time I fly VX this \\u201cear worm\\u201d won\\u2019t go away :)\"}\n",
    "{\"inputs\": \"VirginAmerica Really missed a prime opportunity for Men Without ...\"}\n",
    "{\"inputs\": \"virginamerica Well, I didn't\\u2026but NOW I DO! :-D\"}\n",
    "{\"inputs\": \"VirginAmerica it was amazing, and arrived an hour early. You're too good to me.\"}\n",
    "{\"inputs\": \"VirginAmerica did you know that suicide is the second leading cause of death among teens 10-24\"}\n",
    "{\"inputs\": \"VirginAmerica I &lt;3 pretty graphics. so much better than minimal iconography. :D\"}\n",
    "{\"inputs\": \"VirginAmerica This is such a great deal! Already thinking about my 2nd trip ...\"}\n",
    "....\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9490741c",
   "metadata": {},
   "source": [
    "## Create Inference Transformer to run the batch job\n",
    "\n",
    "We use the [twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment) model running our batch transform job. This is a RoBERTa-base model trained on ~58M tweets and finetuned for sentiment analysis with the TweetEval benchmark.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6278b087",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.huggingface.model import HuggingFaceModel\n",
    "\n",
    "# Hub Model configuration. <https://huggingface.co/models>\n",
    "hub = {\n",
    "    'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment',\n",
    "    'HF_TASK':'text-classification'\n",
    "}\n",
    "\n",
    "# create Hugging Face Model Class\n",
    "huggingface_model = HuggingFaceModel(\n",
    "   env=hub, # configuration for loading model from Hub\n",
    "   role=role, # iam role with permissions to create an Endpoint\n",
    "   transformers_version=\"4.26\", # transformers version used\n",
    "   pytorch_version=\"1.13\", # pytorch version used\n",
    "   py_version='py39', # python version used\n",
    ")\n",
    "\n",
    "# create Transformer to run our batch job\n",
    "batch_job = huggingface_model.transformer(\n",
    "    instance_count=1,\n",
    "    instance_type='ml.p3.2xlarge',\n",
    "    output_path=output_s3_path, # we are using the same s3 path to save the output with the input\n",
    "    strategy='SingleRecord')\n",
    "\n",
    "# starts batch transform job and uses s3 data as input\n",
    "batch_job.transform(\n",
    "    data=s3_file_uri,\n",
    "    content_type='application/json',    \n",
    "    split_type='Line')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c3442915",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'label': 'LABEL_1', 'score': 0.766870379447937}, {'label': 'LABEL_0', 'score': 0.8912611603736877}, {'label': 'LABEL_1', 'score': 0.5760677456855774}]\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from sagemaker.s3 import S3Downloader\n",
    "from ast import literal_eval\n",
    "# creating s3 uri for result file -> input file + .out\n",
    "output_file = f\"{dataset_jsonl_file}.out\"\n",
    "output_path = s3_path_join(output_s3_path,output_file)\n",
    "\n",
    "# download file\n",
    "S3Downloader.download(output_path,'.')\n",
    "\n",
    "batch_transform_result = []\n",
    "with open(output_file) as f:\n",
    "    for line in f:\n",
    "        # converts jsonline array to normal array\n",
    "        line = \"[\" + line.replace(\"[\",\"\").replace(\"]\",\",\") + \"]\"\n",
    "        batch_transform_result = literal_eval(line) \n",
    "        \n",
    "# print results \n",
    "print(batch_transform_result[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7164cd20-1518-403f-b7fa-60314ac5f316",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c281c456f1b8161c8906f4af2c08ed2c40c50136979eaae69688b01f70e9f4a9"
  },
  "kernelspec": {
   "display_name": "conda_pytorch_p39",
   "language": "python",
   "name": "conda_pytorch_p39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}