   ml_ops/sm-pipelines_lambda_step/sm-pipelines_lambda_step.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0425e1c2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:52.158481Z",
     "iopub.status.busy": "2022-07-13T15:58:52.157838Z",
     "iopub.status.idle": "2022-07-13T15:58:52.159725Z",
     "shell.execute_reply": "2022-07-13T15:58:52.160093Z"
    },
    "papermill": {
     "duration": 0.021573,
     "end_time": "2022-07-13T15:58:52.160240",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.138667",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": [
     "injected-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "kms_key = \"arn:aws:kms:us-west-2:000000000000:1234abcd-12ab-34cd-56ef-1234567890ab\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66b60a2b",
   "metadata": {
    "papermill": {
     "duration": 0.015296,
     "end_time": "2022-07-13T15:58:52.191291",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.175995",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "# SageMaker Pipelines Lambda Step\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66b60a2b",
   "metadata": {
    "papermill": {
     "duration": 0.015296,
     "end_time": "2022-07-13T15:58:52.191291",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.175995",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "\n",
    "This notebook illustrates how a Lambda function can be run as a step in a SageMaker Pipeline.\n",
    "\n",
    "The steps in this pipeline include:\n",
    "* Preprocess the Abalone dataset\n",
    "* Train an XGBoost Model\n",
    "* Evaluate the model performance\n",
    "* Create a model\n",
    "* Deploy the model to a SageMaker Hosted Endpoint using a Lambda Function, through SageMaker Pipelines\n",
    "\n",
    "A step to register the model into a Model Registry can be added to the pipeline using the `RegisterModel` step."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c741e28",
   "metadata": {
    "papermill": {
     "duration": 0.015223,
     "end_time": "2022-07-13T15:58:52.221806",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.206583",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Runtime\n",
    "\n",
    "This notebook takes approximately 15 minutes to run.\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Prerequisites](#Prerequisites)\n",
    "1. [Configuration Setup](#Configuration-Setup)\n",
    "1. [Data Preparation](#Data-Preparation)\n",
    "1. [Model Training and Evaluation](#Model-Training-and-Evaluation)\n",
    "1. [Setting up Lambda](#Setting-up-Lambda)\n",
    "1. [Execute the Pipeline](#Execute-the-Pipeline)\n",
    "1. [Clean up resources](#Clean-up-resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5a68c81",
   "metadata": {
    "papermill": {
     "duration": 0.015125,
     "end_time": "2022-07-13T15:58:52.252140",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.237015",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Prerequisites\n",
    "\n",
    "The notebook execution role should have policies which enable the notebook to create a Lambda function. The Amazon managed policy `AmazonSageMakerPipelinesIntegrations` can be added to the notebook execution role to achieve the same effect.\n",
    "\n",
    "The policy description is as follows:\n",
    "\n",
    "```\n",
    "\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"lambda:CreateFunction\",\n",
    "                \"lambda:DeleteFunction\",\n",
    "                \"lambda:InvokeFunction\",\n",
    "                \"lambda:UpdateFunctionCode\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:lambda:*:*:function:*sagemaker*\",\n",
    "                \"arn:aws:lambda:*:*:function:*sageMaker*\",\n",
    "                \"arn:aws:lambda:*:*:function:*SageMaker*\"\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"sqs:CreateQueue\",\n",
    "                \"sqs:SendMessage\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:sqs:*:*:*sagemaker*\",\n",
    "                \"arn:aws:sqs:*:*:*sageMaker*\",\n",
    "                \"arn:aws:sqs:*:*:*SageMaker*\"\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"iam:PassRole\"\n",
    "            ],\n",
    "            \"Resource\": \"arn:aws:iam::*:role/*\",\n",
    "            \"Condition\": {\n",
    "                \"StringEquals\": {\n",
    "                    \"iam:PassedToService\": [\n",
    "                        \"lambda.amazonaws.com\"\n",
    "                    ]\n",
    "                }\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09611418",
   "metadata": {
    "papermill": {
     "duration": 0.015163,
     "end_time": "2022-07-13T15:58:52.282544",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.267381",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "Let's start by importing necessary packages and installing the SageMaker Python SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b7a871ec",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:52.322697Z",
     "iopub.status.busy": "2022-07-13T15:58:52.321906Z",
     "iopub.status.idle": "2022-07-13T15:58:52.827711Z",
     "shell.execute_reply": "2022-07-13T15:58:52.827193Z"
    },
    "papermill": {
     "duration": 0.52726,
     "end_time": "2022-07-13T15:58:52.827823",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.300563",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "from sagemaker.estimator import Estimator\n",
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "from sagemaker.processing import (\n",
    "    ProcessingInput,\n",
    "    ProcessingOutput,\n",
    "    Processor,\n",
    "    ScriptProcessor,\n",
    ")\n",
    "\n",
    "from sagemaker import Model\n",
    "from sagemaker.xgboost import XGBoostPredictor\n",
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "from sagemaker.workflow.parameters import (\n",
    "    ParameterInteger,\n",
    "    ParameterString,\n",
    ")\n",
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.properties import PropertyFile\n",
    "from sagemaker.workflow.steps import ProcessingStep, TrainingStep, CacheConfig\n",
    "from sagemaker.workflow.lambda_step import (\n",
    "    LambdaStep,\n",
    "    LambdaOutput,\n",
    "    LambdaOutputTypeEnum,\n",
    ")\n",
    "from sagemaker.workflow.model_step import ModelStep\n",
    "from sagemaker.workflow.conditions import ConditionLessThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import ConditionStep\n",
    "from sagemaker.workflow.functions import JsonGet\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession\n",
    "\n",
    "from sagemaker.lambda_helper import Lambda\n",
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "481497d6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:52.862463Z",
     "iopub.status.busy": "2022-07-13T15:58:52.861935Z",
     "iopub.status.idle": "2022-07-13T15:58:54.581086Z",
     "shell.execute_reply": "2022-07-13T15:58:54.581465Z"
    },
    "papermill": {
     "duration": 1.738298,
     "end_time": "2022-07-13T15:58:54.581614",
     "exception": false,
     "start_time": "2022-07-13T15:58:52.843316",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead\r\n",
      "  from cryptography.utils import int_from_bytes\r\n",
      "/opt/conda/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead\r\n",
      "  from cryptography.utils import int_from_bytes\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker>=2.99.0 in /opt/conda/lib/python3.7/site-packages (2.99.0)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (0.1.5)\r\n",
      "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (0.2.0)\r\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (1.0.1)\r\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (1.21.1)\r\n",
      "Requirement already satisfied: boto3<2.0,>=1.20.21 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (1.20.47)\r\n",
      "Requirement already satisfied: attrs<22,>=20.3.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (21.4.0)\r\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (1.0.1)\r\n",
      "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (1.5.0)\r\n",
      "Requirement already satisfied: protobuf<4.0,>=3.1 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (3.17.3)\r\n",
      "Requirement already satisfied: pathos in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (0.2.8)\r\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker>=2.99.0) (20.1)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: botocore<1.24.0,>=1.23.47 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.20.21->sagemaker>=2.99.0) (1.23.47)\r\n",
      "Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.20.21->sagemaker>=2.99.0) (0.5.0)\r\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.20.21->sagemaker>=2.99.0) (0.10.0)\r\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.7/site-packages (from botocore<1.24.0,>=1.23.47->boto3<2.0,>=1.20.21->sagemaker>=2.99.0) (1.26.6)\r\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/lib/python3.7/site-packages (from botocore<1.24.0,>=1.23.47->boto3<2.0,>=1.20.21->sagemaker>=2.99.0) (2.8.1)\r\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker>=2.99.0) (2.2.0)\r\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->sagemaker>=2.99.0) (1.14.0)\r\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->sagemaker>=2.99.0) (2.4.6)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.7/site-packages (from pandas->sagemaker>=2.99.0) (2019.3)\r\n",
      "Requirement already satisfied: ppft>=1.6.6.4 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker>=2.99.0) (1.6.6.4)\r\n",
      "Requirement already satisfied: pox>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker>=2.99.0) (0.3.0)\r\n",
      "Requirement already satisfied: multiprocess>=0.70.12 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker>=2.99.0) (0.70.12.2)\r\n",
      "Requirement already satisfied: dill>=0.3.4 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker>=2.99.0) (0.3.4)\r\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\r\n",
      "\u001b[33mWARNING: You are using pip version 21.1.3; however, version 22.1.2 is available.\r\n",
      "You should consider upgrading via the '/opt/conda/bin/python -m pip install --upgrade pip' command.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!{sys.executable} -m pip install \"sagemaker>=2.99.0\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda7b3e5",
   "metadata": {
    "papermill": {
     "duration": 0.017215,
     "end_time": "2022-07-13T15:58:54.616067",
     "exception": false,
     "start_time": "2022-07-13T15:58:54.598852",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Configuration Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ce0242d",
   "metadata": {
    "papermill": {
     "duration": 0.016806,
     "end_time": "2022-07-13T15:58:54.649817",
     "exception": false,
     "start_time": "2022-07-13T15:58:54.633011",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "Let's now configure the setup we need, which includes the session object from the SageMaker Python SDK, and neccessary configurations for the pipelines, such as object types, input and output buckets and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a55e1178",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:54.696396Z",
     "iopub.status.busy": "2022-07-13T15:58:54.695661Z",
     "iopub.status.idle": "2022-07-13T15:58:54.900760Z",
     "shell.execute_reply": "2022-07-13T15:58:54.900289Z"
    },
    "papermill": {
     "duration": 0.234346,
     "end_time": "2022-07-13T15:58:54.900874",
     "exception": false,
     "start_time": "2022-07-13T15:58:54.666528",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create the SageMaker Session\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "pipeline_session = PipelineSession()\n",
    "sm_client = sagemaker_session.sagemaker_client\n",
    "region = sagemaker_session.boto_region_name\n",
    "prefix = \"lambda-step-pipeline\"\n",
    "\n",
    "account_id = sagemaker_session.account_id()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1832cf2e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:54.947742Z",
     "iopub.status.busy": "2022-07-13T15:58:54.947270Z",
     "iopub.status.idle": "2022-07-13T15:58:55.684395Z",
     "shell.execute_reply": "2022-07-13T15:58:55.683908Z"
    },
    "papermill": {
     "duration": 0.766426,
     "end_time": "2022-07-13T15:58:55.684513",
     "exception": false,
     "start_time": "2022-07-13T15:58:54.918087",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define variables and parameters needed for the Pipeline steps\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "default_bucket = sagemaker_session.default_bucket()\n",
    "base_job_prefix = \"lambda-step-example\"\n",
    "s3_prefix = \"lambda-step-pipeline\"\n",
    "default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    s3_prefix = f\"{default_bucket_prefix}/{s3_prefix}\"\n",
    "    base_job_prefix = f\"{default_bucket_prefix}/{base_job_prefix}\"\n",
    "\n",
    "processing_instance_count = ParameterInteger(name=\"ProcessingInstanceCount\", default_value=1)\n",
    "training_instance_type = ParameterString(name=\"TrainingInstanceType\", default_value=\"ml.m5.xlarge\")\n",
    "model_approval_status = ParameterString(\n",
    "    name=\"ModelApprovalStatus\", default_value=\"PendingManualApproval\"\n",
    ")\n",
    "input_data = ParameterString(\n",
    "    name=\"InputDataUrl\",\n",
    "    default_value=f\"s3://sagemaker-sample-files/datasets/tabular/uci_abalone/abalone.csv\",\n",
    ")\n",
    "model_approval_status = ParameterString(\n",
    "    name=\"ModelApprovalStatus\", default_value=\"PendingManualApproval\"\n",
    ")\n",
    "\n",
    "# Cache Pipeline steps to reduce execution time on subsequent executions\n",
    "cache_config = CacheConfig(enable_caching=True, expire_after=\"30d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba7cd3ba",
   "metadata": {
    "papermill": {
     "duration": 0.016885,
     "end_time": "2022-07-13T15:58:55.718428",
     "exception": false,
     "start_time": "2022-07-13T15:58:55.701543",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Data Preparation\n",
    "\n",
    "An SKLearn processor is used to prepare the dataset for the Hyperparameter Tuning job. Using the script `preprocess.py`, the dataset is featurized and split into train, test, and validation datasets.\n",
    "\n",
    "The output of this step is used as the input to the TrainingStep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6ef53ec4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:55.759267Z",
     "iopub.status.busy": "2022-07-13T15:58:55.755957Z",
     "iopub.status.idle": "2022-07-13T15:58:55.907503Z",
     "shell.execute_reply": "2022-07-13T15:58:55.907053Z"
    },
    "papermill": {
     "duration": 0.172373,
     "end_time": "2022-07-13T15:58:55.907627",
     "exception": false,
     "start_time": "2022-07-13T15:58:55.735254",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "!mkdir -p code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ea68f51c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:55.948608Z",
     "iopub.status.busy": "2022-07-13T15:58:55.947461Z",
     "iopub.status.idle": "2022-07-13T15:58:55.951620Z",
     "shell.execute_reply": "2022-07-13T15:58:55.951193Z"
    },
    "papermill": {
     "duration": 0.026484,
     "end_time": "2022-07-13T15:58:55.951738",
     "exception": false,
     "start_time": "2022-07-13T15:58:55.925254",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing code/preprocess.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile code/preprocess.py\n",
    "\n",
    "\"\"\"Feature engineers the abalone dataset.\"\"\"\n",
    "import argparse\n",
    "import logging\n",
    "import os\n",
    "import pathlib\n",
    "import requests\n",
    "import tempfile\n",
    "\n",
    "import boto3\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.INFO)\n",
    "logger.addHandler(logging.StreamHandler())\n",
    "\n",
    "\n",
    "# Since we get a headerless CSV file we specify the column names here.\n",
    "feature_columns_names = [\n",
    "    \"sex\",\n",
    "    \"length\",\n",
    "    \"diameter\",\n",
    "    \"height\",\n",
    "    \"whole_weight\",\n",
    "    \"shucked_weight\",\n",
    "    \"viscera_weight\",\n",
    "    \"shell_weight\",\n",
    "]\n",
    "label_column = \"rings\"\n",
    "\n",
    "feature_columns_dtype = {\n",
    "    \"sex\": str,\n",
    "    \"length\": np.float64,\n",
    "    \"diameter\": np.float64,\n",
    "    \"height\": np.float64,\n",
    "    \"whole_weight\": np.float64,\n",
    "    \"shucked_weight\": np.float64,\n",
    "    \"viscera_weight\": np.float64,\n",
    "    \"shell_weight\": np.float64,\n",
    "}\n",
    "label_column_dtype = {\"rings\": np.float64}\n",
    "\n",
    "\n",
    "def merge_two_dicts(x, y):\n",
    "    \"\"\"Merges two dicts, returning a new copy.\"\"\"\n",
    "    z = x.copy()\n",
    "    z.update(y)\n",
    "    return z\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    logger.debug(\"Starting preprocessing.\")\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--input-data\", type=str, required=True)\n",
    "    args = parser.parse_args()\n",
    "\n",
    "    base_dir = \"/opt/ml/processing\"\n",
    "    pathlib.Path(f\"{base_dir}/data\").mkdir(parents=True, exist_ok=True)\n",
    "    input_data = args.input_data\n",
    "    bucket = input_data.split(\"/\")[2]\n",
    "    key = \"/\".join(input_data.split(\"/\")[3:])\n",
    "\n",
    "    logger.info(\"Downloading data from bucket: %s, key: %s\", bucket, key)\n",
    "    fn = f\"{base_dir}/data/abalone-dataset.csv\"\n",
    "    s3 = boto3.resource(\"s3\")\n",
    "    s3.Bucket(bucket).download_file(key, fn)\n",
    "\n",
    "    logger.debug(\"Reading downloaded data.\")\n",
    "    df = pd.read_csv(\n",
    "        fn,\n",
    "        header=None,\n",
    "        names=feature_columns_names + [label_column],\n",
    "        dtype=merge_two_dicts(feature_columns_dtype, label_column_dtype),\n",
    "    )\n",
    "    os.unlink(fn)\n",
    "\n",
    "    logger.debug(\"Defining transformers.\")\n",
    "    numeric_features = list(feature_columns_names)\n",
    "    numeric_features.remove(\"sex\")\n",
    "    numeric_transformer = Pipeline(\n",
    "        steps=[\n",
    "            (\"imputer\", SimpleImputer(strategy=\"median\")),\n",
    "            (\"scaler\", StandardScaler()),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    categorical_features = [\"sex\"]\n",
    "    categorical_transformer = Pipeline(\n",
    "        steps=[\n",
    "            (\"imputer\", SimpleImputer(strategy=\"constant\", fill_value=\"missing\")),\n",
    "            (\"onehot\", OneHotEncoder(handle_unknown=\"ignore\")),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    preprocess = ColumnTransformer(\n",
    "        transformers=[\n",
    "            (\"num\", numeric_transformer, numeric_features),\n",
    "            (\"cat\", categorical_transformer, categorical_features),\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    logger.info(\"Applying transforms.\")\n",
    "    y = df.pop(\"rings\")\n",
    "    X_pre = preprocess.fit_transform(df)\n",
    "    y_pre = y.to_numpy().reshape(len(y), 1)\n",
    "\n",
    "    X = np.concatenate((y_pre, X_pre), axis=1)\n",
    "\n",
    "    logger.info(\"Splitting %d rows of data into train, validation, test datasets.\", len(X))\n",
    "    np.random.shuffle(X)\n",
    "    train, validation, test = np.split(X, [int(0.7 * len(X)), int(0.85 * len(X))])\n",
    "\n",
    "    logger.info(\"Writing out datasets to %s.\", base_dir)\n",
    "    pd.DataFrame(train).to_csv(f\"{base_dir}/train/train.csv\", header=False, index=False)\n",
    "    pd.DataFrame(validation).to_csv(\n",
    "        f\"{base_dir}/validation/validation.csv\", header=False, index=False\n",
    "    )\n",
    "    pd.DataFrame(test).to_csv(f\"{base_dir}/test/test.csv\", header=False, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5cc7e807",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:55.993872Z",
     "iopub.status.busy": "2022-07-13T15:58:55.992990Z",
     "iopub.status.idle": "2022-07-13T15:58:56.276627Z",
     "shell.execute_reply": "2022-07-13T15:58:56.277057Z"
    },
    "papermill": {
     "duration": 0.307809,
     "end_time": "2022-07-13T15:58:56.277208",
     "exception": false,
     "start_time": "2022-07-13T15:58:55.969399",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sagemaker/workflow/pipeline_context.py:197: UserWarning: Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "  UserWarning,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Job Name:  lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005\n",
      "Inputs:  [{'InputName': 'code', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/input/code/preprocess.py', 'LocalPath': '/opt/ml/processing/input/code', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n",
      "Outputs:  [{'OutputName': 'train', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/output/train', 'LocalPath': '/opt/ml/processing/train', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'validation', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/output/validation', 'LocalPath': '/opt/ml/processing/validation', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'test', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/output/test', 'LocalPath': '/opt/ml/processing/test', 'S3UploadMode': 'EndOfJob'}}]\n"
     ]
    }
   ],
   "source": [
    "# Process the training data step using a python script.\n",
    "# Split the training data set into train, test, and validation datasets\n",
    "\n",
    "sklearn_processor = SKLearnProcessor(\n",
    "    framework_version=\"1.2-1\",\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=processing_instance_count,\n",
    "    base_job_name=f\"{base_job_prefix}/sklearn-abalone-preprocess\",\n",
    "    sagemaker_session=pipeline_session,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "processor_args = sklearn_processor.run(\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train\", source=\"/opt/ml/processing/train\"),\n",
    "        ProcessingOutput(output_name=\"validation\", source=\"/opt/ml/processing/validation\"),\n",
    "        ProcessingOutput(output_name=\"test\", source=\"/opt/ml/processing/test\"),\n",
    "    ],\n",
    "    code=\"code/preprocess.py\",\n",
    "    arguments=[\"--input-data\", input_data],\n",
    ")\n",
    "\n",
    "step_process = ProcessingStep(\n",
    "    name=\"PreprocessAbaloneData\",\n",
    "    step_args=processor_args,\n",
    "    cache_config=cache_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff61dc5a",
   "metadata": {
    "papermill": {
     "duration": 0.017909,
     "end_time": "2022-07-13T15:58:56.313344",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.295435",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Model Training and Evaluation\n",
    "\n",
    "We will now train an XGBoost model using the SageMaker Python SDK and the output of the ProcessingStep."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda6892e",
   "metadata": {
    "papermill": {
     "duration": 0.01789,
     "end_time": "2022-07-13T15:58:56.350320",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.332430",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "#### Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a575d6e3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:56.394662Z",
     "iopub.status.busy": "2022-07-13T15:58:56.394093Z",
     "iopub.status.idle": "2022-07-13T15:58:56.421180Z",
     "shell.execute_reply": "2022-07-13T15:58:56.421606Z"
    },
    "papermill": {
     "duration": 0.053298,
     "end_time": "2022-07-13T15:58:56.421753",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.368455",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sagemaker/workflow/steps.py:391: UserWarning: Profiling is enabled on the provided estimator. The default profiler rule includes a timestamp which will change each time the pipeline is upserted, causing cache misses. If profiling is not needed, set disable_profiler to True on the estimator.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "# Define the output path for the model artifacts from the Hyperparameter Tuning Job\n",
    "model_path = f\"s3://{default_bucket}/{base_job_prefix}/AbaloneTrain\"\n",
    "\n",
    "image_uri = sagemaker.image_uris.retrieve(\n",
    "    framework=\"xgboost\",\n",
    "    region=region,\n",
    "    version=\"1.0-1\",\n",
    "    py_version=\"py3\",\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    ")\n",
    "\n",
    "xgb_train = Estimator(\n",
    "    image_uri=image_uri,\n",
    "    instance_type=training_instance_type,\n",
    "    instance_count=1,\n",
    "    output_path=model_path,\n",
    "    base_job_name=f\"{prefix}/{base_job_prefix}/sklearn-abalone-preprocess\",\n",
    "    sagemaker_session=pipeline_session,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "xgb_train.set_hyperparameters(\n",
    "    objective=\"reg:linear\",\n",
    "    num_round=50,\n",
    "    max_depth=5,\n",
    "    eta=0.2,\n",
    "    gamma=4,\n",
    "    min_child_weight=6,\n",
    "    subsample=0.7,\n",
    "    silent=0,\n",
    ")\n",
    "\n",
    "train_args = xgb_train.fit(\n",
    "    inputs={\n",
    "        \"train\": TrainingInput(\n",
    "            s3_data=step_process.properties.ProcessingOutputConfig.Outputs[\"train\"].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\",\n",
    "        ),\n",
    "        \"validation\": TrainingInput(\n",
    "            s3_data=step_process.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"validation\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\",\n",
    "        ),\n",
    "    },\n",
    ")\n",
    "\n",
    "step_train = TrainingStep(\n",
    "    name=\"TrainAbaloneModel\",\n",
    "    step_args=train_args,\n",
    "    cache_config=cache_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c40566b",
   "metadata": {
    "papermill": {
     "duration": 0.018193,
     "end_time": "2022-07-13T15:58:56.458560",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.440367",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "#### Evaluating the model\n",
    "\n",
    "Use a processing job to evaluate the model from the TrainingStep. If the output of the evaluation is True, a model is created and a Lambda function is invoked to deploy the model to a SageMaker Endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1545156e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:56.499375Z",
     "iopub.status.busy": "2022-07-13T15:58:56.498823Z",
     "iopub.status.idle": "2022-07-13T15:58:56.501906Z",
     "shell.execute_reply": "2022-07-13T15:58:56.501502Z"
    },
    "papermill": {
     "duration": 0.02515,
     "end_time": "2022-07-13T15:58:56.502014",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.476864",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing code/evaluate.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile code/evaluate.py\n",
    "\n",
    "\"\"\"Evaluation script for measuring mean squared error.\"\"\"\n",
    "import json\n",
    "import logging\n",
    "import pathlib\n",
    "import pickle\n",
    "import tarfile\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import xgboost\n",
    "\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.INFO)\n",
    "logger.addHandler(logging.StreamHandler())\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    logger.debug(\"Starting evaluation.\")\n",
    "    model_path = \"/opt/ml/processing/model/model.tar.gz\"\n",
    "    with tarfile.open(model_path) as tar:\n",
    "        tar.extractall(path=\".\")\n",
    "\n",
    "    logger.debug(\"Loading xgboost model.\")\n",
    "    model = pickle.load(open(\"xgboost-model\", \"rb\"))\n",
    "\n",
    "    logger.debug(\"Reading test data.\")\n",
    "    test_path = \"/opt/ml/processing/test/test.csv\"\n",
    "    df = pd.read_csv(test_path, header=None)\n",
    "\n",
    "    logger.debug(\"Reading test data.\")\n",
    "    y_test = df.iloc[:, 0].to_numpy()\n",
    "    df.drop(df.columns[0], axis=1, inplace=True)\n",
    "    X_test = xgboost.DMatrix(df.values)\n",
    "\n",
    "    logger.info(\"Performing predictions against test data.\")\n",
    "    predictions = model.predict(X_test)\n",
    "\n",
    "    logger.debug(\"Calculating mean squared error.\")\n",
    "    mse = mean_squared_error(y_test, predictions)\n",
    "    std = np.std(y_test - predictions)\n",
    "    report_dict = {\n",
    "        \"regression_metrics\": {\n",
    "            \"mse\": {\"value\": mse, \"standard_deviation\": std},\n",
    "        },\n",
    "    }\n",
    "\n",
    "    output_dir = \"/opt/ml/processing/evaluation\"\n",
    "    pathlib.Path(output_dir).mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "    logger.info(\"Writing out evaluation report with mse: %f\", mse)\n",
    "    evaluation_path = f\"{output_dir}/evaluation.json\"\n",
    "    with open(evaluation_path, \"w\") as f:\n",
    "        f.write(json.dumps(report_dict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "edd37d25",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:56.549199Z",
     "iopub.status.busy": "2022-07-13T15:58:56.548644Z",
     "iopub.status.idle": "2022-07-13T15:58:56.640043Z",
     "shell.execute_reply": "2022-07-13T15:58:56.640488Z"
    },
    "papermill": {
     "duration": 0.119643,
     "end_time": "2022-07-13T15:58:56.640642",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.520999",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Job Name:  lambda-step-pipeline/lambda-step-exampl-2022-07-13-15-58-56-545\n",
      "Inputs:  [{'InputName': 'input-1', 'AppManaged': False, 'S3Input': {'S3Uri': <sagemaker.workflow.properties.Properties object at 0x7f15b0ee9350>, 'LocalPath': '/opt/ml/processing/model', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'input-2', 'AppManaged': False, 'S3Input': {'S3Uri': <sagemaker.workflow.properties.Properties object at 0x7f15b0f199d0>, 'LocalPath': '/opt/ml/processing/test', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'code', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-pipeline/lambda-step-exampl-2022-07-13-15-58-56-545/input/code/evaluate.py', 'LocalPath': '/opt/ml/processing/input/code', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n",
      "Outputs:  [{'OutputName': 'evaluation', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-pipeline/evaluation_report', 'LocalPath': '/opt/ml/processing/evaluation', 'S3UploadMode': 'EndOfJob'}}]\n"
     ]
    }
   ],
   "source": [
    "# A ProcessingStep is used to evaluate the performance of the trained model.\n",
    "# Based on the results of the evaluation, the model is created and deployed.\n",
    "\n",
    "script_eval = ScriptProcessor(\n",
    "    image_uri=image_uri,\n",
    "    command=[\"python3\"],\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    instance_count=1,\n",
    "    base_job_name=f\"{prefix}/{base_job_prefix}/sklearn-abalone-preprocess\",\n",
    "    sagemaker_session=pipeline_session,\n",
    "    role=role,\n",
    ")\n",
    "\n",
    "evaluation_report = PropertyFile(\n",
    "    name=\"AbaloneEvaluationReport\",\n",
    "    output_name=\"evaluation\",\n",
    "    path=\"evaluation.json\",\n",
    ")\n",
    "\n",
    "eval_args = script_eval.run(\n",
    "    inputs=[\n",
    "        ProcessingInput(\n",
    "            source=step_train.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "            destination=\"/opt/ml/processing/model\",\n",
    "        ),\n",
    "        ProcessingInput(\n",
    "            source=step_process.properties.ProcessingOutputConfig.Outputs[\"test\"].S3Output.S3Uri,\n",
    "            destination=\"/opt/ml/processing/test\",\n",
    "        ),\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(\n",
    "            output_name=\"evaluation\",\n",
    "            source=\"/opt/ml/processing/evaluation\",\n",
    "            destination=f\"s3://{default_bucket}/{s3_prefix}/evaluation_report\",\n",
    "        ),\n",
    "    ],\n",
    "    code=\"code/evaluate.py\",\n",
    ")\n",
    "step_eval = ProcessingStep(\n",
    "    name=\"EvaluateAbaloneModel\",\n",
    "    step_args=eval_args,\n",
    "    property_files=[evaluation_report],\n",
    "    cache_config=cache_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce969364",
   "metadata": {
    "papermill": {
     "duration": 0.018851,
     "end_time": "2022-07-13T15:58:56.678471",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.659620",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "#### Creating the final model object\n",
    "\n",
    "The model is created and the name of the model is provided to the Lambda function for deployment. The `CreateModelStep` dynamically assigns a name to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8017b70a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:56.721922Z",
     "iopub.status.busy": "2022-07-13T15:58:56.721196Z",
     "iopub.status.idle": "2022-07-13T15:58:56.723177Z",
     "shell.execute_reply": "2022-07-13T15:58:56.723545Z"
    },
    "papermill": {
     "duration": 0.026308,
     "end_time": "2022-07-13T15:58:56.723689",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.697381",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create Model\n",
    "model = Model(\n",
    "    image_uri=image_uri,\n",
    "    model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    sagemaker_session=pipeline_session,\n",
    "    role=role,\n",
    "    predictor_cls=XGBoostPredictor,\n",
    ")\n",
    "\n",
    "step_create_model = ModelStep(\n",
    "    name=\"CreateModel\",\n",
    "    step_args=model.create(\"ml.m4.large\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8d94f5b",
   "metadata": {
    "papermill": {
     "duration": 0.018913,
     "end_time": "2022-07-13T15:58:56.761761",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.742848",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Setting up Lambda\n",
    "\n",
    "When defining the LambdaStep, the SageMaker Lambda helper class provides helper functions for creating the Lambda function. Users can either use the `lambda_func` argument to provide the function ARN to an already deployed Lambda function OR use the `Lambda` class to create a Lambda function by providing a script, function name and role for the Lambda function.\n",
    "\n",
    "When passing inputs to the Lambda, the `inputs` argument can be used and within the Lambda function's handler, the `event` argument can be used to retrieve the inputs.\n",
    "\n",
    "The dictionary response from the Lambda function is parsed through the `LambdaOutput` objects provided to the `outputs` argument. The `output_name` in `LambdaOutput` corresponds to the dictionary key in the Lambda's return dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f813825f",
   "metadata": {
    "papermill": {
     "duration": 0.018947,
     "end_time": "2022-07-13T15:58:56.799670",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.780723",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "#### Define the Lambda function\n",
    "\n",
    "Users can choose the leverage the Lambda helper class to create a Lambda function and provide that function object to the LambdaStep. Alternatively, users can use a pre-deployed Lambda function and provide the function ARN to the `Lambda` helper class in the Lambda step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c3782959",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:56.842251Z",
     "iopub.status.busy": "2022-07-13T15:58:56.841631Z",
     "iopub.status.idle": "2022-07-13T15:58:56.844526Z",
     "shell.execute_reply": "2022-07-13T15:58:56.844085Z"
    },
    "papermill": {
     "duration": 0.026015,
     "end_time": "2022-07-13T15:58:56.844646",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.818631",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing code/lambda_helper.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile code/lambda_helper.py\n",
    "\n",
    "\"\"\"\n",
    "This Lambda function creates an Endpoint Configuration and deploys a model to an Endpoint.\n",
    "The name of the model to deploy is provided via the `event` argument\n",
    "\"\"\"\n",
    "\n",
    "import json\n",
    "import boto3\n",
    "\n",
    "\n",
    "def lambda_handler(event, context):\n",
    "    \"\"\" \"\"\"\n",
    "    sm_client = boto3.client(\"sagemaker\")\n",
    "\n",
    "    # The name of the model created in the Pipeline CreateModelStep\n",
    "    model_name = event[\"model_name\"]\n",
    "\n",
    "    endpoint_config_name = event[\"endpoint_config_name\"]\n",
    "    endpoint_name = event[\"endpoint_name\"]\n",
    "\n",
    "    create_endpoint_config_response = sm_client.create_endpoint_config(\n",
    "        EndpointConfigName=endpoint_config_name,\n",
    "        ProductionVariants=[\n",
    "            {\n",
    "                \"InstanceType\": \"ml.m4.xlarge\",\n",
    "                \"InitialVariantWeight\": 1,\n",
    "                \"InitialInstanceCount\": 1,\n",
    "                \"ModelName\": model_name,\n",
    "                \"VariantName\": \"AllTraffic\",\n",
    "            }\n",
    "        ],\n",
    "    )\n",
    "\n",
    "    create_endpoint_response = sm_client.create_endpoint(\n",
    "        EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name\n",
    "    )\n",
    "\n",
    "    return {\n",
    "        \"statusCode\": 200,\n",
    "        \"body\": json.dumps(\"Created Endpoint!\"),\n",
    "        \"other_key\": \"example_value\",\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7835966e",
   "metadata": {
    "papermill": {
     "duration": 0.019367,
     "end_time": "2022-07-13T15:58:56.883384",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.864017",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "#### Setting up the custom IAM Role\n",
    "\n",
    "The Lambda function needs an IAM role that allows it to deploy a SageMaker Endpoint. The role ARN must be provided in the LambdaStep.\n",
    "\n",
    "The Lambda role should at minimum have policies to allow `sagemaker:CreateModel`, `sagemaker:CreateEndpointConfig`, `sagemaker:CreateEndpoint` in addition to the based Lambda execution policies.\n",
    "\n",
    "A helper function in `iam_helper.py` is available to create the Lambda function role. Please note that the role uses the Amazon managed policy - `SageMakerFullAccess`. This should be replaced with an IAM policy with least privileges as per AWS IAM best practices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "928270f8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:56.926249Z",
     "iopub.status.busy": "2022-07-13T15:58:56.925421Z",
     "iopub.status.idle": "2022-07-13T15:58:57.385000Z",
     "shell.execute_reply": "2022-07-13T15:58:57.384557Z"
    },
    "papermill": {
     "duration": 0.482377,
     "end_time": "2022-07-13T15:58:57.385112",
     "exception": false,
     "start_time": "2022-07-13T15:58:56.902735",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using ARN from existing role: lambda-deployment-role\n"
     ]
    }
   ],
   "source": [
    "from iam_helper import create_lambda_role\n",
    "\n",
    "lambda_role = create_lambda_role(\"lambda-deployment-role\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "430e04f4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:57.431162Z",
     "iopub.status.busy": "2022-07-13T15:58:57.430310Z",
     "iopub.status.idle": "2022-07-13T15:58:57.540082Z",
     "shell.execute_reply": "2022-07-13T15:58:57.540527Z"
    },
    "papermill": {
     "duration": 0.135811,
     "end_time": "2022-07-13T15:58:57.540673",
     "exception": false,
     "start_time": "2022-07-13T15:58:57.404862",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Custom Lambda Step\n",
    "\n",
    "current_time = time.strftime(\"%m-%d-%H-%M-%S\", time.localtime())\n",
    "model_name = \"demo-lambda-model\" + current_time\n",
    "endpoint_config_name = \"demo-lambda-deploy-endpoint-config-\" + current_time\n",
    "endpoint_name = \"demo-lambda-deploy-endpoint-\" + current_time\n",
    "\n",
    "function_name = \"sagemaker-lambda-step-endpoint-deploy-\" + current_time\n",
    "\n",
    "# Lambda helper class can be used to create the Lambda function\n",
    "func = Lambda(\n",
    "    function_name=function_name,\n",
    "    execution_role_arn=lambda_role,\n",
    "    script=\"code/lambda_helper.py\",\n",
    "    handler=\"lambda_helper.lambda_handler\",\n",
    ")\n",
    "\n",
    "output_param_1 = LambdaOutput(output_name=\"statusCode\", output_type=LambdaOutputTypeEnum.String)\n",
    "output_param_2 = LambdaOutput(output_name=\"body\", output_type=LambdaOutputTypeEnum.String)\n",
    "output_param_3 = LambdaOutput(output_name=\"other_key\", output_type=LambdaOutputTypeEnum.String)\n",
    "\n",
    "step_deploy_lambda = LambdaStep(\n",
    "    name=\"LambdaStep\",\n",
    "    lambda_func=func,\n",
    "    inputs={\n",
    "        \"model_name\": step_create_model.properties.ModelName,\n",
    "        \"endpoint_config_name\": endpoint_config_name,\n",
    "        \"endpoint_name\": endpoint_name,\n",
    "    },\n",
    "    outputs=[output_param_1, output_param_2, output_param_3],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5c6c4475",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:57.584145Z",
     "iopub.status.busy": "2022-07-13T15:58:57.583668Z",
     "iopub.status.idle": "2022-07-13T15:58:57.585521Z",
     "shell.execute_reply": "2022-07-13T15:58:57.585892Z"
    },
    "papermill": {
     "duration": 0.025694,
     "end_time": "2022-07-13T15:58:57.586019",
     "exception": false,
     "start_time": "2022-07-13T15:58:57.560325",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# ConditionStep for evaluating model quality and branching execution.\n",
    "# The `json_path` value is based on the `report_dict` variable in `evaluate.py`\n",
    "\n",
    "cond_lte = ConditionLessThanOrEqualTo(\n",
    "    left=JsonGet(\n",
    "        step_name=step_eval.name,\n",
    "        property_file=evaluation_report,\n",
    "        json_path=\"regression_metrics.mse.value\",\n",
    "    ),\n",
    "    right=6.0,\n",
    ")\n",
    "\n",
    "step_cond = ConditionStep(\n",
    "    name=\"CheckMSEAbaloneEvaluation\",\n",
    "    conditions=[cond_lte],\n",
    "    if_steps=[step_create_model, step_deploy_lambda],\n",
    "    else_steps=[],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "36c4a4c5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:57.629412Z",
     "iopub.status.busy": "2022-07-13T15:58:57.628880Z",
     "iopub.status.idle": "2022-07-13T15:58:57.630967Z",
     "shell.execute_reply": "2022-07-13T15:58:57.630510Z"
    },
    "papermill": {
     "duration": 0.025542,
     "end_time": "2022-07-13T15:58:57.631070",
     "exception": false,
     "start_time": "2022-07-13T15:58:57.605528",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Use the same pipeline name across executions for cache usage.\n",
    "\n",
    "pipeline_name = \"lambda-step-pipeline\" + current_time\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=pipeline_name,\n",
    "    parameters=[\n",
    "        processing_instance_count,\n",
    "        training_instance_type,\n",
    "        input_data,\n",
    "        model_approval_status,\n",
    "    ],\n",
    "    steps=[step_process, step_train, step_eval, step_cond],\n",
    "    sagemaker_session=pipeline_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc75af4c",
   "metadata": {
    "papermill": {
     "duration": 0.019407,
     "end_time": "2022-07-13T15:58:57.670110",
     "exception": false,
     "start_time": "2022-07-13T15:58:57.650703",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Execute the Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0183c74f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:57.713276Z",
     "iopub.status.busy": "2022-07-13T15:58:57.712514Z",
     "iopub.status.idle": "2022-07-13T15:58:57.996122Z",
     "shell.execute_reply": "2022-07-13T15:58:57.995686Z"
    },
    "papermill": {
     "duration": 0.30664,
     "end_time": "2022-07-13T15:58:57.996232",
     "exception": false,
     "start_time": "2022-07-13T15:58:57.689592",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Version': '2020-12-01',\n",
       " 'Metadata': {},\n",
       " 'Parameters': [{'Name': 'ProcessingInstanceCount',\n",
       "   'Type': 'Integer',\n",
       "   'DefaultValue': 1},\n",
       "  {'Name': 'TrainingInstanceType',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 'ml.m5.xlarge'},\n",
       "  {'Name': 'InputDataUrl',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 's3://sagemaker-sample-files/datasets/tabular/uci_abalone/abalone.csv'},\n",
       "  {'Name': 'ModelApprovalStatus',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 'PendingManualApproval'}],\n",
       " 'PipelineExperimentConfig': {'ExperimentName': {'Get': 'Execution.PipelineName'},\n",
       "  'TrialName': {'Get': 'Execution.PipelineExecutionId'}},\n",
       " 'Steps': [{'Name': 'PreprocessAbaloneData',\n",
       "   'Type': 'Processing',\n",
       "   'Arguments': {'ProcessingResources': {'ClusterConfig': {'InstanceType': 'ml.m5.xlarge',\n",
       "      'InstanceCount': {'Get': 'Parameters.ProcessingInstanceCount'},\n",
       "      'VolumeSizeInGB': 30}},\n",
       "    'AppSpecification': {'ImageUri': '246618743249.dkr.ecr.us-west-2.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3',\n",
       "     'ContainerArguments': ['--input-data',\n",
       "      {'Get': 'Parameters.InputDataUrl'}],\n",
       "     'ContainerEntrypoint': ['python3',\n",
       "      '/opt/ml/processing/input/code/preprocess.py']},\n",
       "    'RoleArn': 'arn:aws:iam::000000000000:role/SageMakerRole',\n",
       "    'ProcessingInputs': [{'InputName': 'code',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/input/code/preprocess.py',\n",
       "       'LocalPath': '/opt/ml/processing/input/code',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}}],\n",
       "    'ProcessingOutputConfig': {'Outputs': [{'OutputName': 'train',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/output/train',\n",
       "        'LocalPath': '/opt/ml/processing/train',\n",
       "        'S3UploadMode': 'EndOfJob'}},\n",
       "      {'OutputName': 'validation',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/output/validation',\n",
       "        'LocalPath': '/opt/ml/processing/validation',\n",
       "        'S3UploadMode': 'EndOfJob'}},\n",
       "      {'OutputName': 'test',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/sklearn-abalone-pre-2022-07-13-15-58-56-005/output/test',\n",
       "        'LocalPath': '/opt/ml/processing/test',\n",
       "        'S3UploadMode': 'EndOfJob'}}]}},\n",
       "   'CacheConfig': {'Enabled': True, 'ExpireAfter': '30d'}},\n",
       "  {'Name': 'TrainAbaloneModel',\n",
       "   'Type': 'Training',\n",
       "   'Arguments': {'AlgorithmSpecification': {'TrainingInputMode': 'File',\n",
       "     'TrainingImage': '246618743249.dkr.ecr.us-west-2.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3'},\n",
       "    'OutputDataConfig': {'S3OutputPath': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/AbaloneTrain'},\n",
       "    'StoppingCondition': {'MaxRuntimeInSeconds': 86400},\n",
       "    'ResourceConfig': {'VolumeSizeInGB': 30,\n",
       "     'InstanceCount': 1,\n",
       "     'InstanceType': {'Get': 'Parameters.TrainingInstanceType'}},\n",
       "    'RoleArn': 'arn:aws:iam::000000000000:role/SageMakerRole',\n",
       "    'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
       "        'S3Uri': {'Get': \"Steps.PreprocessAbaloneData.ProcessingOutputConfig.Outputs['train'].S3Output.S3Uri\"},\n",
       "        'S3DataDistributionType': 'FullyReplicated'}},\n",
       "      'ContentType': 'text/csv',\n",
       "      'ChannelName': 'train'},\n",
       "     {'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
       "        'S3Uri': {'Get': \"Steps.PreprocessAbaloneData.ProcessingOutputConfig.Outputs['validation'].S3Output.S3Uri\"},\n",
       "        'S3DataDistributionType': 'FullyReplicated'}},\n",
       "      'ContentType': 'text/csv',\n",
       "      'ChannelName': 'validation'}],\n",
       "    'HyperParameters': {'objective': 'reg:linear',\n",
       "     'num_round': '50',\n",
       "     'max_depth': '5',\n",
       "     'eta': '0.2',\n",
       "     'gamma': '4',\n",
       "     'min_child_weight': '6',\n",
       "     'subsample': '0.7',\n",
       "     'silent': '0'},\n",
       "    'ProfilerRuleConfigurations': [{'RuleConfigurationName': 'ProfilerReport-1657727936',\n",
       "      'RuleEvaluatorImage': '895741380848.dkr.ecr.us-west-2.amazonaws.com/sagemaker-debugger-rules:latest',\n",
       "      'RuleParameters': {'rule_to_invoke': 'ProfilerReport'}}],\n",
       "    'ProfilerConfig': {'S3OutputPath': 's3://sagemaker-us-west-2-000000000000/lambda-step-example/AbaloneTrain'}},\n",
       "   'CacheConfig': {'Enabled': True, 'ExpireAfter': '30d'}},\n",
       "  {'Name': 'EvaluateAbaloneModel',\n",
       "   'Type': 'Processing',\n",
       "   'Arguments': {'ProcessingResources': {'ClusterConfig': {'InstanceType': 'ml.m5.xlarge',\n",
       "      'InstanceCount': 1,\n",
       "      'VolumeSizeInGB': 30}},\n",
       "    'AppSpecification': {'ImageUri': '246618743249.dkr.ecr.us-west-2.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n",
       "     'ContainerEntrypoint': ['python3',\n",
       "      '/opt/ml/processing/input/code/evaluate.py']},\n",
       "    'RoleArn': 'arn:aws:iam::000000000000:role/SageMakerRole',\n",
       "    'ProcessingInputs': [{'InputName': 'input-1',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': {'Get': 'Steps.TrainAbaloneModel.ModelArtifacts.S3ModelArtifacts'},\n",
       "       'LocalPath': '/opt/ml/processing/model',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}},\n",
       "     {'InputName': 'input-2',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': {'Get': \"Steps.PreprocessAbaloneData.ProcessingOutputConfig.Outputs['test'].S3Output.S3Uri\"},\n",
       "       'LocalPath': '/opt/ml/processing/test',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}},\n",
       "     {'InputName': 'code',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-pipeline/lambda-step-exampl-2022-07-13-15-58-56-545/input/code/evaluate.py',\n",
       "       'LocalPath': '/opt/ml/processing/input/code',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}}],\n",
       "    'ProcessingOutputConfig': {'Outputs': [{'OutputName': 'evaluation',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/lambda-step-pipeline/evaluation_report',\n",
       "        'LocalPath': '/opt/ml/processing/evaluation',\n",
       "        'S3UploadMode': 'EndOfJob'}}]}},\n",
       "   'CacheConfig': {'Enabled': True, 'ExpireAfter': '30d'},\n",
       "   'PropertyFiles': [{'PropertyFileName': 'AbaloneEvaluationReport',\n",
       "     'OutputName': 'evaluation',\n",
       "     'FilePath': 'evaluation.json'}]},\n",
       "  {'Name': 'CheckMSEAbaloneEvaluation',\n",
       "   'Type': 'Condition',\n",
       "   'Arguments': {'Conditions': [{'Type': 'LessThanOrEqualTo',\n",
       "      'LeftValue': {'Std:JsonGet': {'PropertyFile': {'Get': 'Steps.EvaluateAbaloneModel.PropertyFiles.AbaloneEvaluationReport'},\n",
       "        'Path': 'regression_metrics.mse.value'}},\n",
       "      'RightValue': 6.0}],\n",
       "    'IfSteps': [{'Name': 'CreateModel-CreateModel',\n",
       "      'Type': 'Model',\n",
       "      'Arguments': {'ExecutionRoleArn': 'arn:aws:iam::000000000000:role/SageMakerRole',\n",
       "       'PrimaryContainer': {'Image': '246618743249.dkr.ecr.us-west-2.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n",
       "        'Environment': {},\n",
       "        'ModelDataUrl': {'Get': 'Steps.TrainAbaloneModel.ModelArtifacts.S3ModelArtifacts'}}}},\n",
       "     {'Name': 'LambdaStep',\n",
       "      'Type': 'Lambda',\n",
       "      'Arguments': {'model_name': {'Get': 'Steps.CreateModel-CreateModel.ModelName'},\n",
       "       'endpoint_config_name': 'demo-lambda-deploy-endpoint-config-07-13-15-58-57',\n",
       "       'endpoint_name': 'demo-lambda-deploy-endpoint-07-13-15-58-57'},\n",
       "      'FunctionArn': 'arn:aws:lambda:us-west-2:000000000000:function:sagemaker-lambda-step-endpoint-deploy-07-13-15-58-57',\n",
       "      'OutputParameters': [{'OutputName': 'statusCode',\n",
       "        'OutputType': 'String'},\n",
       "       {'OutputName': 'body', 'OutputType': 'String'},\n",
       "       {'OutputName': 'other_key', 'OutputType': 'String'}]}],\n",
       "    'ElseSteps': []}}]}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "definition = json.loads(pipeline.definition())\n",
    "definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "468eb3d7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:58.041896Z",
     "iopub.status.busy": "2022-07-13T15:58:58.041369Z",
     "iopub.status.idle": "2022-07-13T15:58:59.758190Z",
     "shell.execute_reply": "2022-07-13T15:58:59.757766Z"
    },
    "papermill": {
     "duration": 1.74154,
     "end_time": "2022-07-13T15:58:59.758302",
     "exception": false,
     "start_time": "2022-07-13T15:58:58.016762",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PipelineArn': 'arn:aws:sagemaker:us-west-2:000000000000:pipeline/lambda-step-pipeline07-13-15-58-57',\n",
       " 'ResponseMetadata': {'RequestId': 'b8946d56-ca49-4275-add7-d48f765b4931',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'b8946d56-ca49-4275-add7-d48f765b4931',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '102',\n",
       "   'date': 'Wed, 13 Jul 2022 15:58:59 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.upsert(role_arn=role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bd52d7e0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:58:59.805754Z",
     "iopub.status.busy": "2022-07-13T15:58:59.805233Z",
     "iopub.status.idle": "2022-07-13T15:59:00.068810Z",
     "shell.execute_reply": "2022-07-13T15:59:00.068289Z"
    },
    "papermill": {
     "duration": 0.289644,
     "end_time": "2022-07-13T15:59:00.068954",
     "exception": false,
     "start_time": "2022-07-13T15:58:59.779310",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution = pipeline.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d30b7748",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T15:59:00.116180Z",
     "iopub.status.busy": "2022-07-13T15:59:00.115665Z",
     "iopub.status.idle": "2022-07-13T16:13:07.234138Z",
     "shell.execute_reply": "2022-07-13T16:13:07.233620Z"
    },
    "papermill": {
     "duration": 847.144553,
     "end_time": "2022-07-13T16:13:07.234262",
     "exception": false,
     "start_time": "2022-07-13T15:59:00.089709",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ca40dc3a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T16:13:07.286763Z",
     "iopub.status.busy": "2022-07-13T16:13:07.286297Z",
     "iopub.status.idle": "2022-07-13T16:16:07.831769Z",
     "shell.execute_reply": "2022-07-13T16:16:07.831267Z"
    },
    "papermill": {
     "duration": 180.576174,
     "end_time": "2022-07-13T16:16:07.831903",
     "exception": false,
     "start_time": "2022-07-13T16:13:07.255729",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a SageMaker client\n",
    "sm_client = sagemaker.Session().sagemaker_client\n",
    "\n",
    "# Wait for the endpoint to be in service\n",
    "waiter = sm_client.get_waiter(\"endpoint_in_service\")\n",
    "waiter.wait(EndpointName=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66aa70a4",
   "metadata": {
    "papermill": {
     "duration": 0.021213,
     "end_time": "2022-07-13T16:16:07.874605",
     "exception": false,
     "start_time": "2022-07-13T16:16:07.853392",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Clean up resources\n",
    "\n",
    "Running the following cell will delete the following resources created in this notebook -\n",
    "* SageMaker Model\n",
    "* SageMaker Endpoint Configuration\n",
    "* SageMaker Endpoint\n",
    "* SageMaker Pipeline\n",
    "* Lambda Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6b8f340a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-07-13T16:16:07.926674Z",
     "iopub.status.busy": "2022-07-13T16:16:07.926158Z",
     "iopub.status.idle": "2022-07-13T16:16:08.804031Z",
     "shell.execute_reply": "2022-07-13T16:16:08.804450Z"
    },
    "papermill": {
     "duration": 0.908752,
     "end_time": "2022-07-13T16:16:08.804592",
     "exception": false,
     "start_time": "2022-07-13T16:16:07.895840",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PipelineArn': 'arn:aws:sagemaker:us-west-2:000000000000:pipeline/lambda-step-pipeline07-13-15-58-57',\n",
       " 'ResponseMetadata': {'RequestId': 'f71a3ce5-bac1-4250-9129-4161507f62d4',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'f71a3ce5-bac1-4250-9129-4161507f62d4',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '102',\n",
       "   'date': 'Wed, 13 Jul 2022 16:16:08 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the model name from the EndpointCofig. The CreateModelStep properties are not available\n",
    "# outside the Pipeline execution context so `step_create_model.properties.ModelName`\n",
    "# cannot be used while deleting the model.\n",
    "\n",
    "model_name = sm_client.describe_endpoint_config(EndpointConfigName=endpoint_config_name)[\n",
    "    \"ProductionVariants\"\n",
    "][0][\"ModelName\"]\n",
    "\n",
    "# Delete the Model\n",
    "sm_client.delete_model(ModelName=model_name)\n",
    "\n",
    "# Delete the EndpointConfig\n",
    "sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
    "\n",
    "# Delete the Endpoint\n",
    "sm_client.delete_endpoint(EndpointName=endpoint_name)\n",
    "\n",
    "# Delete the Lambda function\n",
    "func.delete()\n",
    "\n",
    "# Delete the Pipeline\n",
    "sm_client.delete_pipeline(PipelineName=pipeline_name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/ml_ops|sm-pipelines_lambda_step|sm-pipelines_lambda_step.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "metadata": {
   "interpreter": {
    "hash": "ac2eaa0ea0ebeafcc7822e65e46aa9d4f966f30b695406963e145ea4a91cd4fc"
   }
  },
  "papermill": {
   "default_parameters": {},
   "duration": 1037.911631,
   "end_time": "2022-07-13T16:16:09.344184",
   "environment_variables": {},
   "exception": null,
   "input_path": "sagemaker-pipelines-lambda-step.ipynb",
   "output_path": "/opt/ml/processing/output/sagemaker-pipelines-lambda-step-2022-07-13-15-54-23.ipynb",
   "parameters": {
    "kms_key": "arn:aws:kms:us-west-2:000000000000:1234abcd-12ab-34cd-56ef-1234567890ab"
   },
   "start_time": "2022-07-13T15:58:51.432553",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}