   ml_ops/sm-pipelines_local_mode/sm-pipelines_local_mode.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# SageMaker Pipeline - Local Mode\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "\n",
    "This notebook demonstrates how to orchestrate SageMaker jobs locally using SageMaker Pipelines.\n",
    "\n",
    "Using a `LocalPipelineSession` object, you can now run your pipelines on your local machine before running them in the cloud.\n",
    "\n",
    "The `LocalPipelineSession` object is used while defining each pipeline step and when defining the complete Pipeline object. To run this pipeline in the cloud, each step along with the Pipeline object must be redefined using `PipelineSession`.\n",
    "\n",
    "Note: This notebook will not run in SageMaker Studio. You can run this on SageMaker Classic Notebook instances OR your local IDE."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "In this notebook, we will execute in our local environment a pipeline that will perform the following steps:\n",
    "\n",
    "* ProcessingStep by using `FrameworkProcessor`\n",
    "* TrainingStep by using `Estimator` with a custom PyTorch container"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Dataset\n",
    "\n",
    "We are using a subset of ~20000 records of synthetic transactions, each of which is labeled as fraudulent or not fraudulent.\n",
    "We'd like to train a model based on the features of these transactions so that we can predict risky or fraudulent transactions in the future.\n",
    "\n",
    "This is a binary classification problem:\n",
    "\n",
    "* 1 - Fraud\n",
    "* 0 - No Fraud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! rm -rf ./data && mkdir -p ./data\n",
    "\n",
    "import boto3\n",
    "\n",
    "s3 = boto3.client(\"s3\")\n",
    "s3.download_file(\n",
    "    f\"sagemaker-example-files-prod-{boto3.session.Session().region_name}\",\n",
    "    \"datasets/tabular/synthetic_credit_card_transactions/user0_credit_card_transactions.csv\",\n",
    "    \"data/creditcard_csv.csv\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Prerequisites\n",
    "\n",
    "Install the latest version of the SageMaker Python SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! pip install 'sagemaker' --upgrade"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Build Container\n",
    "\n",
    "In order to use Amazon SageMaker Training Job with a custom image, the first step is to build it and push in a private [Amazon ECR Repository](https://docs.aws.amazon.com/en_en/AmazonECR/latest/userguide/what-is-ecr.html).\n",
    "\n",
    "The Dockerfile defined is creating starting from the public [torch 1.12.1 image](https://hub.docker.com/layers/pytorch/pytorch/1.12.1-cuda11.3-cudnn8-runtime/images/sha256-0bc0971dc8ae319af610d493aced87df46255c9508a8b9e9bc365f11a56e7b75?context=explore), and by the usage of\n",
    "[sagemaker-training-toolkit](https://github.com/aws/sagemaker-training-toolkit) we are making our container compatible with Amazon SageMaker for providing our training script during the definition of the `Estimator`.\n",
    "\n",
    "For facilitating the steps of building the Docker Image and push it in the Amazon ECR Repository, we are providing a utility script [build_image.sh](./code/build_image.sh).\n",
    "\n",
    "For more information on the usage, please read the [README](./code/README.md)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! pygmentize ./code/training/Dockerfile"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Part 1/3 - Setup\n",
    "\n",
    "Here we'll import some libraries and define some variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "import logging\n",
    "import sagemaker\n",
    "from sagemaker.estimator import Estimator\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.workflow.pipeline_context import LocalPipelineSession\n",
    "from sagemaker.processing import FrameworkProcessor, ProcessingInput, ProcessingOutput\n",
    "from sagemaker.sklearn.estimator import SKLearn\n",
    "from sagemaker.workflow.parameters import ParameterString\n",
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.steps import ProcessingStep, TrainingStep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "logging.basicConfig(level=logging.INFO)\n",
    "LOGGER = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "s3_client = boto3.client(\"s3\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Create `LocalPipelineSession` object so that each pipeline step will run locally.\n",
    "\n",
    "To run this pipeline in the cloud, you must change `LocalPipelineSession()` to `PipelineSession()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sagemaker_session = LocalPipelineSession()\n",
    "\n",
    "region = sagemaker_session.boto_region_name\n",
    "\n",
    "default_bucket = sagemaker_session.default_bucket()\n",
    "default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"{default_bucket_prefix}/\"\n",
    "\n",
    "role = None"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Please Note: Provide SageMaker Execution Role ARN if not running on SageMaker Notebook environment"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<div class=\"alert alert-info\"> 💡 \n",
    "    <strong> Set Execution Role for Permissions </strong>\n",
    "    If you are running this notebook from a local machine, as opposed to within the SageMaker Jupyter environment, you will need to add the code below, after filling in the name for a valid SageMaker Execution Role.\n",
    "    <p>\n",
    "        <strong>\n",
    "            <a style=\"color: #0397a7\" href=\"https://console.aws.amazon.com/iam/home#/roles\">\n",
    "                <u>Click here to lookup IAM SageMaker Execution Roles</u>\n",
    "            </a>\n",
    "            The except block below will look up the ARN from the role name.\n",
    "        </strong>\n",
    "    </p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "if role == None:\n",
    "    role = sagemaker.get_execution_role()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Upload Dataset in the Default Amazon S3 Bucket\n",
    "\n",
    "In order to make the data available, we are uploading the downloaded dataset into the default S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "s3_client.delete_object(\n",
    "    Bucket=default_bucket, Key=f\"{default_bucket_prefix_path}sg-pipeline-local/data/input\"\n",
    ")\n",
    "\n",
    "input_data = sagemaker_session.upload_data(\n",
    "    \"./data/creditcard_csv.csv\",\n",
    "    key_prefix=f\"{default_bucket_prefix_path}sg-pipeline-local/data/input\",\n",
    ")\n",
    "\n",
    "input_data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Part 2/3 - Create Amazon SageMaker Pipeline\n",
    "\n",
    "In this section, we are creating the Amazon SageMaker Pipeline and define the proper Input Parameters for making it usable for both local mode and for cloud executions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Compress source code for installing additional python modules\n",
    "\n",
    "By using [sagemaker-training-toolkit](https://github.com/aws/sagemaker-training-toolkit), we can provide the execution scripts and the requirements.txt for installing additional dependencies to the `Estimator` that we will define some steps below.\n",
    "\n",
    "In order to make sure that Amazon SageMaker will install our additional Python modules by reading `requirements.txt`, we are compressing the content of the [training](./code/training) folder and uploading it in the default S3 Bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! cd ./code/training && rm -rf ./../dist/training && mkdir -p ./../dist/training && tar --exclude='Dockerfile' --exclude='.dockerignore' -czvf ./../dist/training/sourcedir.tar.gz *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Download the\n",
    "# clean the buckets first\n",
    "s3_client.delete_object(\n",
    "    Bucket=default_bucket, Key=f\"{default_bucket_prefix_path}sg-pipeline-local/artifact/training\"\n",
    ")\n",
    "\n",
    "code_path = sagemaker_session.upload_data(\n",
    "    \"./code/dist/training/sourcedir.tar.gz\",\n",
    "    key_prefix=f\"{default_bucket_prefix_path}sg-pipeline-local/artifact/training\",\n",
    ")\n",
    "\n",
    "code_path"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "By using `FrameworkProcessor`, we can provide to the Amazon SageMaker Job the execution scripts and the requirements.txt for installing additional Python modules. Look at the [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job-frameworks.html) for additional info.\n",
    "\n",
    "In order to make sure that Amazon SageMaker will install our additional Python modules by reading `requirements.txt`, we are compressing the content of the [processing](./code/processing) folder and uploading it in the default S3 Bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "! cd ./code/processing && rm -rf ./../dist/processing && mkdir -p ./../dist/processing && tar -czvf ./../dist/processing/sourcedir.tar.gz *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Download the\n",
    "# clean the buckets first\n",
    "s3_client.delete_object(\n",
    "    Bucket=default_bucket, Key=f\"{default_bucket_prefix_path}sg-pipeline-local/artifact/processing\"\n",
    ")\n",
    "\n",
    "code_path = sagemaker_session.upload_data(\n",
    "    \"./code/dist/processing/sourcedir.tar.gz\",\n",
    "    key_prefix=f\"{default_bucket_prefix_path}sg-pipeline-local/artifact/processing\",\n",
    ")\n",
    "\n",
    "code_path"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Global Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processing_artifact_path = \"sg-pipeline-local/artifact/processing\"\n",
    "processing_artifact_name = \"sourcedir.tar.gz\"\n",
    "processing_framework_version = \"0.23-1\"\n",
    "processing_instance_count = 1\n",
    "processing_input_files_path = \"sg-pipeline-local/data/input\"\n",
    "processing_output_files_path = \"sg-pipeline-local/data/output\"\n",
    "\n",
    "training_image_name = \"torch-1.12.1\"\n",
    "training_image_version = \"latest\"\n",
    "training_artifact_path = \"sg-pipeline-local/artifact/training\"\n",
    "training_artifact_name = \"sourcedir.tar.gz\"\n",
    "training_output_files_path = \"sg-pipeline-local/models\"\n",
    "training_python_version = \"py37\"\n",
    "training_instance_count = 1\n",
    "training_hyperparameters = {\"epochs\": 6, \"learning_rate\": 1.34e-4, \"batch_size\": 100}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Pipeline Parameters\n",
    "\n",
    "In order to make the Amazon SageMaker Pipeline available for executing it both in `local mode` and in the cloud, we are defining the following `ParameterString` for providing the execution type at runtime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processing_instance_type = ParameterString(\n",
    "    name=\"ProcessingInstanceType\", default_value=\"ml.t3.large\"\n",
    ")\n",
    "\n",
    "training_instance_type = ParameterString(name=\"TrainingInstanceType\", default_value=\"ml.m5.large\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### SageMaker Processing Step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processing_inputs = [\n",
    "    ProcessingInput(\n",
    "        source=\"s3://{}/{}{}\".format(\n",
    "            default_bucket, default_bucket_prefix_path, processing_input_files_path\n",
    "        ),\n",
    "        destination=\"/opt/ml/processing/input\",\n",
    "    )\n",
    "]\n",
    "\n",
    "processing_outputs = [\n",
    "    ProcessingOutput(\n",
    "        output_name=\"output\",\n",
    "        source=\"/opt/ml/processing/output\",\n",
    "        destination=\"s3://{}/{}{}\".format(\n",
    "            default_bucket, default_bucket_prefix_path, processing_output_files_path\n",
    "        ),\n",
    "    )\n",
    "]\n",
    "\n",
    "processing_source_dir = \"s3://{}/{}{}/{}\".format(\n",
    "    default_bucket, processing_artifact_path, default_bucket_prefix_path, processing_artifact_name\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Define the `FrameworkProcessor` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "processor = FrameworkProcessor(\n",
    "    estimator_cls=SKLearn,\n",
    "    framework_version=processing_framework_version,\n",
    "    role=role,\n",
    "    instance_count=processing_instance_count,\n",
    "    instance_type=processing_instance_type,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "run_args = processor.get_run_args(\n",
    "    \"processing.py\",\n",
    "    source_dir=processing_source_dir,\n",
    "    inputs=processing_inputs,\n",
    "    outputs=processing_outputs,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "step_process = ProcessingStep(\n",
    "    name=\"ProcessData\",\n",
    "    code=run_args.code,\n",
    "    processor=processor,\n",
    "    inputs=run_args.inputs,\n",
    "    outputs=run_args.outputs,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### SageMaker Training Step"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Utility methods\n",
    "\n",
    "For providing the compressed `sourcedir` to `Estimator`, we are defining a utility method for encoding the job `hyperparameters`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def json_encode_hyperparameters(hyperparameters):\n",
    "    return {str(k): json.dumps(v) for (k, v) in hyperparameters.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "source_dir = \"s3://{}/{}{}/{}\".format(\n",
    "    default_bucket, default_bucket_prefix_path, training_artifact_path, training_artifact_name\n",
    ")\n",
    "\n",
    "training_hyperparameters[\"sagemaker_program\"] = \"train.py\"\n",
    "training_hyperparameters[\"sagemaker_submit_directory\"] = source_dir\n",
    "\n",
    "training_hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "training_input = TrainingInput(\n",
    "    s3_data=\"s3://{}/{}{}/train\".format(\n",
    "        default_bucket, default_bucket_prefix_path, processing_output_files_path\n",
    "    ),\n",
    "    content_type=\"text/csv\",\n",
    ")\n",
    "\n",
    "test_input = TrainingInput(\n",
    "    s3_data=\"s3://{}/{}{}/test\".format(\n",
    "        default_bucket, default_bucket_prefix_path, processing_output_files_path\n",
    "    ),\n",
    "    content_type=\"text/csv\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "output_path = \"s3://{}/{}{}\".format(\n",
    "    default_bucket, default_bucket_prefix_path, training_output_files_path\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "##### Get ECR image uri\n",
    "\n",
    "Let's take the `image_uri` related to our custom image we want to use for our training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "container = \"{}.dkr.ecr.{}.amazonaws.com/{}:{}\".format(\n",
    "    boto3.client(\"sts\").get_caller_identity().get(\"Account\"),\n",
    "    boto3.session.Session().region_name,\n",
    "    training_image_name,\n",
    "    training_image_version,\n",
    ")\n",
    "\n",
    "print(container)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Define the `Estimator` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "estimator = Estimator(\n",
    "    image_uri=container,\n",
    "    output_path=output_path,\n",
    "    hyperparameters=json_encode_hyperparameters(training_hyperparameters),\n",
    "    enable_sagemaker_metrics=True,\n",
    "    role=role,\n",
    "    instance_count=training_instance_count,\n",
    "    instance_type=training_instance_type,\n",
    "    disable_profiler=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "step_train = TrainingStep(\n",
    "    depends_on=[step_process],\n",
    "    name=\"TrainModel\",\n",
    "    estimator=estimator,\n",
    "    inputs={\"train\": training_input, \"test\": test_input},\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Pipeline definition\n",
    "\n",
    "Let's create the pipeline object, which contains as `parameters` the inputs defined in the previous sections, and as steps the `ProcessingStep` and `TrainingStep` defined few cells above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline = Pipeline(\n",
    "    name=\"FraudTrainingPipeline\",\n",
    "    parameters=[processing_instance_type, training_instance_type],\n",
    "    steps=[step_process, step_train],\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pipeline.upsert(role_arn=role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "json.loads(pipeline.definition())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Part 3/3 - Run SageMaker Pipeline\n",
    "\n",
    "For executing the Amazon SageMaker Pipeline in our local environment, we are providing for both the `ProcessingStep` and `TrainingStep` the parameter `local` for the `instance_type` to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "execution = pipeline.start(\n",
    "    parameters={\"ProcessingInstanceType\": \"local\", \"TrainingInstanceType\": \"local\"}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "execution.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "execution.list_steps()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/ml_ops|sm-pipelines_local_mode|sm-pipelines_local_mode.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}