   ml_ops/sm-pipelines_selective_execution/sm-pipelines_selective_execution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bf679922-4032-40ef-b370-6a9b34293ea7",
   "metadata": {},
   "source": [
    "# SageMaker Pipelines: Selective Execution Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65e8d9da-ab2f-4e52-ac05-a0c0f46de9b4",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.\n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf30edaa-ad14-4ae0-a490-afe1a0626661",
   "metadata": {},
   "source": [
    "  In this demo notebook, we will walk a user through,\n",
    "1/ Setting up a new SageMaker pipeline which consists of Pre-Processing, Training, Evaluation and Model Registration - aka end-2-end ML Pipeline\n",
    "2/ How to leverage the previously defined end-2-end pipeline for Selective Execution - to selectively run subsets of your pipeline\n",
    "\n",
    "The flow of this notebook is as follows,\n",
    "- [Download Dataset](#Download-Dataset)\n",
    "- [Define a New Pipeline](#Define-a-New-Pipeline)\n",
    "    - [Setup](#Setup)\n",
    "    - [PreProcessing](#PreProcessing)\n",
    "    - [Train](#Train)\n",
    "    - [Evaluation](#Evaluation)\n",
    "    - [Create Model](#Create-Model)\n",
    "    - [Register Model](#Register-Model)\n",
    "    - [Conditional Check](#Conditional-Check)\n",
    "- [Execute Pipeline](#Execute-Pipeline)\n",
    "- [Selective Execution](#Selective-Execution)\n",
    "    - [Setup](#Setup)\n",
    "    - [Selective Execution Scenarios](#Selective-Execution-Scenarios)\n",
    "        - [Execute Single Pipeline Step with Changes to Parameters](#Execute-Single-Pipeline-Step-with-Changes-to-Parameters)\n",
    "        - [Execute Multiple Contiguous Pipeline Steps or the Entire Pipeline Manually](#Execute-Multiple-Contiguous-Pipeline-Steps-or-the-Entire-Pipeline-Manually)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1a69b6b5-dc28-4987-a945-0a7bc4e76613",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker==2.162.0 in /opt/conda/lib/python3.10/site-packages (2.162.0)\n",
      "Requirement already satisfied: attrs<24,>=23.1.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (23.1.0)\n",
      "Requirement already satisfied: boto3<2.0,>=1.26.131 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.26.153)\n",
      "Requirement already satisfied: cloudpickle==2.2.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (2.2.1)\n",
      "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.2.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.24.3)\n",
      "Requirement already satisfied: protobuf<4.0,>=3.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (3.20.3)\n",
      "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.1.5)\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.0.1)\n",
      "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (4.11.3)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (21.3)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.4.4)\n",
      "Requirement already satisfied: pathos in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.3.0)\n",
      "Requirement already satisfied: schema in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (0.7.5)\n",
      "Requirement already satisfied: PyYAML==6.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (6.0)\n",
      "Requirement already satisfied: jsonschema in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (3.2.0)\n",
      "Requirement already satisfied: platformdirs in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (2.5.2)\n",
      "Requirement already satisfied: tblib==1.7.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker==2.162.0) (1.7.0)\n",
      "Requirement already satisfied: botocore<1.30.0,>=1.29.153 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker==2.162.0) (1.29.153)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker==2.162.0) (0.10.0)\n",
      "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from boto3<2.0,>=1.26.131->sagemaker==2.162.0) (0.6.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.10/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker==2.162.0) (3.8.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->sagemaker==2.162.0) (3.0.9)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.10/site-packages (from protobuf3-to-dict<1.0,>=0.1.5->sagemaker==2.162.0) (1.16.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker==2.162.0) (0.18.0)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from jsonschema->sagemaker==2.162.0) (67.8.0)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.10/site-packages (from pandas->sagemaker==2.162.0) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->sagemaker==2.162.0) (2022.1)\n",
      "Requirement already satisfied: ppft>=1.7.6.6 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (1.7.6.6)\n",
      "Requirement already satisfied: dill>=0.3.6 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (0.3.6)\n",
      "Requirement already satisfied: pox>=0.3.2 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (0.3.2)\n",
      "Requirement already satisfied: multiprocess>=0.70.14 in /opt/conda/lib/python3.10/site-packages (from pathos->sagemaker==2.162.0) (0.70.14)\n",
      "Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.10/site-packages (from schema->sagemaker==2.162.0) (21.6.0)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.10/site-packages (from botocore<1.30.0,>=1.29.153->boto3<2.0,>=1.26.131->sagemaker==2.162.0) (1.26.16)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.1.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.2.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!python3 -m pip install -U sagemaker==2.162.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9dfe4f3c-5d4f-4a70-80db-3a59a89b5bea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import pprint\n",
    "import sagemaker\n",
    "from sagemaker.workflow.pipeline_context import PipelineSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4112dc0c-102c-4094-b15f-bd36911187fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sagemaker_session = sagemaker.session.Session()\n",
    "region = sagemaker_session.boto_region_name\n",
    "role = sagemaker.get_execution_role()\n",
    "pipeline_session = PipelineSession()\n",
    "default_bucket = sagemaker_session.default_bucket()\n",
    "default_bucket_prefix = sagemaker_session.default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"/{default_bucket_prefix}\"\n",
    "\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "s3 = boto3.resource(\"s3\")\n",
    "model_package_group_name = \"Abalone-Models\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d081b072-3023-49b4-8d03-e6f65bcd482d",
   "metadata": {},
   "source": [
    "# Download Dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e58905df-0b7f-4f39-b4af-ecfcfefc0874",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!mkdir -p raw-data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3117a396-d41f-42a9-9b91-2c9fde3fa2b8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3.Bucket(f\"sagemaker-example-files-prod-{region}\").download_file(\n",
    "    \"datasets/tabular/uci_abalone/abalone.csv\", \"raw-data/abalone.csv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "264e1cac-cb5f-4858-b24c-5532f7434f39",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-681580474417/sample-dataset/abalone/abalone.csv\n"
     ]
    }
   ],
   "source": [
    "input_data_uri = sagemaker.s3.S3Uploader.upload(\n",
    "    local_path=\"raw-data/abalone.csv\",\n",
    "    desired_s3_uri=f\"s3://{default_bucket}{default_bucket_prefix_path}/sample-dataset/abalone\",\n",
    ")\n",
    "print(input_data_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ca7c782-a4df-4d59-95e1-80929389b152",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Define a New Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2c85b0c-08e8-4245-a798-04bf4998e91f",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ee6395a7-5573-427d-a51f-16fdb4730235",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.parameters import (\n",
    "    ParameterInteger,\n",
    "    ParameterString,\n",
    "    ParameterFloat,\n",
    ")\n",
    "from sagemaker.workflow.steps import CacheConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51d36af3-9869-49a0-8759-2032ffd1347f",
   "metadata": {},
   "source": [
    "Lets define parameters which would be leveraged throught our pipeline, we can change these parameters during execution and control various aspects of our pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f6230bc2-07c1-4797-af29-215685742044",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "step_cache_config = CacheConfig(\n",
    "    enable_caching=True,\n",
    "    expire_after=\"T12H\",\n",
    ")\n",
    "# processing params\n",
    "processing_instance_count = ParameterInteger(name=\"ProcessingInstanceCount\", default_value=1)\n",
    "processing_instance_type = ParameterString(\n",
    "    name=\"ProcessingInstanceType\", default_value=\"ml.c5.xlarge\"\n",
    ")\n",
    "# training params\n",
    "train_instance_count = ParameterInteger(name=\"TrainingInstanceCount\", default_value=1)\n",
    "train_instance_type = ParameterString(name=\"TrainingInstanceType\", default_value=\"ml.c5.xlarge\")\n",
    "\n",
    "xgb_num_round = ParameterInteger(name=\"XGBNumRounds\", default_value=50)\n",
    "xgb_max_depth = ParameterInteger(name=\"XGBMaxDepth\", default_value=5)\n",
    "xgb_eta = ParameterFloat(name=\"XGBETA\", default_value=0.2)\n",
    "xgb_gamma = ParameterInteger(name=\"XGBGamma\", default_value=4)\n",
    "xgb_min_child_weight = ParameterInteger(name=\"XGBMinChildWeight\", default_value=6)\n",
    "xgb_subsample = ParameterFloat(name=\"XGBSubSample\", default_value=0.7)\n",
    "\n",
    "# model params\n",
    "model_path = ParameterString(\n",
    "    name=\"ModelPath\",\n",
    "    default_value=f\"s3://{default_bucket}{default_bucket_prefix_path}/Abalone/models/\",\n",
    ")\n",
    "\n",
    "default_model_status = ParameterString(name=\"ApprovalStatus\", default_value=\"PendingManualApproval\")\n",
    "\n",
    "input_data = ParameterString(\n",
    "    name=\"InputData\",\n",
    "    default_value=input_data_uri,\n",
    ")\n",
    "\n",
    "threshold = ParameterFloat(name=\"MseThreshold\", default_value=6.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d62033-efd6-4cc9-a6f1-b74d83ea02e3",
   "metadata": {},
   "source": [
    "## PreProcessing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9bd273d5-526d-48e2-819a-592239337e1d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.workflow.steps import ProcessingStep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8501a89a-c780-4f3a-b6ab-5582b319878d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The input argument instance_type of function (sagemaker.image_uris.retrieve) is a pipeline variable (<class 'sagemaker.workflow.parameters.ParameterString'>), which is not allowed. The default_value of this Parameter object will be used to override it. Please make sure the default_value is valid.\n"
     ]
    }
   ],
   "source": [
    "sklearn_processor = SKLearnProcessor(\n",
    "    framework_version=\"0.23-1\",\n",
    "    instance_type=processing_instance_type,\n",
    "    instance_count=processing_instance_count,\n",
    "    base_job_name=\"abalone-preprocess\",\n",
    "    role=role,\n",
    "    sagemaker_session=pipeline_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "78305be6-e205-4f59-96e8-468682e31175",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/sagemaker/workflow/pipeline_context.py:286: UserWarning: Running within a PipelineSession, there will be No Wait, No Logs, and No Job being started.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "processor_args = sklearn_processor.run(\n",
    "    inputs=[\n",
    "        ProcessingInput(source=input_data, destination=\"/opt/ml/processing/input\"),\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train\", source=\"/opt/ml/processing/output/train\"),\n",
    "        ProcessingOutput(output_name=\"validation\", source=\"/opt/ml/processing/output/validation\"),\n",
    "        ProcessingOutput(output_name=\"test\", source=\"/opt/ml/processing/output/test\"),\n",
    "    ],\n",
    "    code=\"code/preprocessing.py\",\n",
    "    arguments=[\n",
    "        \"--train_size\",\n",
    "        str(0.7),\n",
    "        \"--val_size\",\n",
    "        str(0.2),\n",
    "        \"--test_size\",\n",
    "        str(0.1),\n",
    "        \"--random_state\",\n",
    "        str(10),\n",
    "        \"--cat_feature_cols\",\n",
    "        \"sex\",\n",
    "        \"--num_feature_cols\",\n",
    "        \"length,diameter,height,whole_weight,shucked_weight,viscera_weight,shell_weight\",\n",
    "        \"--target_col\",\n",
    "        \"rings\",\n",
    "    ],\n",
    ")\n",
    "\n",
    "step_preprocess = ProcessingStep(\n",
    "    name=\"Abalone-Preprocess\", step_args=processor_args, cache_config=step_cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de432bf9-ba4d-4cf2-9286-62943bd7c1e6",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "72bbb9cf-6b68-42ca-9443-4452b23126d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.estimator import Estimator\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.workflow.steps import TrainingStep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9f7490db-845c-432c-abce-b54ae8e42cba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "image_uri = sagemaker.image_uris.retrieve(\n",
    "    framework=\"xgboost\",\n",
    "    region=region,\n",
    "    version=\"1.0-1\",\n",
    "    py_version=\"py3\",\n",
    "    instance_type=train_instance_type.default_value,\n",
    ")\n",
    "xgb_train = Estimator(\n",
    "    image_uri=image_uri,\n",
    "    instance_count=train_instance_count,\n",
    "    output_path=model_path,\n",
    "    base_job_name=\"abalone-train\",\n",
    "    instance_type=train_instance_type.default_value,\n",
    "    role=role,\n",
    "    sagemaker_session=pipeline_session,\n",
    ")\n",
    "xgb_train.set_hyperparameters(\n",
    "    objective=\"reg:linear\",\n",
    "    num_round=xgb_num_round,\n",
    "    max_depth=xgb_max_depth,\n",
    "    eta=xgb_eta,\n",
    "    gamma=xgb_gamma,\n",
    "    min_child_weight=xgb_gamma,\n",
    "    subsample=xgb_subsample,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "30997087-53ec-45ab-af11-d513d2505d4c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_args = xgb_train.fit(\n",
    "    inputs={\n",
    "        \"train\": TrainingInput(\n",
    "            s3_data=step_preprocess.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"train\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\",\n",
    "        ),\n",
    "        \"validation\": TrainingInput(\n",
    "            s3_data=step_preprocess.properties.ProcessingOutputConfig.Outputs[\n",
    "                \"validation\"\n",
    "            ].S3Output.S3Uri,\n",
    "            content_type=\"text/csv\",\n",
    "        ),\n",
    "    }\n",
    ")\n",
    "\n",
    "step_train = TrainingStep(\n",
    "    name=\"Abalone-Train\", step_args=train_args, cache_config=step_cache_config\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e173cc4-4562-420f-afb0-b98a043105e4",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "52154379-ee37-4cc2-be56-448830ffa651",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.processing import ScriptProcessor\n",
    "from sagemaker.workflow.properties import PropertyFile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ac8bba1a-31bc-46a8-b153-444010f28aba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "script_eval = ScriptProcessor(\n",
    "    image_uri=image_uri,\n",
    "    command=[\"python3\"],\n",
    "    instance_type=processing_instance_type,\n",
    "    volume_size_in_gb=10,\n",
    "    instance_count=1,\n",
    "    base_job_name=\"abalone-postproc\",\n",
    "    role=role,\n",
    "    sagemaker_session=pipeline_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c492ecba-1ae4-4414-b863-a090268fb1be",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "eval_args = script_eval.run(\n",
    "    inputs=[\n",
    "        ProcessingInput(\n",
    "            source=step_train.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "            destination=\"/opt/ml/processing/model\",\n",
    "        ),\n",
    "        ProcessingInput(\n",
    "            source=step_preprocess.properties.ProcessingOutputConfig.Outputs[\"test\"].S3Output.S3Uri,\n",
    "            destination=\"/opt/ml/processing/output/test\",\n",
    "        ),\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\"),\n",
    "    ],\n",
    "    code=\"code/evaluation.py\",\n",
    ")\n",
    "\n",
    "evaluation_report = PropertyFile(\n",
    "    name=\"EvaluationReport\", output_name=\"evaluation\", path=\"evaluation.json\"\n",
    ")\n",
    "\n",
    "step_eval = ProcessingStep(\n",
    "    name=\"Abalone-Evaluate\",\n",
    "    step_args=eval_args,\n",
    "    cache_config=step_cache_config,\n",
    "    property_files=[evaluation_report],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "924756cb-1613-4644-b804-574be2f1071a",
   "metadata": {},
   "source": [
    "## Create Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "30baedc4-0af3-4abc-a8e4-a679cef394de",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.model import Model\n",
    "from sagemaker.inputs import CreateModelInput\n",
    "from sagemaker.workflow.model_step import ModelStep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f876f774-748e-4259-ac6f-77a667c29309",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model = Model(\n",
    "    image_uri=image_uri,\n",
    "    model_data=step_train.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "    sagemaker_session=pipeline_session,\n",
    "    role=role,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8619e368-54eb-4d2f-9252-27bb085c05f7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "step_create_model = ModelStep(\n",
    "    name=\"Abalone-CreateModel\",\n",
    "    step_args=model.create(instance_type=\"ml.m5.large\", accelerator_type=\"ml.eia1.medium\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4adc007a-4536-4f88-9c65-959a8c3dbc0a",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Register Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "29a25590-9d5d-4590-8568-704060c98c75",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.model_metrics import MetricsSource, ModelMetrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "313fcdcb-35e7-477a-bcb1-8b777c15aabc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_metrics = ModelMetrics(\n",
    "    model_statistics=MetricsSource(\n",
    "        s3_uri=\"{}/evaluation.json\".format(\n",
    "            step_eval.arguments[\"ProcessingOutputConfig\"][\"Outputs\"][0][\"S3Output\"][\"S3Uri\"]\n",
    "        ),\n",
    "        content_type=\"application/json\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "658a5f92-4154-47c8-ab6f-f31379e26dd6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "register_args = model.register(\n",
    "    content_types=[\"text/csv\"],\n",
    "    response_types=[\"text/csv\"],\n",
    "    inference_instances=[\"ml.t2.medium\", \"ml.m5.xlarge\"],\n",
    "    transform_instances=[\"ml.m5.xlarge\"],\n",
    "    model_package_group_name=model_package_group_name,\n",
    "    approval_status=default_model_status,\n",
    "    model_metrics=model_metrics,\n",
    ")\n",
    "\n",
    "step_register = ModelStep(name=\"Abalone-ModelRegister\", step_args=register_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "550ca0c0-916c-4a42-bcf9-5c123d61df3b",
   "metadata": {},
   "source": [
    "## Conditional Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "693c4936-ecff-4b70-8f7c-512af15b2231",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.fail_step import FailStep\n",
    "from sagemaker.workflow.functions import Join\n",
    "from sagemaker.workflow.conditions import ConditionLessThanOrEqualTo\n",
    "from sagemaker.workflow.condition_step import ConditionStep\n",
    "from sagemaker.workflow.functions import JsonGet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2c2a3351-73aa-4e18-9a55-ac2dba566f1f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "step_fail = FailStep(\n",
    "    name=\"Abalone-FailNotify\",\n",
    "    error_message=Join(on=\" \", values=[\"Execution failed due to MSE >\", threshold]),\n",
    ")\n",
    "\n",
    "cond_lte = ConditionLessThanOrEqualTo(\n",
    "    left=JsonGet(\n",
    "        step_name=step_eval.name,\n",
    "        property_file=evaluation_report,\n",
    "        json_path=\"regression_metrics.mse.value\",\n",
    "    ),\n",
    "    right=threshold,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "490dcd45-b727-4d95-998e-ce2184abddf2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "step_cond = ConditionStep(\n",
    "    name=\"Abalone-MSECheck\",\n",
    "    conditions=[cond_lte],\n",
    "    if_steps=[step_register, step_create_model],\n",
    "    else_steps=[step_fail],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a976de6e-c5a5-4061-9fa8-cdb088c50dec",
   "metadata": {},
   "source": [
    "# Execute Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4359ccfc-cb81-41a1-8ceb-0a1e00076faf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "from sagemaker.workflow.execution_variables import ExecutionVariables\n",
    "from sagemaker.workflow.pipeline import PipelineExperimentConfig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d83036c2-06d2-4f33-92c4-d96ad4941bd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "experiment_name = \"AbaloneExperimentsPipeline\"\n",
    "\n",
    "# Pipeline experiment config\n",
    "abalone_experiments_config = PipelineExperimentConfig(\n",
    "    experiment_name, ExecutionVariables.PIPELINE_EXECUTION_ID\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "9fc5c324-cb5f-400f-b56a-711e542d9e96",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_name = \"AbalonePipeline-SelectExec\"\n",
    "\n",
    "pipeline = Pipeline(\n",
    "    name=pipeline_name,\n",
    "    pipeline_experiment_config=abalone_experiments_config,\n",
    "    parameters=[\n",
    "        processing_instance_count,\n",
    "        processing_instance_type,\n",
    "        train_instance_count,\n",
    "        train_instance_type,\n",
    "        xgb_num_round,\n",
    "        xgb_max_depth,\n",
    "        xgb_eta,\n",
    "        xgb_gamma,\n",
    "        xgb_min_child_weight,\n",
    "        xgb_subsample,\n",
    "        model_path,\n",
    "        default_model_status,\n",
    "        input_data,\n",
    "        threshold,\n",
    "    ],\n",
    "    steps=[step_preprocess, step_train, step_eval, step_cond],\n",
    "    sagemaker_session=pipeline_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "05290a86-8bd9-4488-aad7-f334fcbda78f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker.workflow._utils:Popping out 'CertifyForMarketplace' from the pipeline definition since it will be overridden in pipeline execution time.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'Version': '2020-12-01',\n",
       " 'Metadata': {},\n",
       " 'Parameters': [{'Name': 'ProcessingInstanceCount',\n",
       "   'Type': 'Integer',\n",
       "   'DefaultValue': 1},\n",
       "  {'Name': 'ProcessingInstanceType',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 'ml.c5.xlarge'},\n",
       "  {'Name': 'TrainingInstanceCount', 'Type': 'Integer', 'DefaultValue': 1},\n",
       "  {'Name': 'TrainingInstanceType',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 'ml.c5.xlarge'},\n",
       "  {'Name': 'XGBNumRounds', 'Type': 'Integer', 'DefaultValue': 50},\n",
       "  {'Name': 'XGBMaxDepth', 'Type': 'Integer', 'DefaultValue': 5},\n",
       "  {'Name': 'XGBETA', 'Type': 'Float', 'DefaultValue': 0.2},\n",
       "  {'Name': 'XGBGamma', 'Type': 'Integer', 'DefaultValue': 4},\n",
       "  {'Name': 'XGBMinChildWeight', 'Type': 'Integer', 'DefaultValue': 6},\n",
       "  {'Name': 'XGBSubSample', 'Type': 'Float', 'DefaultValue': 0.7},\n",
       "  {'Name': 'ModelPath',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 's3://sagemaker-us-east-1-681580474417/Abalone/models/'},\n",
       "  {'Name': 'ApprovalStatus',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 'PendingManualApproval'},\n",
       "  {'Name': 'InputData',\n",
       "   'Type': 'String',\n",
       "   'DefaultValue': 's3://sagemaker-us-east-1-681580474417/sample-dataset/abalone/abalone.csv'},\n",
       "  {'Name': 'MseThreshold', 'Type': 'Float', 'DefaultValue': 6.0}],\n",
       " 'PipelineExperimentConfig': {'ExperimentName': 'AbaloneExperimentsPipeline',\n",
       "  'TrialName': {'Get': 'Execution.PipelineExecutionId'}},\n",
       " 'Steps': [{'Name': 'Abalone-Preprocess',\n",
       "   'Type': 'Processing',\n",
       "   'Arguments': {'ProcessingResources': {'ClusterConfig': {'InstanceType': {'Get': 'Parameters.ProcessingInstanceType'},\n",
       "      'InstanceCount': {'Get': 'Parameters.ProcessingInstanceCount'},\n",
       "      'VolumeSizeInGB': 30}},\n",
       "    'AppSpecification': {'ImageUri': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-scikit-learn:0.23-1-cpu-py3',\n",
       "     'ContainerArguments': ['--train_size',\n",
       "      '0.7',\n",
       "      '--val_size',\n",
       "      '0.2',\n",
       "      '--test_size',\n",
       "      '0.1',\n",
       "      '--random_state',\n",
       "      '10',\n",
       "      '--cat_feature_cols',\n",
       "      'sex',\n",
       "      '--num_feature_cols',\n",
       "      'length,diameter,height,whole_weight,shucked_weight,viscera_weight,shell_weight',\n",
       "      '--target_col',\n",
       "      'rings'],\n",
       "     'ContainerEntrypoint': ['python3',\n",
       "      '/opt/ml/processing/input/code/preprocessing.py']},\n",
       "    'RoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n",
       "    'ProcessingInputs': [{'InputName': 'input-1',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': {'Get': 'Parameters.InputData'},\n",
       "       'LocalPath': '/opt/ml/processing/input',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}},\n",
       "     {'InputName': 'code',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-681580474417/AbalonePipeline-SelectExec/code/5d9f8cf978eb635fdf8359a0de190535/preprocessing.py',\n",
       "       'LocalPath': '/opt/ml/processing/input/code',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}}],\n",
       "    'ProcessingOutputConfig': {'Outputs': [{'OutputName': 'train',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': {'Std:Join': {'On': '/',\n",
       "          'Values': ['s3:/',\n",
       "           'sagemaker-us-east-1-681580474417',\n",
       "           'AbalonePipeline-SelectExec',\n",
       "           {'Get': 'Execution.PipelineExecutionId'},\n",
       "           'Abalone-Preprocess',\n",
       "           'output',\n",
       "           'train']}},\n",
       "        'LocalPath': '/opt/ml/processing/output/train',\n",
       "        'S3UploadMode': 'EndOfJob'}},\n",
       "      {'OutputName': 'validation',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': {'Std:Join': {'On': '/',\n",
       "          'Values': ['s3:/',\n",
       "           'sagemaker-us-east-1-681580474417',\n",
       "           'AbalonePipeline-SelectExec',\n",
       "           {'Get': 'Execution.PipelineExecutionId'},\n",
       "           'Abalone-Preprocess',\n",
       "           'output',\n",
       "           'validation']}},\n",
       "        'LocalPath': '/opt/ml/processing/output/validation',\n",
       "        'S3UploadMode': 'EndOfJob'}},\n",
       "      {'OutputName': 'test',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': {'Std:Join': {'On': '/',\n",
       "          'Values': ['s3:/',\n",
       "           'sagemaker-us-east-1-681580474417',\n",
       "           'AbalonePipeline-SelectExec',\n",
       "           {'Get': 'Execution.PipelineExecutionId'},\n",
       "           'Abalone-Preprocess',\n",
       "           'output',\n",
       "           'test']}},\n",
       "        'LocalPath': '/opt/ml/processing/output/test',\n",
       "        'S3UploadMode': 'EndOfJob'}}]}},\n",
       "   'CacheConfig': {'Enabled': True, 'ExpireAfter': 'T12H'}},\n",
       "  {'Name': 'Abalone-Train',\n",
       "   'Type': 'Training',\n",
       "   'Arguments': {'AlgorithmSpecification': {'TrainingInputMode': 'File',\n",
       "     'TrainingImage': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3'},\n",
       "    'OutputDataConfig': {'S3OutputPath': {'Get': 'Parameters.ModelPath'}},\n",
       "    'StoppingCondition': {'MaxRuntimeInSeconds': 86400},\n",
       "    'ResourceConfig': {'VolumeSizeInGB': 30,\n",
       "     'InstanceCount': {'Get': 'Parameters.TrainingInstanceCount'},\n",
       "     'InstanceType': 'ml.c5.xlarge'},\n",
       "    'RoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n",
       "    'InputDataConfig': [{'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
       "        'S3Uri': {'Get': \"Steps.Abalone-Preprocess.ProcessingOutputConfig.Outputs['train'].S3Output.S3Uri\"},\n",
       "        'S3DataDistributionType': 'FullyReplicated'}},\n",
       "      'ContentType': 'text/csv',\n",
       "      'ChannelName': 'train'},\n",
       "     {'DataSource': {'S3DataSource': {'S3DataType': 'S3Prefix',\n",
       "        'S3Uri': {'Get': \"Steps.Abalone-Preprocess.ProcessingOutputConfig.Outputs['validation'].S3Output.S3Uri\"},\n",
       "        'S3DataDistributionType': 'FullyReplicated'}},\n",
       "      'ContentType': 'text/csv',\n",
       "      'ChannelName': 'validation'}],\n",
       "    'HyperParameters': {'objective': 'reg:linear',\n",
       "     'num_round': {'Std:Join': {'On': '',\n",
       "       'Values': [{'Get': 'Parameters.XGBNumRounds'}]}},\n",
       "     'max_depth': {'Std:Join': {'On': '',\n",
       "       'Values': [{'Get': 'Parameters.XGBMaxDepth'}]}},\n",
       "     'eta': {'Std:Join': {'On': '', 'Values': [{'Get': 'Parameters.XGBETA'}]}},\n",
       "     'gamma': {'Std:Join': {'On': '',\n",
       "       'Values': [{'Get': 'Parameters.XGBGamma'}]}},\n",
       "     'min_child_weight': {'Std:Join': {'On': '',\n",
       "       'Values': [{'Get': 'Parameters.XGBGamma'}]}},\n",
       "     'subsample': {'Std:Join': {'On': '',\n",
       "       'Values': [{'Get': 'Parameters.XGBSubSample'}]}}},\n",
       "    'DebugHookConfig': {'S3OutputPath': {'Get': 'Parameters.ModelPath'},\n",
       "     'CollectionConfigurations': []},\n",
       "    'ProfilerConfig': {'S3OutputPath': {'Get': 'Parameters.ModelPath'},\n",
       "     'DisableProfiler': False}},\n",
       "   'CacheConfig': {'Enabled': True, 'ExpireAfter': 'T12H'}},\n",
       "  {'Name': 'Abalone-Evaluate',\n",
       "   'Type': 'Processing',\n",
       "   'Arguments': {'ProcessingResources': {'ClusterConfig': {'InstanceType': {'Get': 'Parameters.ProcessingInstanceType'},\n",
       "      'InstanceCount': 1,\n",
       "      'VolumeSizeInGB': 10}},\n",
       "    'AppSpecification': {'ImageUri': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n",
       "     'ContainerEntrypoint': ['python3',\n",
       "      '/opt/ml/processing/input/code/evaluation.py']},\n",
       "    'RoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n",
       "    'ProcessingInputs': [{'InputName': 'input-1',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': {'Get': 'Steps.Abalone-Train.ModelArtifacts.S3ModelArtifacts'},\n",
       "       'LocalPath': '/opt/ml/processing/model',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}},\n",
       "     {'InputName': 'input-2',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': {'Get': \"Steps.Abalone-Preprocess.ProcessingOutputConfig.Outputs['test'].S3Output.S3Uri\"},\n",
       "       'LocalPath': '/opt/ml/processing/output/test',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}},\n",
       "     {'InputName': 'code',\n",
       "      'AppManaged': False,\n",
       "      'S3Input': {'S3Uri': 's3://sagemaker-us-east-1-681580474417/AbalonePipeline-SelectExec/code/af3d0ac9a2a46ba2e53409799b1c4dbe/evaluation.py',\n",
       "       'LocalPath': '/opt/ml/processing/input/code',\n",
       "       'S3DataType': 'S3Prefix',\n",
       "       'S3InputMode': 'File',\n",
       "       'S3DataDistributionType': 'FullyReplicated',\n",
       "       'S3CompressionType': 'None'}}],\n",
       "    'ProcessingOutputConfig': {'Outputs': [{'OutputName': 'evaluation',\n",
       "       'AppManaged': False,\n",
       "       'S3Output': {'S3Uri': 's3://sagemaker-us-east-1-681580474417/abalone-postproc-2023-07-25-20-12-29-933/output/evaluation',\n",
       "        'LocalPath': '/opt/ml/processing/evaluation',\n",
       "        'S3UploadMode': 'EndOfJob'}}]}},\n",
       "   'CacheConfig': {'Enabled': True, 'ExpireAfter': 'T12H'},\n",
       "   'PropertyFiles': [{'PropertyFileName': 'EvaluationReport',\n",
       "     'OutputName': 'evaluation',\n",
       "     'FilePath': 'evaluation.json'}]},\n",
       "  {'Name': 'Abalone-MSECheck',\n",
       "   'Type': 'Condition',\n",
       "   'Arguments': {'Conditions': [{'Type': 'LessThanOrEqualTo',\n",
       "      'LeftValue': {'Std:JsonGet': {'PropertyFile': {'Get': 'Steps.Abalone-Evaluate.PropertyFiles.EvaluationReport'},\n",
       "        'Path': 'regression_metrics.mse.value'}},\n",
       "      'RightValue': {'Get': 'Parameters.MseThreshold'}}],\n",
       "    'IfSteps': [{'Name': 'Abalone-ModelRegister-RegisterModel',\n",
       "      'Type': 'RegisterModel',\n",
       "      'Arguments': {'ModelPackageGroupName': 'Abalone-Models',\n",
       "       'ModelMetrics': {'ModelQuality': {'Statistics': {'ContentType': 'application/json',\n",
       "          'S3Uri': 's3://sagemaker-us-east-1-681580474417/abalone-postproc-2023-07-25-20-12-29-933/output/evaluation/evaluation.json'}},\n",
       "        'Bias': {},\n",
       "        'Explainability': {}},\n",
       "       'InferenceSpecification': {'Containers': [{'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n",
       "          'Environment': {},\n",
       "          'ModelDataUrl': {'Get': 'Steps.Abalone-Train.ModelArtifacts.S3ModelArtifacts'}}],\n",
       "        'SupportedContentTypes': ['text/csv'],\n",
       "        'SupportedResponseMIMETypes': ['text/csv'],\n",
       "        'SupportedRealtimeInferenceInstanceTypes': ['ml.t2.medium',\n",
       "         'ml.m5.xlarge'],\n",
       "        'SupportedTransformInstanceTypes': ['ml.m5.xlarge']},\n",
       "       'ModelApprovalStatus': {'Get': 'Parameters.ApprovalStatus'}}},\n",
       "     {'Name': 'Abalone-CreateModel-CreateModel',\n",
       "      'Type': 'Model',\n",
       "      'Arguments': {'ExecutionRoleArn': 'arn:aws:iam::681580474417:role/SageMakerCFDomainCreation-SageMakerExecutionRole-12TMZ4EI9FOSA',\n",
       "       'PrimaryContainer': {'Image': '683313688378.dkr.ecr.us-east-1.amazonaws.com/sagemaker-xgboost:1.0-1-cpu-py3',\n",
       "        'Environment': {},\n",
       "        'ModelDataUrl': {'Get': 'Steps.Abalone-Train.ModelArtifacts.S3ModelArtifacts'}}}}],\n",
       "    'ElseSteps': [{'Name': 'Abalone-FailNotify',\n",
       "      'Type': 'Fail',\n",
       "      'Arguments': {'ErrorMessage': {'Std:Join': {'On': ' ',\n",
       "         'Values': ['Execution failed due to MSE >',\n",
       "          {'Get': 'Parameters.MseThreshold'}]}}}}]}}]}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "definition = json.loads(pipeline.definition())\n",
    "definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0c2aca96-1cd7-4410-aa26-dc22f9689f1e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec',\n",
       " 'ResponseMetadata': {'RequestId': '18bfd020-370d-4e95-ba59-b674c77832b8',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '18bfd020-370d-4e95-ba59-b674c77832b8',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '94',\n",
       "   'date': 'Tue, 25 Jul 2023 20:12:33 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.upsert(role_arn=role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d5267f20-8875-457a-b900-5e4d069623c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution = pipeline.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "289f8ff5-1122-4203-a7f5-20dfaa70f264",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PipelineArn': 'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec',\n",
       " 'PipelineExecutionArn': 'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec/execution/is6ph1fp4go8',\n",
       " 'PipelineExecutionDisplayName': 'execution-1690315954896',\n",
       " 'PipelineExecutionStatus': 'Executing',\n",
       " 'PipelineExperimentConfig': {'ExperimentName': 'AbaloneExperimentsPipeline',\n",
       "  'TrialName': 'is6ph1fp4go8'},\n",
       " 'CreationTime': datetime.datetime(2023, 7, 25, 20, 12, 34, 781000, tzinfo=tzlocal()),\n",
       " 'LastModifiedTime': datetime.datetime(2023, 7, 25, 20, 12, 34, 781000, tzinfo=tzlocal()),\n",
       " 'CreatedBy': {'UserProfileArn': 'arn:aws:sagemaker:us-east-1:681580474417:user-profile/d-biwxq9wovumj/ds-user',\n",
       "  'UserProfileName': 'ds-user',\n",
       "  'DomainId': 'd-biwxq9wovumj'},\n",
       " 'LastModifiedBy': {'UserProfileArn': 'arn:aws:sagemaker:us-east-1:681580474417:user-profile/d-biwxq9wovumj/ds-user',\n",
       "  'UserProfileName': 'ds-user',\n",
       "  'DomainId': 'd-biwxq9wovumj'},\n",
       " 'ResponseMetadata': {'RequestId': '8e12d252-e17c-456e-ba25-88c42feb477f',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '8e12d252-e17c-456e-ba25-88c42feb477f',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '821',\n",
       "   'date': 'Tue, 25 Jul 2023 20:12:35 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "execution.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3a666e2f-71d9-430b-bf8f-2fa2a345c6b1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baab933d-ecc4-488a-b2a4-92a77a4040a5",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Selective Execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c1428589-ddd5-496c-a2b4-9f3c19e652a7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.workflow.selective_execution_config import SelectiveExecutionConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8712e01-e68c-4ade-bb07-698e84f0ec6e",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "562a85e3-090f-4d68-8240-2e340ce288ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pipeline_name = \"AbalonePipeline-SelectExec\"\n",
    "\n",
    "pipeline_arn = Pipeline(name=pipeline_name)\n",
    "\n",
    "# filter for succesfull executions only\n",
    "pipeline_executions = [\n",
    "    _exec\n",
    "    for _exec in pipeline_arn.list_executions()[\"PipelineExecutionSummaries\"]\n",
    "    if _exec[\"PipelineExecutionStatus\"] == \"Succeeded\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "36b2f3b9-c360-4823-8654-b0fccaa68450",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get the last successful execution\n",
    "latest_pipeline_arn = pipeline_executions[0][\"PipelineExecutionArn\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b2df32c6-6fd1-4f66-8707-6393df1b8786",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'arn:aws:sagemaker:us-east-1:681580474417:pipeline/AbalonePipeline-SelectExec/execution/is6ph1fp4go8'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "latest_pipeline_arn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "f5342572-9651-4365-b07a-cbfe082d92d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# list pipeline steps\n",
    "execution_steps = sm_client.list_pipeline_execution_steps(PipelineExecutionArn=latest_pipeline_arn)[\n",
    "    \"PipelineExecutionSteps\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ad2c45f8-a6d0-4dd7-a3e5-e7f70fa9a9c4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'StepName': 'Abalone-Preprocess',\n",
       "  'StartTime': datetime.datetime(2023, 7, 25, 20, 12, 35, 695000, tzinfo=tzlocal()),\n",
       "  'EndTime': datetime.datetime(2023, 7, 25, 20, 17, 1, 128000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Succeeded',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:processing-job/pipelines-is6ph1fp4go8-Abalone-Preprocess-vRZZNaYdbc'}}},\n",
       " {'StepName': 'Abalone-Train',\n",
       "  'StartTime': datetime.datetime(2023, 7, 25, 20, 17, 2, 192000, tzinfo=tzlocal()),\n",
       "  'EndTime': datetime.datetime(2023, 7, 25, 20, 19, 33, 989000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Succeeded',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:training-job/pipelines-is6ph1fp4go8-Abalone-Train-5F3a1Eb2Zd'}}},\n",
       " {'StepName': 'Abalone-Evaluate',\n",
       "  'StartTime': datetime.datetime(2023, 7, 25, 20, 19, 34, 645000, tzinfo=tzlocal()),\n",
       "  'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 4, 114000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Succeeded',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'ProcessingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:processing-job/pipelines-is6ph1fp4go8-Abalone-Evaluate-3Ntw7qvw1A'}}},\n",
       " {'StepName': 'Abalone-MSECheck',\n",
       "  'StartTime': datetime.datetime(2023, 7, 25, 20, 24, 4, 650000, tzinfo=tzlocal()),\n",
       "  'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 4, 863000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Succeeded',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'Condition': {'Outcome': 'True'}}},\n",
       " {'StepName': 'Abalone-ModelRegister-RegisterModel',\n",
       "  'StartTime': datetime.datetime(2023, 7, 25, 20, 24, 5, 676000, tzinfo=tzlocal()),\n",
       "  'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 6, 529000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Succeeded',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'RegisterModel': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:model-package/abalone-models/14'}}},\n",
       " {'StepName': 'Abalone-CreateModel-CreateModel',\n",
       "  'StartTime': datetime.datetime(2023, 7, 25, 20, 24, 5, 676000, tzinfo=tzlocal()),\n",
       "  'EndTime': datetime.datetime(2023, 7, 25, 20, 24, 7, 178000, tzinfo=tzlocal()),\n",
       "  'StepStatus': 'Succeeded',\n",
       "  'AttemptCount': 0,\n",
       "  'Metadata': {'Model': {'Arn': 'arn:aws:sagemaker:us-east-1:681580474417:model/pipelines-is6ph1fp4go8-abalone-createmodel--m2fs0kxign'}}}]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "execution_steps[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "47557f03-ffdb-4580-b020-956fa29e2374",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# list all configureable pipeline parameters - params can be altered during selective execution\n",
    "parameters = sm_client.list_pipeline_parameters_for_execution(\n",
    "    PipelineExecutionArn=latest_pipeline_arn\n",
    ")[\"PipelineParameters\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "572a1abd-01bb-4fac-9443-7e5249073d22",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'Name': 'XGBNumRounds', 'Value': '50'},\n",
       " {'Name': 'XGBSubSample', 'Value': '0.7'},\n",
       " {'Name': 'XGBGamma', 'Value': '4'},\n",
       " {'Name': 'TrainingInstanceCount', 'Value': '1'},\n",
       " {'Name': 'XGBMinChildWeight', 'Value': '6'},\n",
       " {'Name': 'XGBETA', 'Value': '0.2'},\n",
       " {'Name': 'ApprovalStatus', 'Value': 'PendingManualApproval'},\n",
       " {'Name': 'ProcessingInstanceCount', 'Value': '1'},\n",
       " {'Name': 'ProcessingInstanceType', 'Value': 'ml.c5.xlarge'},\n",
       " {'Name': 'MseThreshold', 'Value': '6.0'},\n",
       " {'Name': 'ModelPath',\n",
       "  'Value': 's3://sagemaker-us-east-1-681580474417/Abalone/models/'},\n",
       " {'Name': 'XGBMaxDepth', 'Value': '5'},\n",
       " {'Name': 'TrainingInstanceType', 'Value': 'ml.c5.xlarge'},\n",
       " {'Name': 'InputData',\n",
       "  'Value': 's3://sagemaker-us-east-1-681580474417/sample-dataset/abalone/abalone.csv'}]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56d17938-66c5-4457-8aee-852068e2ec41",
   "metadata": {},
   "source": [
    "## Selective Execution Scenarios"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81f1abe2-1d0e-4605-89dc-1e44f0f652fe",
   "metadata": {},
   "source": [
    "### Execute Single Pipeline Step with Changes to Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f63440eb-96b1-4eee-af2c-8550c4febff9",
   "metadata": {},
   "source": [
    "I would like to retrain the model in hopes to get a better performant model with different hyper parameters without retraining the whole pipeline flow, this is useful in scenarios where a DS would prefer to just test custom hyper parameters before executing the full pipeline flow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "daae6090-470d-46d9-9a1d-b7ae8a6bfa18",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "selective_execution_config = SelectiveExecutionConfig(\n",
    "    source_pipeline_execution_arn=pipeline_executions[0][\"PipelineExecutionArn\"],\n",
    "    selected_steps=[\"Abalone-Train\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "ec964a47-fa5a-401d-80b5-0f5156bd2dcb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "select_execution = pipeline_arn.start(\n",
    "    selective_execution_config=selective_execution_config,\n",
    "    parameters={\n",
    "        \"XGBNumRounds\": 120,\n",
    "        \"XGBSubSample\": 0.9,\n",
    "        \"XGBGamma\": 2,\n",
    "        \"XGBMinChildWeight\": 4,\n",
    "        \"XGBETA\": 0.25,\n",
    "        \"XGBMaxDepth\": 12,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9516d60-1be4-47c5-b71a-083a96870349",
   "metadata": {},
   "source": [
    "Before:\n",
    "\n",
    "![Before Single Step](./media/BeforeSingleStep.jpg)\n",
    "\n",
    "After\n",
    "\n",
    "![After Single Step](./media/AfterSingleStep.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2f577fd-afcc-4b9e-b357-2907e99365c4",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Execute Multiple Contiguous Pipeline Steps or the Entire Pipeline Manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "92543458-21df-4539-ab63-4df1e4f7e8ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "selective_execution_config = SelectiveExecutionConfig(\n",
    "    source_pipeline_execution_arn=latest_pipeline_arn,\n",
    "    selected_steps=[\"Abalone-Train\", \"Abalone-Evaluate\", \"Abalone-MSECheck\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73c09375-0782-4156-a7ba-f138a8bb74a5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "select_execution = pipeline_arn.start(\n",
    "    selective_execution_config=selective_execution_config,\n",
    "    parameters={\n",
    "        \"XGBNumRounds\": 120,\n",
    "        \"XGBSubSample\": 0.9,\n",
    "        \"XGBGamma\": 2,\n",
    "        \"XGBMinChildWeight\": 4,\n",
    "        \"XGBETA\": 0.25,\n",
    "        \"XGBMaxDepth\": 12,\n",
    "        \"MseThreshold\": 6,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "596da7bd-61ba-474a-a129-c4e970b02989",
   "metadata": {},
   "source": [
    "Execute Contiguous Blocks of the Pipeline\n",
    "\n",
    "\n",
    "![Multiple Steps](./media/MultipleSteps.jpg)\n",
    "\n",
    "\n",
    "OR Execute the full pipeline \n",
    "\n",
    "![Full Steps](./media/FullPipeline.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98a9d988-11a2-432c-b3df-67395eb8e3e1",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/ml_ops|sm-pipelines_selective_execution|sm-pipelines_selective_execution.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ace16482-c9cf-4134-8cf9-290b3221d8f6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}