 archived/xgboost-census-debugger-rules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Explainability-with-Amazon-SageMaker-Debugger\" data-toc-modified-id=\"Explainability-with-Amazon-SageMaker-Debugger-1\">Explainability with Amazon SageMaker Debugger</a></span><ul class=\"toc-item\"><li><span><a href=\"#Introduction-\" data-toc-modified-id=\"Introduction--1.1\">Introduction <a id=\"intro\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#Saving-tensors\" data-toc-modified-id=\"Saving-tensors-1.1.1\">Saving model parameters</a></span></li><li><span><a href=\"#Analysis\" data-toc-modified-id=\"Analysis-1.1.2\">Analysis</a></span></li></ul></li><li><span><a href=\"#Section-1---Setup-\" data-toc-modified-id=\"Section-1---Setup--1.2\">Section 1 - Setup <a id=\"setup\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#1.1-Import-necessary-libraries\" data-toc-modified-id=\"1.1-Import-necessary-libraries-1.2.1\">1.1 Import necessary libraries</a></span></li><li><span><a href=\"#1.2-AWS-region-and--IAM-Role\" data-toc-modified-id=\"1.2-AWS-region-and--IAM-Role-1.2.2\">1.2 AWS region and  IAM Role</a></span></li><li><span><a href=\"#1.3-S3-bucket-and-prefix-to-hold-training-data,-debugger-information-and-model-artifact\" data-toc-modified-id=\"1.3-S3-bucket-and-prefix-to-hold-training-data,-debugger-information-and-model-artifact-1.2.3\">1.3 S3 bucket and prefix to hold training data, debugger information and model artifact</a></span></li></ul></li><li><span><a href=\"#Section-2---Data-preparation-\" data-toc-modified-id=\"Section-2---Data-preparation--1.3\">Section 2 - Data preparation <a id=\"prep-data\"></a></a></span></li><li><span><a href=\"#Section-3---Train-XGBoost-model-in-Amazon-SageMaker-with--debugger-enabled.-\" data-toc-modified-id=\"Section-3---Train-XGBoost-model-in-Amazon-SageMaker-with--debugger-enabled.--1.4\">Section 3 - Train XGBoost model in Amazon SageMaker with  debugger enabled. <a id=\"train\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#3.1-Install-the-'smdebug'-open-source-library\" data-toc-modified-id=\"3.1-Install-the-'smdebug'-open-source-library-1.4.1\">3.1 Install the 'smdebug' open source library</a></span></li><li><span><a href=\"#3.2-Build-the-XGBoost-container\" data-toc-modified-id=\"3.2-Build-the-XGBoost-container-1.4.2\">3.2 Build the XGBoost container</a></span></li><li><span><a href=\"#3.3-Enabling-Debugger-in-Estimator-object\" data-toc-modified-id=\"3.3-Enabling-Debugger-in-Estimator-object-1.4.3\">3.3 Enabling Debugger in Estimator object</a></span><ul class=\"toc-item\"><li><span><a href=\"#DebuggerHookConfig\" data-toc-modified-id=\"DebuggerHookConfig-1.4.3.1\">DebuggerHookConfig</a></span></li><li><span><a href=\"#Rules\" data-toc-modified-id=\"Rules-1.4.3.2\">Rules</a></span></li></ul></li><li><span><a href=\"#3.4-Result\" data-toc-modified-id=\"3.4-Result-1.4.4\">3.4 Result</a></span></li><li><span><a href=\"#3.5-Check-the-status-of-the-Rule-Evaluation-Job\" data-toc-modified-id=\"3.5-Check-the-status-of-the-Rule-Evaluation-Job-1.4.5\">3.5 Check the status of the Rule Evaluation Job</a></span></li></ul></li><li><span><a href=\"#Section-4---Analyze-debugger-output-\" data-toc-modified-id=\"Section-4---Analyze-debugger-output--1.5\">Section 4 - Analyze debugger output <a id=\"analyze-debugger-ouput\"></a></a></span><ul class=\"toc-item\"><li><span><a href=\"#Retrieving-and-Analyzing-tensors\" data-toc-modified-id=\"Retrieving-and-Analyzing-tensors-1.5.1\">Retrieving and Analyzing model parameters</a></span></li><li><span><a href=\"#Plot-Performance-metrics\" data-toc-modified-id=\"Plot-Performance-metrics-1.5.2\">Plot Performance metrics</a></span></li><li><span><a href=\"#Feature-importance\" data-toc-modified-id=\"Feature-importance-1.5.3\">Feature importance</a></span></li><li><span><a href=\"#SHAP\" data-toc-modified-id=\"SHAP-1.5.4\">SHAP</a></span></li><li><span><a href=\"#Global-explanations\" data-toc-modified-id=\"Global-explanations-1.5.5\">Global explanations</a></span></li><li><span><a href=\"#Local-explanations\" data-toc-modified-id=\"Local-explanations-1.5.6\">Local explanations</a></span><ul class=\"toc-item\"><li><span><a href=\"#Force-plot\" data-toc-modified-id=\"Force-plot-1.5.6.1\">Force plot</a></span></li><li><span><a href=\"#Stacked-force-plot\" data-toc-modified-id=\"Stacked-force-plot-1.5.6.2\">Stacked force plot</a></span></li></ul></li><li><span><a href=\"#Outliers\" data-toc-modified-id=\"Outliers-1.5.7\">Outliers</a></span></li></ul></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-1.6\">Conclusion</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explainability with Amazon SageMaker Debugger\n",
    "_**Explain a XGBoost model that predicts an individual's income**_\n",
    "\n",
    "This notebook demonstrates how to use Amazon SageMaker Debugger to capture the feature importance and SHAP values for a XGBoost model.\n",
    "\n",
    "*This notebook was created and tested on an ml.t2.medium notebook instance.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction <a id='intro'></a>\n",
    "\n",
    "Amazon SageMaker Debugger is the capability of Amazon SageMaker that allows debugging machine learning training. The capability helps you monitor the training jobs in near real time using rules and alert you once it has detected inconsistency in training. \n",
    "\n",
    "Using Amazon SageMaker Debugger is a two step process: Saving model parameters and Analysis.\n",
    "Let's look at each one of them closely.\n",
    "\n",
    "### Saving model parameters\n",
    "\n",
    "In machine learning process, model parameters are updated every forward and backward pass and can describe the state of the training job at any particular instant in an ML lifecycle. Amazon SageMaker Debugger  allows you to capture the model parameters and save them for analysis. Although XGBoost is not a deep learning algorithm, Amazon SageMaker Debugger is highly customizable and can help you interpret results by saving insightful metrics. For example, performance metrics or the importance of features at different frequencies. Refer to [SageMaker Debugger  documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/debugger-configuration.html) for details on how to save the metrics you want.  \n",
    "\n",
    "The saved model parameters in this notebook include feature importance and SHAP values for all features in the dataset. The feature importance and SHAP values are what we will use to provide local and global explainability.\n",
    "\n",
    "\n",
    "### Analysis\n",
    "\n",
    "After the model parameters are saved, perform automatic analysis by running debugging ***Rules***.\n",
    "On a very broad level, a rule is Python code used to detect certain conditions during training.\n",
    "Some of the conditions that a data scientist training an algorithm may care about are monitoring for gradients getting too large or too small, detecting overfitting, and so on.\n",
    "Amazon SageMaker Debugger comes pre-packaged with certain rules that can be invoked on Amazon SageMaker. Users can also write their own rules using the Amazon SageMaker Debugger APIs. \n",
    "For more information about automatic analysis using a rule, see the [rules documentation](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/analysis.md).\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1 - Setup <a id='setup'></a>\n",
    "\n",
    "In this section, we will import the necessary libraries, setup variables and examine dataset used. that was used to train the XGBoost model to predict an individual's income.\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "* The AWS region used to host your model.\n",
    "* The IAM role associated with this SageMaker notebook instance.\n",
    "* The S3 bucket used to store the data used to train the model, save debugger information during training and the trained model artifact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='red'>**Important**</font>: To use the new Debugger features, you need to upgrade the SageMaker Python SDK and the SMDebug libary, which we do in the following cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install -U sagemaker smdebug numpy==1.20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Import necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "from sagemaker import get_execution_role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 AWS region and  IAM Role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region = boto3.Session().region_name\n",
    "print(\"AWS Region: {}\".format(region))\n",
    "\n",
    "role = get_execution_role()\n",
    "print(\"RoleArn: {}\".format(role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 S3 bucket and prefix to hold training data, debugger information, and model artifact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = sagemaker.Session().default_bucket()\n",
    "prefix = \"DEMO-smdebug-xgboost-adult-income-prediction\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2 - Data preparation <a id='prep-data'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be using the [Adult Census dataset](https://archive.ics.uci.edu/ml/datasets/adult) for this exercise. \n",
    "This data was extracted from the [1994 Census bureau database](http://www.census.gov/en.html) by Ronny Kohavi and Barry Becker (Data Mining and Visualization, Silicon Graphics), with the task being to predict if an individual person makes over 50K a year. \n",
    "\n",
    "We'll be using the [SHAP](https://github.com/slundberg/shap) library to perform visual analysis. The library contains the dataset pre-loaded which we will utilize here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install shap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shap\n",
    "\n",
    "X, y = shap.datasets.adult()\n",
    "X_display, y_display = shap.datasets.adult(display=True)\n",
    "feature_names = list(X.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a train/test split\n",
    "from sklearn.model_selection import train_test_split  # For splitting the dataset\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=7)\n",
    "X_train_display = X_display.loc[X_train.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.concat(\n",
    "    [pd.Series(y_train, index=X_train.index, name=\"Income>50K\", dtype=int), X_train],\n",
    "    axis=1,\n",
    ")\n",
    "test = pd.concat(\n",
    "    [pd.Series(y_test, index=X_test.index, name=\"Income>50K\", dtype=int), X_test],\n",
    "    axis=1,\n",
    ")\n",
    "\n",
    "# Use 'csv' format to store the data\n",
    "# The first column is expected to be the output column\n",
    "train.to_csv(\"train.csv\", index=False, header=False)\n",
    "test.to_csv(\"validation.csv\", index=False, header=False)\n",
    "\n",
    "boto3.Session().resource(\"s3\").Bucket(bucket).Object(\n",
    "    os.path.join(prefix, \"data/train.csv\")\n",
    ").upload_file(\"train.csv\")\n",
    "boto3.Session().resource(\"s3\").Bucket(bucket).Object(\n",
    "    os.path.join(prefix, \"data/validation.csv\")\n",
    ").upload_file(\"validation.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3 - Train XGBoost model in Amazon SageMaker with  debugger enabled. <a id='train'></a>\n",
    "\n",
    "Now train an XGBoost model with Amazon SageMaker Debugger enabled and monitor the training jobs. This is done using the Amazon SageMaker Estimator API. While the training job is running, use Amazon SageMaker Debugger API to access saved model parameters in real time and visualize them. You can rely on Amazon SageMaker Debugger to take care of downloading a fresh set of model parameters every time you query for them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon SageMaker Debugger is available in Amazon SageMaker XGBoost container version 0.90-2 or later. If you want to use XGBoost with Amazon SageMaker Debugger, you have to specify `repo_version='0.90-2'` in the `get_image_uri` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Build the XGBoost container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon SageMaker Debugger is available in Amazon SageMaker XGBoost container version 0.90-2 or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "container = sagemaker.image_uris.retrieve(\"xgboost\", region, \"0.90-2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_job_name = \"demo-smdebug-xgboost-adult-income-prediction-classification\"\n",
    "bucket_path = \"s3://{}\".format(bucket)\n",
    "\n",
    "hyperparameters = {\n",
    "    \"max_depth\": \"5\",\n",
    "    \"eta\": \"0.2\",\n",
    "    \"gamma\": \"4\",\n",
    "    \"min_child_weight\": \"6\",\n",
    "    \"subsample\": \"0.7\",\n",
    "    \"silent\": \"0\",\n",
    "    \"objective\": \"binary:logistic\",\n",
    "    \"num_round\": \"51\",\n",
    "}\n",
    "save_interval = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Enabling Debugger in Estimator object\n",
    "\n",
    "\n",
    "#### DebuggerHookConfig\n",
    "\n",
    "Enabling Amazon SageMaker Debugger in training job can be accomplished by adding its configuration into Estimator object constructor:\n",
    "\n",
    "```python\n",
    "from sagemaker.debugger import DebuggerHookConfig, CollectionConfig\n",
    "\n",
    "estimator = Estimator(\n",
    "    ...,\n",
    "    debugger_hook_config = DebuggerHookConfig(\n",
    "        s3_output_path=\"s3://{bucket_name}/{location_in_bucket}\",  # Required\n",
    "        collection_configs=[\n",
    "            CollectionConfig(\n",
    "                name=\"metrics\",\n",
    "                parameters={\n",
    "                    \"save_interval\": \"10\"\n",
    "                }\n",
    "            )\n",
    "        ]\n",
    "    )\n",
    ")\n",
    "```\n",
    "Here, the `DebuggerHookConfig` object instructs `Estimator` what data we are interested in.\n",
    "Two parameters are provided in the example:\n",
    "\n",
    "- `s3_output_path`: Points to an S3 bucket where you intend to store model parameters. Amount of data saved depends on multiple factors, major ones are training job, data set, model, frequency of saving model parameters. This S3 bucket should be in your AWS account so that you have full access to control over the stored data. **Note**: The S3 bucket should be originally created in the same Region where your training job is running, otherwise you might run into problems with cross-Region access.\n",
    "\n",
    "- `collection_configs`: It enumerates named collections of model parameters to save. Collections are a convenient way to organize relevant model parameters under same umbrella to make it easy to navigate them during analysis. In this particular example, you are interested in a single collection named metrics. You also configured Amazon SageMaker Debugger to save metrics every 10 iterations. See [Collection](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/api.md#collection) documentation for all parameters that are supported by Collections and DebuggerConfig documentation for more details about all parameters DebuggerConfig supports.\n",
    "  \n",
    "#### Rules\n",
    "\n",
    "Enabling Rules in training job can be accomplished by adding the `rules` configuration into Estimator object constructor.\n",
    "\n",
    "- `rules`: This parameter will accept a list of rules you want to evaluate against training jobs.\n",
    "  For rules, Amazon SageMaker Debugger supports two types:\n",
    "  - SageMaker Rules: These are rules specially curated by the data science and engineering teams in Amazon SageMaker which you can opt to evaluate against your training job.\n",
    "  - Custom Rules: You can optionally choose to write your own rule as a Python source file and have it evaluated against your training job.\n",
    "    To provide Amazon SageMaker Debugger to evaluate this rule, you would have to provide the S3 location of the rule source and the evaluator image.\n",
    "\n",
    "In this example, you will use a Amazon SageMaker's LossNotDecreasing rule, which helps you identify if you are running into a situation where the training loss is not going down.\n",
    "\n",
    "```python\n",
    "from sagemaker.debugger import rule_configs, Rule\n",
    "\n",
    "estimator = Estimator(\n",
    "    ...,\n",
    "    rules=[\n",
    "        Rule.sagemaker(\n",
    "            rule_configs.loss_not_decreasing(),\n",
    "            rule_parameters={\n",
    "                \"collection_names\": \"metrics\",\n",
    "                \"num_steps\": \"10\",\n",
    "            },\n",
    "        ),\n",
    "    ],\n",
    ")\n",
    "```\n",
    "\n",
    "- `rule_parameters`: In this parameter, you provide the runtime values of the parameter in your constructor.\n",
    "  You can still choose to pass in other values which may be necessary for your rule to be evaluated.\n",
    "  In this example, you will use Amazon SageMaker's LossNotDecreasing rule to monitor the `metircs` collection.\n",
    "  The rule will alert you if the loss value in the `metrics` collection has not decreased for more than 10 steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.debugger import rule_configs, Rule, DebuggerHookConfig, CollectionConfig\n",
    "from sagemaker.estimator import Estimator\n",
    "\n",
    "xgboost_estimator = Estimator(\n",
    "    role=role,\n",
    "    base_job_name=base_job_name,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.4xlarge\",\n",
    "    image_uri=container,\n",
    "    hyperparameters=hyperparameters,\n",
    "    max_run=1800,\n",
    "    debugger_hook_config=DebuggerHookConfig(\n",
    "        s3_output_path=bucket_path,  # Required\n",
    "        collection_configs=[\n",
    "            CollectionConfig(name=\"metrics\", parameters={\"save_interval\": str(save_interval)}),\n",
    "            CollectionConfig(\n",
    "                name=\"feature_importance\",\n",
    "                parameters={\"save_interval\": str(save_interval)},\n",
    "            ),\n",
    "            CollectionConfig(name=\"full_shap\", parameters={\"save_interval\": str(save_interval)}),\n",
    "            CollectionConfig(name=\"average_shap\", parameters={\"save_interval\": str(save_interval)}),\n",
    "        ],\n",
    "    ),\n",
    "    rules=[\n",
    "        Rule.sagemaker(\n",
    "            rule_configs.loss_not_decreasing(),\n",
    "            rule_parameters={\n",
    "                \"collection_names\": \"metrics\",\n",
    "                \"num_steps\": str(save_interval * 2),\n",
    "            },\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the next step, start a training job by using the Estimator object you created above. This job is started in an asynchronous, non-blocking way. This means that control is passed back to the notebook and further commands can be run while the training job is progressing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.session import TrainingInput\n",
    "\n",
    "train_input = TrainingInput(\n",
    "    \"s3://{}/{}/{}\".format(bucket, prefix, \"data/train.csv\"), content_type=\"csv\"\n",
    ")\n",
    "validation_input = TrainingInput(\n",
    "    \"s3://{}/{}/{}\".format(bucket, prefix, \"data/validation.csv\"), content_type=\"csv\"\n",
    ")\n",
    "xgboost_estimator.fit(\n",
    "    {\"train\": train_input, \"validation\": validation_input},\n",
    "    # This is a fire and forget event. By setting wait=False, you submit the job to run in the background.\n",
    "    # Amazon SageMaker starts one training job and release control to next cells in the notebook.\n",
    "    # Follow this notebook to see status of the training job.\n",
    "    wait=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Result\n",
    "\n",
    "As a result of the above command, Amazon SageMaker starts **one training job and one rule job** for you. The first one is the job that produces the model parameters to be analyzed. The second one analyzes the model parameters to check if `train-error` and `validation-error` are not decreasing at any point during training.\n",
    "\n",
    "Check the status of the training job below.\n",
    "After your training job is started, Amazon SageMaker starts a rule-execution job to run the LossNotDecreasing rule.  \n",
    "\n",
    "The cell below will block till the training job is complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "for _ in range(36):\n",
    "    job_name = xgboost_estimator.latest_training_job.name\n",
    "    client = xgboost_estimator.sagemaker_session.sagemaker_client\n",
    "    description = client.describe_training_job(TrainingJobName=job_name)\n",
    "    training_job_status = description[\"TrainingJobStatus\"]\n",
    "    rule_job_summary = xgboost_estimator.latest_training_job.rule_job_summary()\n",
    "    rule_evaluation_status = rule_job_summary[0][\"RuleEvaluationStatus\"]\n",
    "    print(\n",
    "        \"Training job status: {}, Rule Evaluation Status: {}\".format(\n",
    "            training_job_status, rule_evaluation_status\n",
    "        )\n",
    "    )\n",
    "\n",
    "    if training_job_status in [\"Completed\", \"Failed\"]:\n",
    "        break\n",
    "\n",
    "    time.sleep(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Check the status of the Rule Evaluation Job\n",
    "\n",
    "To get the rule evaluation job that Amazon SageMaker started for you, run the command below. The results show you the `RuleConfigurationName`, `RuleEvaluationJobArn`, `RuleEvaluationStatus`, `StatusDetails`, and `RuleEvaluationJobArn`.\n",
    "If the model parameters meet a rule evaluation condition, the rule execution job throws a client error with `RuleEvaluationConditionMet`.\n",
    "\n",
    "The logs of the rule evaluation job are available in the Cloudwatch Logstream `/aws/sagemaker/ProcessingJobs` with `RuleEvaluationJobArn`.\n",
    "\n",
    "You can see that once the rule execution job starts, it identifies the loss not decreasing situation in the training job, it raises the `RuleEvaluationConditionMet` exception, and it ends the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgboost_estimator.latest_training_job.rule_job_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4 - Analyze debugger output <a id='analyze-debugger-ouput'></a>\n",
    "\n",
    "Now that you've trained the system, analyze the data.  Here, you focus on after-the-fact analysis.\n",
    "\n",
    "You import a basic analysis library, which defines the concept of trial, which represents a single training run.\n",
    "\n",
    "### Retrieving and Analyzing tensors\n",
    "\n",
    "Before getting to analysis, here are some notes on concepts being used in Amazon SageMaker Debugger that help with analysis.\n",
    "- ***Trial*** - Object that is a centerpiece of the SageMaker Debugger API when it comes to getting access to model parameters. It is a top level abstract that represents a single run of a training job. All model parameters emitted by a training job are associated with its trial.\n",
    "- ***Tensor*** - Object that represents model parameters, such as weights, gradients, accuracy, and loss, that are saved during training job.\n",
    "\n",
    "For more details on aforementioned concepts as well as on SageMaker Debugger API in general (including examples) see [SageMaker Debugger Analysis API](https://github.com/awslabs/sagemaker-debugger/blob/master/docs/analysis.md) documentation.\n",
    "\n",
    "In the following code cell, use a ***Trial*** to access model parameters. You can do that by inspecting currently running training job and extract necessary parameters from its debug configuration to instruct SageMaker Debugger where the data you are looking for is located. Keep in mind the following:\n",
    "- model parameters are being stored in your own S3 bucket to which you can navigate and manually inspect its content if desired.\n",
    "- You might notice a slight delay before trial object is created. This is normal as SageMaker Debugger monitors the corresponding bucket and waits until model parameters to appear. The delay is introduced by less than instantaneous upload of model parameters from a training container to your S3 bucket. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from smdebug.trials import create_trial\n",
    "\n",
    "s3_output_path = xgboost_estimator.latest_job_debugger_artifacts_path()\n",
    "trial = create_trial(s3_output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can list all model parameters that you want to analyze. Each one of these names is the name of a model parameter. The name is a combination of the feature name, which in these cases, is auto-assigned by XGBoost, and whether it's an evaluation metric, feature importance, or SHAP value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trial.tensor_names()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each model parameter, we can get the values at all saved steps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trial.tensor(\"average_shap/f1\").values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot Performance metrics\n",
    "\n",
    "You can also create a simple function that visualizes the training and validation errors as the training progresses.\n",
    "The error should get smaller over time, as the system converges to a good solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import islice\n",
    "import matplotlib.pyplot as plt\n",
    "import re\n",
    "\n",
    "MAX_PLOTS = 35\n",
    "\n",
    "\n",
    "def get_data(trial, tname):\n",
    "    \"\"\"\n",
    "    For the given tensor name, walks though all the iterations\n",
    "    for which you have data and fetches the values.\n",
    "    Returns the set of steps and the values.\n",
    "    \"\"\"\n",
    "    tensor = trial.tensor(tname)\n",
    "    steps = tensor.steps()\n",
    "    vals = [tensor.value(s) for s in steps]\n",
    "    return steps, vals\n",
    "\n",
    "\n",
    "def match_tensor_name_with_feature_name(tensor_name, feature_names=feature_names):\n",
    "    feature_tag = tensor_name.split(\"/\")\n",
    "    for ifeat, feature_name in enumerate(feature_names):\n",
    "        if feature_tag[-1] == \"f{}\".format(str(ifeat)):\n",
    "            return feature_name\n",
    "    return tensor_name\n",
    "\n",
    "\n",
    "def plot_collection(trial, collection_name, regex=\".*\", figsize=(8, 6)):\n",
    "    \"\"\"\n",
    "    Takes a `trial` and a collection name, and\n",
    "    plots all tensors that match the given regex.\n",
    "    \"\"\"\n",
    "    fig, ax = plt.subplots(figsize=figsize)\n",
    "    tensors = trial.collection(collection_name).tensor_names\n",
    "    matched_tensors = [t for t in tensors if re.match(regex, t)]\n",
    "    for tensor_name in islice(matched_tensors, MAX_PLOTS):\n",
    "        steps, data = get_data(trial, tensor_name)\n",
    "        ax.plot(steps, data, label=match_tensor_name_with_feature_name(tensor_name))\n",
    "\n",
    "    ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))\n",
    "    ax.set_xlabel(\"Iteration\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_collection(trial, \"metrics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature importance\n",
    "\n",
    "You can also visualize the feature priorities as determined by\n",
    "[xgboost.get_score()](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.get_score).\n",
    "If you instructed Estimator to log the `feature_importance` collection, all importance types supported by `xgboost.get_score()` will be available in the collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_feature_importance(trial, importance_type=\"weight\"):\n",
    "    SUPPORTED_IMPORTANCE_TYPES = [\n",
    "        \"weight\",\n",
    "        \"gain\",\n",
    "        \"cover\",\n",
    "        \"total_gain\",\n",
    "        \"total_cover\",\n",
    "    ]\n",
    "    if importance_type not in SUPPORTED_IMPORTANCE_TYPES:\n",
    "        raise ValueError(f\"{importance_type} is not one of the supported importance types.\")\n",
    "    plot_collection(trial, \"feature_importance\", regex=f\"feature_importance/{importance_type}/.*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_feature_importance(trial, importance_type=\"cover\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SHAP\n",
    "\n",
    "[SHAP](https://github.com/slundberg/shap) (SHapley Additive exPlanations) is\n",
    "another approach to explain the output of machine learning models.\n",
    "SHAP values represent a feature's contribution to a change in the model output.\n",
    "You instructed Estimator to log the average SHAP values in this example so the SHAP values (as calculated by [xgboost.predict(pred_contribs=True)](https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.predict)) will be available the `average_shap` collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_collection(trial, \"average_shap\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Global explanations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Global explanatory methods allow understanding the model and its feature contributions in aggregate over multiple datapoints. Here we show an aggregate bar plot that plots the mean absolute SHAP value for each feature. \n",
    "\n",
    "Specifically, the below plot indicates that the value of relationship (Wife=5, Husband=4, Own-child=3, Other-relative=2, Unmarried=1, Not-in-family=0) plays the most important role in predicting the income probability being higher than 50K."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap_values = trial.tensor(\"full_shap/f0\").value(trial.last_complete_step)\n",
    "shap_no_base = shap_values[:, :-1]\n",
    "shap_base_value = shap_values[0, -1]\n",
    "shap.summary_plot(shap_no_base, plot_type=\"bar\", feature_names=feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap_base_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The detailed summary plot below can provide more context over the above bar chart. It tells which features are most important and, in addition, their range of effects over the dataset. The color allows us to match how changes in the value of a feature effect the change in prediction. \n",
    "\n",
    "The 'red' indicates higher value of the feature and 'blue' indicates lower (normalized over the features). This allows conclusions such as 'increase in age leads to higher log odds for prediction, eventually leading to `True` predictions more often. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.summary_plot(shap_no_base, X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local explanations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Local explainability aims to explain model behavior for a fixed input point. This can be used for either auditing models before deployment or to provide explanations for specific inference predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.initjs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Force plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A force plot explanation shows how features are contributing to push the model output from the base value (the average model output over the dataset) to the model output. Features pushing the prediction higher are shown in **red**, those pushing the prediction lower are in **blue**.\n",
    "\n",
    "Plot below indicates that for this particular data point the prediction probability (0.48) is higher than the average (~0.2) primarily because this person is in a relationship (`Relationship = Wife`), and to smaller degree because of the higher-than-average age. Similarly the model reduces the probability due specific `Sex` and `Race` values indicating existence of bias in model behavior (possibly due to bias in the data). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.force_plot(\n",
    "    shap_base_value,\n",
    "    shap_no_base[100, :],\n",
    "    X_train_display.iloc[100, :],\n",
    "    link=\"logit\",\n",
    "    matplotlib=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stacked force plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SHAP allows stacking multiple force-plots after rotating 90 degress to understand the explanations for multiple datapoints. If Javascript is enabled, then in the notebook this plot is interactive, allowing understanding the change in output for each feature independently. This stacking of force plots provides a balance between local and global explainability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "N_ROWS = shap_no_base.shape[0]\n",
    "N_SAMPLES = min(100, N_ROWS)\n",
    "sampled_indices = np.random.randint(N_ROWS, size=N_SAMPLES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.force_plot(\n",
    "    shap_base_value,\n",
    "    shap_no_base[sampled_indices, :],\n",
    "    X_train_display.iloc[sampled_indices, :],\n",
    "    link=\"logit\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers\n",
    "\n",
    "Outliers are extreme values that deviate from other observations on data. It's useful to understand the influence of various features for outlier predictions to determine if it's a novelty, an experimental error, or a shortcoming in the model.\n",
    "\n",
    "Here we show force plot for prediction outliers that are on either side of the baseline value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# top outliers\n",
    "from scipy import stats\n",
    "\n",
    "N_OUTLIERS = 3  # number of outliers on each side of the tail\n",
    "\n",
    "shap_sum = np.sum(shap_no_base, axis=1)\n",
    "z_scores = stats.zscore(shap_sum)\n",
    "outlier_indices = (np.argpartition(z_scores, -N_OUTLIERS)[-N_OUTLIERS:]).tolist()\n",
    "outlier_indices += (np.argpartition(z_scores, N_OUTLIERS)[:N_OUTLIERS]).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for fig_index, outlier_index in enumerate(outlier_indices, start=1):\n",
    "    shap.force_plot(\n",
    "        shap_base_value,\n",
    "        shap_no_base[outlier_index, :],\n",
    "        X_train_display.iloc[outlier_index, :],\n",
    "        matplotlib=False,\n",
    "        link=\"logit\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook discussed the importance of explainability for improved ML\n",
    "adoption and. We introduced the Amazon SageMaker Debugger capability with built-in\n",
    "model parameter collections to enable model explainability.\n",
    "The notebook walked you through training an ML model for a financial services use case\n",
    "of individual income prediction. We further analyzed the global and local\n",
    "explanations of the model by visualizing the captured model parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker-debugger|xgboost_census_explanations|xgboost-census-debugger-rules.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}