      build_and_train_models/sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperband Automatic Model Tuning for Distributed Training\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project is based on the [HPO XGBoost notebook](https://github.com/aws/amazon-sagemaker-examples/blob/main/hyperparameter_tuning/xgboost_direct_marketing/hpo_xgboost_direct_marketing_sagemaker_python_sdk.ipynb)\n",
    "\n",
    "---\n",
    "\n",
    "Kernel `Python 3 (Data Science)` works well with this notebook.\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Background](#Background)\n",
    "1. [Preparation](#Preparation)\n",
    "1. [Data Downloading and Transformation](#Data_Downloading)\n",
    "1. [Demo setup](#Demo_setup)  \n",
    "1. [Run Demo](#Run_Demo)  \n",
    "1. [Setup Hyperparameter Tuning](#Setup_Hyperparameter_Tuning)\n",
    "1. [Launch Hyperparameter Tuning](#Launch_Hyperparameter_Tuning)\n",
    "1. [Analyze Hyperparameter Tuning Results](#Analyze_Hyperparameter_Tuning_Results)\n",
    "1. [Tuning with multiple scaling options](#Tuning_with_multiple_scaling_options)\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Background\n",
    "With increased adoption of XXL models such as LLMs and Generative AI, distributed training is becoming a standard method for model training: using data parallelism to scale over multiple nodes and workers and model parallelism in order to fit the actual models on designated infrastructure. Amazon SageMaker has continuously innovated in this space by launching features like Heterogeneous Clusters and Distributed training Library.\n",
    "However, moving from single to distributed environment has an impact on training hyperparameters which, in turn, impact model accuracy.  \n",
    "Thus, distributed training needs to balance 3 vectors: distribution, hyperparameters and accuracy.\n",
    "\n",
    "This notebook will demo the need for Hyperparameter Optimization (HPO) when moving from single to distributed environment, and show how to configure a SageMaker Automatic Model Tuning (AMT) job using Hyperband strategy.\n",
    "We will train a model which can be used to predict if a customer will enroll for a term deposit at a bank, after one or more phone calls. Hyperparameter tuning will be used in order to try multiple hyperparameter settings and produce the best model.\n",
    "\n",
    "We will use SageMaker Python SDK, a high level SDK, to simplify the way we interact with SageMaker Hyperparameter Tuning.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "## Preparation\n",
    "\n",
    "Let's start by:\n",
    "- Installing latest SageMaker SDK \n",
    "- Specifying the S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as SageMaker training.\n",
    "- Specifying the IAM role used to give training access to your data. See SageMaker documentation for how to create these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "! pip install sagemaker --upgrade --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "from sagemaker.tuner import (\n",
    "    IntegerParameter,\n",
    "    CategoricalParameter,\n",
    "    ContinuousParameter,\n",
    "    HyperparameterTuner,\n",
    "    StrategyConfig,\n",
    "    HyperbandStrategyConfig,\n",
    ")\n",
    "\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker.s3 import S3Downloader\n",
    "\n",
    "import numpy as np  # For matrix operations and numerical processing\n",
    "import pandas as pd  # For munging tabular data\n",
    "import os\n",
    "from time import gmtime, strftime\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "smclient = boto3.Session().client(\"sagemaker\")\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
    "prefix = \"sagemaker/DEMO-dist-hpo-xgboost-dm\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    prefix = f\"{default_bucket_prefix}/{prefix}\"\n",
    "\n",
    "bucket"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Data Downloading and Transformation\n",
    "In the next few cells we will download and prepare our data for training.  \n",
    "Let's start by downloading the [direct marketing dataset](https://archive.ics.uci.edu/ml/datasets/bank+marketing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_uri = f\"s3://sagemaker-example-files-prod-{region}/datasets/tabular/uci_bank_marketing/bank-additional-full.csv\"\n",
    "S3Downloader.download(\n",
    "    s3_uri, local_path=\"./bank-additional/\", sagemaker_session=sagemaker.Session()\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will read into a Pandas data frame and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!ls -lh ./bank-additional/bank-additional-full.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"./bank-additional/bank-additional-full.csv\", sep=\",\")\n",
    "pd.set_option(\"display.max_columns\", 500)  # Make sure we can see all of the columns\n",
    "pd.set_option(\"display.max_rows\", 50)  # Keep the output on one page\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some high level description of the data used:\n",
    "\n",
    "* We have a little over 40K customer records, and 20 features for each customer\n",
    "* The features are mixed; some numeric, some categorical\n",
    "* The data appears to be sorted, at least by `time` and `contact`, maybe more\n",
    "* It includes features related to customer demographics, past events (like credit defaults and loans) and past campaigns\n",
    "\n",
    "The target variable `y` indicates whether the client has subscribed a term deposit (binary: 'yes','no').  \n",
    "For more details on this data set, please see the original [HPO XGBoost notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/main/hyperparameter_tuning/xgboost_direct_marketing/hpo_xgboost_direct_marketing_sagemaker_python_sdk.ipynb)  \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data_Transformation\n",
    "Our next steps will be to prepare our data for training.   \n",
    "Because data preparation is not the focus of this example, we will not elaborate on these steps, but they include: feature encoding, grouping and feature selection.  \n",
    "For detailed explanation about these steps please see the original [HPO XGBoost notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/main/hyperparameter_tuning/xgboost_direct_marketing/hpo_xgboost_direct_marketing_sagemaker_python_sdk.ipynb)  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data[\"no_previous_contact\"] = np.where(\n",
    "    data[\"pdays\"] == 999, 1, 0\n",
    ")  # Indicator variable to capture when pdays takes a value of 999\n",
    "\n",
    "data[\"not_working\"] = np.where(\n",
    "    np.in1d(data[\"job\"], [\"student\", \"retired\", \"unemployed\"]), 1, 0\n",
    ")  # Indicator for individuals not actively employed\n",
    "\n",
    "model_data = pd.get_dummies(data)  # Convert categorical variables to sets of indicators\n",
    "model_data = model_data.drop(\n",
    "    [\"duration\", \"emp.var.rate\", \"cons.price.idx\", \"cons.conf.idx\", \"euribor3m\", \"nr.employed\"],\n",
    "    axis=1,\n",
    ")\n",
    "model_data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Demo_setup\n",
    "In the next few cells we will prepare the data for demoing how scaling out training, without hyperparameter tuning impacts model performance.  \n",
    "In this experiment we will train an xgboost model once on a single instance and once using 3 instances."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll then split the dataset into training (85%), validation (5%), and test (10%) datasets and convert the datasets to the right format the algorithm expects. We will use training and validation datasets during training. Test dataset can be used to evaluate model performance after it is deployed to an endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_data, validation_data, test_data = np.split(\n",
    "    model_data.sample(frac=1, random_state=1729),\n",
    "    [int(0.85 * len(model_data)), int(0.9 * len(model_data))],\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purpose of having a meaningful distributed training job, we will replicate our initial training and validation dataset by a factor of 100, and then shuffle the records in place. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "replicated_df = pd.DataFrame(np.repeat(train_data.values, 100, axis=0))\n",
    "replicated_df.columns = train_data.columns\n",
    "train_data = replicated_df\n",
    "train_data = train_data.sample(frac=1).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "replicated_df = pd.DataFrame(np.repeat(validation_data.values, 100, axis=0))\n",
    "replicated_df.columns = validation_data.columns\n",
    "validation_data = replicated_df\n",
    "validation_data = validation_data.sample(frac=1).reset_index(drop=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we are using SageMaker built-in XGBoost algorithm, the first column must be the target variable.  Also, notice that although repetitive it's easiest to do this after the train|validation|test split rather than before.  This avoids any misalignment issues due to random reordering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_data = pd.concat([train_data[\"y_yes\"], train_data.drop([\"y_no\", \"y_yes\"], axis=1)], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.concat(\n",
    "    [validation_data[\"y_yes\"], validation_data.drop([\"y_no\", \"y_yes\"], axis=1)], axis=1\n",
    ").to_csv(\"validation.csv\", index=False, header=False)\n",
    "\n",
    "pd.concat([test_data[\"y_yes\"], test_data.drop([\"y_no\", \"y_yes\"], axis=1)], axis=1).to_csv(\n",
    "    \"test.csv\", index=False, header=False\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will need to split our training data into multiple files. \n",
    "[SageMaker Documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_S3DataSource.html) says this on how files are distributed in multi-instance training:  \n",
    "> If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.\n",
    "\n",
    "Please note: You shouldn't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data, and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.\n",
    "\n",
    "For this example, we will use 3 instances for the distributed training job, thus we will split our training set to 3 files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "max_instances = 3\n",
    "for indx, partition in enumerate(np.array_split(train_data, max_instances)):\n",
    "    partition.to_csv(f\"train_{indx}.csv\", index=False, header=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, upload the files to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "trainingFiles = glob.glob(\"train_*.csv\")\n",
    "for trainingFile in trainingFiles:\n",
    "    osPath = os.path.join(prefix, \"train/{}\".format(trainingFile))\n",
    "    print(osPath)\n",
    "    boto3.Session().resource(\"s3\").Bucket(bucket).Object(osPath).upload_file(trainingFile)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll copy the validation file to S3 for Amazon SageMaker training to pickup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "boto3.Session().resource(\"s3\").Bucket(bucket).Object(\n",
    "    os.path.join(prefix, \"validation/validation.csv\")\n",
    ").upload_file(\"validation.csv\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will set the train and validation input channels.   \n",
    "\n",
    "\n",
    "In distributed training, where you use multiple ML compute instances, you should choose ```ShardedByS3Key```. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode is set to File), this copies 1/n of the number of objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_input_train = TrainingInput(\n",
    "    distribution=\"ShardedByS3Key\",\n",
    "    s3_data=\"s3://{}/{}/train\".format(bucket, prefix),\n",
    "    content_type=\"csv\",\n",
    ")\n",
    "s3_input_validation = TrainingInput(\n",
    "    s3_data=\"s3://{}/{}/validation/\".format(bucket, prefix), content_type=\"csv\"\n",
    ")\n",
    "\n",
    "print(s3_input_train.config[\"DataSource\"][\"S3DataSource\"][\"S3Uri\"])\n",
    "print(s3_input_validation.config[\"DataSource\"][\"S3DataSource\"][\"S3Uri\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run_Demo"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that our data is ready and channels configured, we'll run a quick experiment: run single Vs. distributed training and observe the effect on model convergence.  \n",
    "  \n",
    "Let's start by defining a simple function to build our estimator.  \n",
    "In this example, we are using SageMaker Python SDK to configure the training job, which includes:\n",
    "\n",
    "* The container image for the algorithm (XGBoost)\n",
    "* Configuration for the output of the training jobs\n",
    "* The values of static algorithm hyperparameters, those that are not specified will be given default values\n",
    "* The type and number of instances to use for the training jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_XGB_estimnator(instnaces=1):\n",
    "    container = sagemaker.image_uris.retrieve(\"xgboost\", boto3.Session().region_name, \"latest\")\n",
    "\n",
    "    max_run = 2400\n",
    "    max_wait = 3600\n",
    "\n",
    "    xgbEstimator = sagemaker.estimator.Estimator(\n",
    "        container,\n",
    "        role,\n",
    "        instance_count=instnaces,\n",
    "        instance_type=\"ml.m5.2xlarge\",\n",
    "        output_path=\"s3://{}/{}/output\".format(bucket, prefix),\n",
    "        sagemaker_session=sagemaker.Session(),\n",
    "        # use_spot_instances=True,\n",
    "        # max_run=max_run,\n",
    "        # max_wait=max_wait,\n",
    "    )\n",
    "\n",
    "    xgbEstimator.set_hyperparameters(\n",
    "        eval_metric=\"auc\",\n",
    "        objective=\"binary:logistic\",\n",
    "        num_round=150,\n",
    "        rate_drop=0.0,\n",
    "        tree_method=\"exact\",\n",
    "        eta=0.3,\n",
    "        colsample_bylevel=0.5,\n",
    "        colsample_bytree=0.8,\n",
    "    )\n",
    "    return xgbEstimator"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run single instance training\n",
    "Lets' start by running the single instance job first. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "xgb = get_XGB_estimnator()\n",
    "training_job_name = \"xgb-single-instance-{}\".format(strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime()))\n",
    "xgb.fit(\n",
    "    {\"train\": s3_input_train, \"validation\": s3_input_validation},\n",
    "    job_name=training_job_name,\n",
    "    wait=True,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see the job progress in the SageMaker console.  \n",
    "Once the job is done, we will use the estimator's ```training_job_analytics``` API to retrieve model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "single_df = xgb.training_job_analytics.dataframe(force_refresh=False)\n",
    "single_df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While some variations can occur, in the current run we see training took 180 seconds (3 minutes) with train:auc of 0.90 and validation:auc at 0.78."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Run distributed training - not optimized\n",
    "Now, we'll run the training job in distributed mode but with no change to the hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "xgb3 = get_XGB_estimnator(3)\n",
    "training_job_name = \"xgb-3-instances-{}\".format(strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime()))\n",
    "\n",
    "xgb3.fit(\n",
    "    {\"train\": s3_input_train, \"validation\": s3_input_validation},\n",
    "    wait=True,\n",
    "    job_name=training_job_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dist_df = xgb3.training_job_analytics.dataframe(force_refresh=True)\n",
    "dist_df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "As expected, the distributed training took about one minute, but the model suffered a drop in performance with train:auc 0.86 and validation:auc of 0.63! "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Result discussion\n",
    "There are a number of potential reasons why this happens.  \n",
    "It was observed that both synchronous distribution algorithms like allreduce (which XGBoost uses) [might perform poorly](https://arxiv.org/pdf/1710.06952v3.pdf), as well as asynchronous distribution methods like parameter server also suffer significantly worse convergence [when traffic to parameter server is congested](https://arxiv.org/pdf/1710.06952v3.pdf). Additionally, for deep learning models, [weight staleness](https://www.cs.cmu.edu/~epxing/papers/2019/Dai_etal_ICLR19.pdf) and degradation in model quality due to [larger mini-batch size](https://arxiv.org/pdf/1609.04836.pdf) have been observed. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Setup_Hyperparameter_Tuning \n",
    "*Note, with the default setting below, the hyperparameter tuning job can take about 30 minutes to complete.*\n",
    "\n",
    "We will use SageMaker hyperparameter tuning to automate the searching process effectively. Specifically, we specify a range, or a list of possible values in the case of categorical hyperparameters, for each of the hyperparameter that we plan to tune. SageMaker hyperparameter tuning will automatically launch multiple training jobs with different hyperparameter settings, evaluate results of those training jobs based on a predefined \"objective metric\", and select the hyperparameter settings for future attempts based on previous results. For each hyperparameter tuning job, we will give it a budget (max number of training jobs) and it will complete once that many training jobs have been executed.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start by specifying the objective metric that we'd like to tune and its definition, which includes the regular expression (Regex) needed to extract that metric from the CloudWatch logs of the training job. Since we are using built-in XGBoost algorithm here, it emits two predefined metrics: *validation:auc* and *train:auc*, and we elected to monitor *validation:auc* as you can see below. In this case, we only need to specify the metric name and do not need to provide regex. If you bring your own algorithm, your algorithm emits metrics by itself. In that case, you'll need to add a MetricDefinition object here to define the format of those metrics through regex, so that SageMaker knows how to extract those metrics from your CloudWatch logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "objective_metric_name = \"validation:auc\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will tune seven hyperparameters in this examples:\n",
    "* *num_round*: The number of rounds for boosting during the training.\n",
    "* *eta*: Step size shrinkage used in updates to prevent overfitting. After each boosting step, you can directly get the weights of new features. The eta parameter actually shrinks the feature weights to make the boosting process more conservative. \n",
    "* *alpha*: L1 regularization term on weights. Increasing this value makes models more conservative. \n",
    "* *min_child_weight*: Minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the sum of instance weight less than min_child_weight, the building process gives up further partitioning. In linear regression models, this simply corresponds to a minimum number of instances needed in each node. The larger the algorithm, the more conservative it is. \n",
    "* *max_depth*: Maximum depth of a tree. Increasing this value makes the model more complex and likely to be overfitted. \n",
    "* *colsample_bylevel*: Subsample ratio of columns for each split, in each level. This subsampling takes place once for every new depth level reached in a tree.\n",
    "* *colsample_bytree*: Subsample ratio of columns when constructing each tree. For every tree constructed the subsampling occurs once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hyperparameter_ranges = {\n",
    "    \"num_round\": IntegerParameter(100, 200),\n",
    "    \"eta\": ContinuousParameter(0, 1),\n",
    "    \"min_child_weight\": ContinuousParameter(1, 10),\n",
    "    \"alpha\": ContinuousParameter(0, 2),\n",
    "    \"max_depth\": IntegerParameter(1, 10),\n",
    "    \"colsample_bylevel\": ContinuousParameter(0, 1),\n",
    "    \"colsample_bytree\": ContinuousParameter(0, 1),\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we provide the Hyperband configuration.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "hsc = HyperbandStrategyConfig(max_resource=30, min_resource=1)\n",
    "sc = StrategyConfig(hyperband_strategy_config=hsc)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll create a `HyperparameterTuner` object, to which we pass:\n",
    "- The XGBoost estimator we created above, set to run with 3 instances\n",
    "- Objective metric name and definition\n",
    "- Our hyperparameter ranges\n",
    "- Tuning resource configurations such as Number of training jobs to run in total and how many training jobs can be run in parallel.\n",
    "- Hyperband settings: strategy and configuration\n",
    "\n",
    "Training jobs can be stopped early when they are unlikely to improve the objective metric of the hyperparameter tuning job. This can help reduce compute time and avoid overfitting your model. Hyperband uses an advanced internal mechanism to apply early stopping. Thus, the parameter TrainingJobEarlyStoppingType in the HyperParameterTuningJobConfig API must be set to OFF when using the Hyperband internal early stopping feature.  \n",
    "- Early stopping set to off"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuner = HyperparameterTuner(\n",
    "    xgb3,\n",
    "    objective_metric_name,\n",
    "    hyperparameter_ranges,\n",
    "    max_jobs=30,\n",
    "    max_parallel_jobs=4,\n",
    "    strategy=\"Hyperband\",\n",
    "    early_stopping_type=\"Off\",\n",
    "    strategy_config=sc,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch_Hyperparameter_Tuning\n",
    "Now we can launch a hyperparameter tuning job by calling *fit()* function. After the hyperparameter tuning job is created, we can go to SageMaker console to track the progress of the hyperparameter tuning job until it is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuner.fit(\n",
    "    {\"train\": s3_input_train, \"validation\": s3_input_validation},\n",
    "    include_cls_metadata=False,\n",
    "    wait=True,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's just run a quick check of the hyperparameter tuning jobs status to make sure it started successfully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sm = boto3.client(\"sagemaker\")\n",
    "sm.describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuner.latest_tuning_job.job_name\n",
    ")[\"HyperParameterTuningJobStatus\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze_Hyperparameter_Tuning_Results\n",
    "Here we analyze the tuning job results - after tuning job is completed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuning_job_name = tuner.latest_tuning_job.job_name\n",
    "tuning_job_name"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how good our tuned model did"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuner = HyperparameterTuner.attach(tuning_job_name=tuning_job_name)\n",
    "tuner.describe()[\"BestTrainingJob\"][\"FinalHyperParameterTuningJobObjectiveMetric\"][\"Value\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Again, some variations may occur but you should see that the best job was able to optimize the model with over to 0.78 on the validation:auc!  \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let fetch results about a hyperparameter tuning job and make them accessible for analysis in a pandas data frame.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tuner_analytics = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
    "\n",
    "full_df = tuner_analytics.dataframe()\n",
    "full_df.sort_values(by=[\"FinalObjectiveValue\"], ascending=False).head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When running the training independently, we saw it took 4 minutes.  \n",
    "Let's see the average time a training job took with Hyperband strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "full_df[\"TrainingElapsedTimeSeconds\"].mean()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average job took about 1 minute!  \n",
    "\n",
    "Finally, we can further examine results to see how multiple parameters are related."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "threedee = plt.figure().add_subplot(projection=\"3d\")\n",
    "threedee.scatter(\n",
    "    full_df[\"TrainingElapsedTimeSeconds\"], full_df[\"num_round\"], full_df[\"FinalObjectiveValue\"]\n",
    ")\n",
    "threedee.set_xlabel(\"Training Time\")\n",
    "threedee.set_ylabel(\"Rounds\")\n",
    "threedee.set_zlabel(\"AUC\")\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning_with_multiple_scaling_options"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='red'>Danger Zone! Tread carefully beyond this point!</font>\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is an example in case you also need to optimize the cluster size.\n",
    "It will loop over a list of the required instance count you wish to test, and will spin a tuning job for each cluster size.  \n",
    "<font color='red'>Please note</font>: this process may have a significant impact on account cost and limits because each tuner will launch jobs with multiple instances!  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# max_instances = 5\n",
    "# tuning_jobs=[]\n",
    "\n",
    "# import time\n",
    "# sm=boto3.client(\"sagemaker\")\n",
    "\n",
    "# def check_tuning_status(tuner_to_check):\n",
    "#     status=sm.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuner_to_check.latest_tuning_job.job_name)[\"HyperParameterTuningJobStatus\"]\n",
    "#     return status\n",
    "\n",
    "\n",
    "# In this example, we will test 3,4 and 5 instances\n",
    "# for num_of_instances in range(3, max_instances+1):\n",
    "#     xgb_estimator=get_XGB_estimnator(num_of_instances) # create the estimator\n",
    "#     tuner = HyperparameterTuner(\n",
    "#         xgb_estimator, objective_metric_name, hyperparameter_ranges, max_jobs=20, max_parallel_jobs=3,strategy=\"Hyperband\",early_stopping_type='Off',strategy_config=sc\n",
    "#     ) # Launch th tuning job\n",
    "\n",
    "#     completed=False\n",
    "#     while not completed: #check for completion\n",
    "#         time.sleep(60) # check every minute\n",
    "#         stat=check_tuning_status(tuner)\n",
    "#         if stat == 'Completed':\n",
    "#             completed = True\n",
    "#     tuning_jobs.append(tuner)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Compare all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# tuning_results= []\n",
    "# for tuning_job in tuning_jobs:\n",
    "#     tuning_job_name=tuning_job.describe()['HyperParameterTuningJobName']\n",
    "#     tuner = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
    "#     tuning_job_df = tuner.dataframe()\n",
    "#     tuning_results.append(tuning_job_df)\n",
    "\n",
    "# all_df=tuning_results[0]\n",
    "# for i in (1,len(tuning_results)):\n",
    "#     all_df.append(tuning_results[i], ignore_index=True)\n",
    "\n",
    "# # get top jobs sorted by: accuraccy and speed\n",
    "# all_df.sort_values(by=['FinalObjectiveValue'], ascending=False).sort_values(by='TrainingElapsedTimeSeconds', ascending=True).head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/build_and_train_models|sm-hyperparameter_tuning_hyperband_automatic_model_tuning.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}