      build_and_train_models/sm-hpo_warmstart_image_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic Model Tuning : Warm Starting Tuning Jobs\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_** Using Warm Start to tune End-to-End Multiclass Image Classification **_\n",
    "\n",
    "---\n",
    "## Important notes:\n",
    "* 3 Hyperparameter tuning jobs will be created in this sample notebook. With current setting, each tuning job takes 30-40 minutes to complete and may cost you up to $3 depending on which region you are in. \n",
    "* Due to cost consideration, the goal of this exmaple is to show you how to use some of the features, not necessarily to achieve the best result.\n",
    "* We use the built-in image classification algorithm in this example, and GPU instance is required. \n",
    "* Please do not use Cell -> Run All since you'll need to wait for the parent tuning job to finish before the new tuning job can be created with warm start.\n",
    "\n",
    "---\n",
    "## Contents\n",
    "\n",
    "1. [Background](#Background)\n",
    "1. [Set_up](#Set-up)\n",
    "1. [Data_preparation](#Data-preparation)\n",
    "1. [Set_up_hyperparameter_tuning_job](#Set-up-hyperparameter-tuning-job)\n",
    "1. [Launch_hyperparameter_tuning_job](#Launch-hyperparameter-tuning-job)\n",
    "1. [Set_up_hyperparameter_tuning_using_warm_start_configuration](#Set-up-hyperparameter-tuning-job-using-warm-start-configuration)\n",
    "1. [Launch_hyperparameter_tuning_job_using_warm_start_configuration](#Launch-hyperparameter-tuning-job-using-warm-start-configuration)\n",
    "1. [Get_the_best_model](#Get-the-best-model)\n",
    "1. [Launch_hyperparameter_tuning_job_using_warm_start_configuration_with_transfer_learning](#Launch-hyperparameter-tuning-job-using-warm-start-configuration-with-transfer-learning)\n",
    "2. [Wrap_up](#Wrap-up)\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Background\n",
    "\n",
    "Selecting the right hyperparameter values for your machine learning model can be difficult. The right answer is dependent on your data; some algorithms have many different hyperparameters that can be tweaked; some are very sensitive to the hyperparameter values selected; and most have a non-linear relationship between model fit and hyperparameter values. \n",
    "\n",
    "Amazon SageMaker Automatic Model Tuning helps with automating the hyperparameter tuning process. In many occasions the tuning process is iterative and requires to run multiple tuning jobs after analyzing the results to get the best objective metric.\n",
    "\n",
    "This notebook will demonstrate how to iteratively tune an image classifer leveraging the warm start feature of Amazon SageMaker Automatic Model Tuning. The [Caltech-256 dataset](https://paperswithcode.com/dataset/caltech-256) will be used to train the image classifier. \n",
    "\n",
    "Warm start configuration allows you to create a new tuning job with the learning gathered in a parent tuning job by specifying up to 5 parent tuning jobs. If a warm start configuration is specified, Automatic Model Tuning will load the previous [hyperparameter set, objective metrics values] to warm start the new tuning job. This means, you can continue optimizing your model from the point you finished your previous tuning job experiment. \n",
    "\n",
    "---\n",
    "\n",
    "## Set up\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for your training and model data. This should be within the same region as SageMaker training.\n",
    "- The IAM role used to give training access to your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "import boto3\n",
    "\n",
    "role = get_execution_role()\n",
    "print(role)\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "default_bucket_prefix = sess.default_bucket_prefix\n",
    "prefix = \"ic-fulltraining\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    prefix = f\"{default_bucket_prefix}/{prefix}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.image_uris import retrieve\n",
    "\n",
    "training_image = retrieve(\n",
    "    region=sess.boto_region_name, framework=\"image-classification\", version=\"1\"\n",
    ")\n",
    "print(training_image)\n",
    "\n",
    "smclient = boto3.Session().client(\"sagemaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "Download the data and transfer to S3 for use in training. In this example, we are using Caltech-256 dataset, which contains 30608 images of 256 objects. For the training and validation data, we follow the splitting scheme in this MXNet [example](https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/data/caltech256.sh). In particular, it randomly selects 60 images per class for training, and uses the remaining data for validation. The algorithm takes `RecordIO` files as input. The user can also provide the image files as input, which will be converted into `RecordIO` format using MXNet's [im2rec](https://mxnet.incubator.apache.org/how_to/recordio.html?highlight=im2rec) tool. It takes around 50 seconds to converted the entire Caltech-256 dataset (~1.2GB) on a p2.xlarge instance. However, for this example, we will directly use recordio format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "\n",
    "def download(url):\n",
    "    filename = url.split(\"/\")[-1]\n",
    "    if not os.path.exists(filename):\n",
    "        urllib.request.urlretrieve(url, filename)\n",
    "\n",
    "\n",
    "# caltech-256\n",
    "download(\"http://data.mxnet.io/data/caltech-256/caltech-256-60-train.rec\")\n",
    "download(\"http://data.mxnet.io/data/caltech-256/caltech-256-60-val.rec\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Four channels: train, validation, train_lst, and validation_lst\n",
    "s3train = \"s3://{}/{}/train/\".format(bucket, prefix)\n",
    "s3validation = \"s3://{}/{}/validation/\".format(bucket, prefix)\n",
    "\n",
    "# upload the lst files to train and validation channels\n",
    "!aws s3 cp caltech-256-60-train.rec $s3train --quiet\n",
    "!aws s3 cp caltech-256-60-val.rec $s3validation --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the data type and channels used for training\n",
    "s3_output_location = \"s3://{}/{}/output\".format(bucket, prefix)\n",
    "s3_input_train = sagemaker.inputs.TrainingInput(\n",
    "    s3train,\n",
    "    distribution=\"FullyReplicated\",\n",
    "    content_type=\"application/x-recordio\",\n",
    "    s3_data_type=\"S3Prefix\",\n",
    ")\n",
    "s3_input_validation = sagemaker.inputs.TrainingInput(\n",
    "    s3validation,\n",
    "    distribution=\"FullyReplicated\",\n",
    "    content_type=\"application/x-recordio\",\n",
    "    s3_data_type=\"S3Prefix\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up hyperparameter tuning job\n",
    "\n",
    "Now that we have prepared the dataset, we are ready to train models. \n",
    "\n",
    "For this example we will use image classification with Stochastic gradient descent (sgd) optimizer and we will tune learning_rate, weight_decay and momentum hyperparameters. Find [here](https://docs.aws.amazon.com/sagemaker/latest/dg/IC-tuning.html) the full list of hyperparameters that can be tuned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can launch the tuning job, we need to configure the training jobs the hyperparameter tuning job will launch by defining an estimator that specifies the following information:\n",
    "* The container image for the algorithm (image-classification)\n",
    "* The type and number of instances to use for the training jobs\n",
    "* The stopping condition for the training jobs\n",
    "* The values of any algorithm hyperparameters that are not tuned in the tuning job (StaticHyperparameters)\n",
    " * **num_layers**: The number of layers (depth) for the network. We use 18 in this samples but other values such as 50, 152 can be used.\n",
    " * **image_shape**: The input image dimensions,'num_channels, height, width', for the network. It should be no larger than the actual image size. The number of channels should be same as in the actual image.\n",
    " * **num_classes**: This is the number of output classes for the new dataset. For caltech, we use 257 because it has 256 object categories + 1 clutter class.\n",
    " * **num_training_samples**: This is the total number of training samples. It is set to 15240 for caltech dataset with the current split.\n",
    " * **mini_batch_size**: The number of training samples used for each mini batch. In distributed training, the number of training samples used per batch will be N * mini_batch_size where N is the number of hosts on which training is run.\n",
    " * **epochs**: Number of training epochs. In this example we set it to only 10 to save the cost. If you would like to get higher accuracy the number of epochs can be increased.\n",
    " * **optimizer**: \"sgd\" Stochastic gradient descent\n",
    " * **top_k**: Report the top-k accuracy during training.\n",
    " * **precision_dtype**: Training datatype precision (default: float32). If set to 'float16', the training will be done in mixed_precision mode and will be faster than float32 mode\n",
    " * **augmentation_type**: crop. Randomly crop the image and flip the image horizontally\n",
    " \n",
    "**Note: you should explicitly add any hyperparameter with the default value to the set of static hyperparameters instead of omitting it. This is important to be able to tune this hyperparameter using warm start in subsequent tuning job that uses the results of this tuning job as a starting point.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = sagemaker.Session()\n",
    "\n",
    "imageclassification = sagemaker.estimator.Estimator(\n",
    "    training_image,\n",
    "    role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.p3.2xlarge\",\n",
    "    output_path=s3_output_location,\n",
    "    sagemaker_session=sess,\n",
    ")\n",
    "\n",
    "imageclassification.set_hyperparameters(\n",
    "    num_layers=18,\n",
    "    image_shape=\"3,224,224\",\n",
    "    num_classes=257,\n",
    "    num_training_samples=15420,\n",
    "    mini_batch_size=128,\n",
    "    epochs=10,\n",
    "    optimizer=\"sgd\",\n",
    "    top_k=\"2\",\n",
    "    precision_dtype=\"float32\",\n",
    "    augmentation_type=\"crop\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we set up the tuning job with the following configuration:\n",
    "* the hyperparameters that SageMaker Automatic Model Tuning will tune: learning_rate, momentum and weight_decay\n",
    "* the maximum number of training jobs it will run to optimize the objective metric: 5\n",
    "* the number of parallel training jobs that will run in the tuning job: 2\n",
    "* the objective metric that Automatic Model Tuning will use: validation:accuracy\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import (\n",
    "    IntegerParameter,\n",
    "    CategoricalParameter,\n",
    "    ContinuousParameter,\n",
    "    HyperparameterTuner,\n",
    ")\n",
    "\n",
    "hyperparameter_ranges = {\n",
    "    \"learning_rate\": ContinuousParameter(0.0001, 0.05),\n",
    "    \"momentum\": ContinuousParameter(0.0, 0.99),\n",
    "    \"weight_decay\": ContinuousParameter(0.0, 0.99),\n",
    "}\n",
    "\n",
    "objective_metric_name = \"validation:accuracy\"\n",
    "\n",
    "tuner = HyperparameterTuner(\n",
    "    imageclassification,\n",
    "    objective_metric_name,\n",
    "    hyperparameter_ranges,\n",
    "    objective_type=\"Maximize\",\n",
    "    max_jobs=5,\n",
    "    max_parallel_jobs=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch hyperparameter tuning job\n",
    "Now we can launch a hyperparameter tuning job by calling fit in tuner. After the hyperparameter tuning job is created, we can go to SageMaker console to track the progress of the hyperparameter tuning job until it is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner.fit({\"train\": s3_input_train, \"validation\": s3_input_validation}, include_cls_metadata=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_You will be unable to successfully run the following cells until the tuning job completes._\n",
    "\n",
    "Once the tuning job finishes, we can bring in a table of metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "tuning_job_name = tuner._current_job_name\n",
    "\n",
    "tuner_parent_metrics = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
    "if not tuner_parent_metrics.dataframe().empty:\n",
    "    df_parent = tuner_parent_metrics.dataframe().sort_values(\n",
    "        [\"FinalObjectiveValue\"], ascending=False\n",
    "    )\n",
    "\n",
    "df_parent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can analyze the results deeper by using HPO_Analyze_TuningJob_Results.ipynb notebook. Here, we will just plot how the objective metric changes overtime as the tuning progresses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bokeh\n",
    "import bokeh.io\n",
    "\n",
    "bokeh.io.output_notebook()\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.models import HoverTool\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "df_parent_objective_value = df_parent[df_parent[\"FinalObjectiveValue\"] > -float(\"inf\")]\n",
    "\n",
    "p = figure(\n",
    "    plot_width=900,\n",
    "    plot_height=400,\n",
    "    x_axis_type=\"datetime\",\n",
    "    x_axis_label=\"datetime\",\n",
    "    y_axis_label=objective_metric_name,\n",
    ")\n",
    "p.circle(\n",
    "    source=df_parent_objective_value, x=\"TrainingStartTime\", y=\"FinalObjectiveValue\", color=\"black\"\n",
    ")\n",
    "\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on how your first hyperparameter tuning job went, you may or may not want to try another tuning job to see whether the model quality can be further improved. When you decide to run another tuning job, you would want to leverage what has been known about the search space from the completed tuning job. In that case, you can create a new hyperparameter tuning job, while warm starting it using the completed tuning job, instead of starting from scratch.\n",
    "\n",
    "To show you how to use warm start, next we will run a second tuning job and enable warm start."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up hyperparameter tuning using warm start configuration\n",
    "\n",
    "To use warm start in the new tuning job, we need to specify 2 parameters: \n",
    "* the list of parent tuning jobs the new tuning job should use as a starting point (The maximum number of parents can be 5).\n",
    "* the type of warm start configuration:\n",
    " * ‘IDENTICAL_DATA_AND_ALGORITHM’ warm starts a tuning job with previous evaluations essentially with the same task, allowing slightly change in the search space. This option should be use when the data set and the algorithm container haven't changed. In this scenario, the only changes to the docker image we recommend are those that do not affect the algorithm, for example changes that only improve logging, or add support of a different data format.\n",
    " * ‘TRANSFER_LEARNING’ warm starts a tuning job with the evaluations from similar tasks, allowing both search space, algorithm image and dataset change. \n",
    "\n",
    "In this example we will use 'IDENTICAL_DATA_AND_ALGORITHM' because we are not changing the data set or algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import WarmStartConfig, WarmStartTypes\n",
    "\n",
    "parent_tuning_job_name = tuning_job_name\n",
    "warm_start_config = WarmStartConfig(\n",
    "    WarmStartTypes.IDENTICAL_DATA_AND_ALGORITHM, parents={parent_tuning_job_name}\n",
    ")\n",
    "\n",
    "parent_tuning_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner_warm_start = HyperparameterTuner(\n",
    "    imageclassification,\n",
    "    objective_metric_name,\n",
    "    hyperparameter_ranges,\n",
    "    objective_type=\"Maximize\",\n",
    "    max_jobs=5,\n",
    "    max_parallel_jobs=2,\n",
    "    base_tuning_job_name=\"warmstart\",\n",
    "    warm_start_config=warm_start_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch hyperparameter tuning job using warm start configuration\n",
    "Now we can launch a hyperparameter tuning job by calling tuner.fit and passing warmStartConfig. After the hyperparameter tuning job is created, we can go to SageMaker console to track the progress of the hyperparameter tuning job until it is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner_warm_start.fit(\n",
    "    {\"train\": s3_input_train, \"validation\": s3_input_validation}, include_cls_metadata=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_You will be unable to successfully run the following cells until the tuning job completes._\n",
    "\n",
    "Once the tuning job finishes, we can bring in a table of metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "warmstart_tuning_job_name = tuner_warm_start._current_job_name\n",
    "\n",
    "tuner_warm_start_metrics = sagemaker.HyperparameterTuningJobAnalytics(warmstart_tuning_job_name)\n",
    "if not tuner_warm_start_metrics.dataframe().empty:\n",
    "    df_warm_start = tuner_warm_start_metrics.dataframe().sort_values(\n",
    "        [\"FinalObjectiveValue\"], ascending=False\n",
    "    )\n",
    "\n",
    "df_warm_start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then plot the objective metrics for the parent job and the current job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bokeh\n",
    "import bokeh.io\n",
    "\n",
    "bokeh.io.output_notebook()\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.models import HoverTool\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "df_parent_objective_value = df_parent[df_parent[\"FinalObjectiveValue\"] > -float(\"inf\")]\n",
    "df_warm_start_objective_value = df_warm_start[df_warm_start[\"FinalObjectiveValue\"] > -float(\"inf\")]\n",
    "\n",
    "p = figure(\n",
    "    plot_width=900,\n",
    "    plot_height=400,\n",
    "    x_axis_type=\"datetime\",\n",
    "    x_axis_label=\"datetime\",\n",
    "    y_axis_label=objective_metric_name,\n",
    ")\n",
    "p.circle(\n",
    "    source=df_parent_objective_value, x=\"TrainingStartTime\", y=\"FinalObjectiveValue\", color=\"black\"\n",
    ")\n",
    "p.circle(\n",
    "    source=df_warm_start_objective_value,\n",
    "    x=\"TrainingStartTime\",\n",
    "    y=\"FinalObjectiveValue\",\n",
    "    color=\"red\",\n",
    ")\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the best model\n",
    "\n",
    "When the job completes, if you are satisfy with the results, you can find the training job that generated the best model by using OverallBestTrainingJob in Automatic Model Tuning describe API. Please note OverallBestTrainingJob may be from the latest hyperparameter tuning job or one of its parent jobs, when 'IdenticalDataAndAlgorithm' warm start type is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_overall_training_job = smclient.describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=warmstart_tuning_job_name\n",
    ")[\"OverallBestTrainingJob\"]\n",
    "\n",
    "best_overall_training_job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch hyperparameter tuning job using warm start configuration with transfer learning\n",
    "\n",
    "Finally, we are going to apply some more data augmentation to the data set to teach the invariance of the same image to our model training and tuning. It is base on the assumption that, for the same object, photos under different composition, lighting condition, or color should all yield the same prediction.\n",
    "\n",
    "To create our last hyperparameter tuning job, we will use 'Transfer learning' warm start type since our data set is going to change due to new data augmentations. We will use both of the 2 previous tuning jobs we ran as parent tuning jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import WarmStartConfig, WarmStartTypes\n",
    "\n",
    "parent_tuning_job_name_2 = warmstart_tuning_job_name\n",
    "transfer_learning_config = WarmStartConfig(\n",
    "    WarmStartTypes.TRANSFER_LEARNING, parents={parent_tuning_job_name, parent_tuning_job_name_2}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To apply more data augmentations we can use ‘augmentation_type’ hyperparameter exposed by the algorithm. We will apply 'crop_color_transform' transformation to the data set during training. With this transformation, in addition to crop and color transformations, random transformations (including rotation, shear, and aspect ratio variations) are applied to the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imageclassification.set_hyperparameters(\n",
    "    num_layers=18,\n",
    "    image_shape=\"3,224,224\",\n",
    "    num_classes=257,\n",
    "    num_training_samples=15420,\n",
    "    mini_batch_size=128,\n",
    "    epochs=10,\n",
    "    optimizer=\"sgd\",\n",
    "    top_k=\"2\",\n",
    "    precision_dtype=\"float32\",\n",
    "    augmentation_type=\"crop_color_transform\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner_transfer_learning = HyperparameterTuner(\n",
    "    imageclassification,\n",
    "    objective_metric_name,\n",
    "    hyperparameter_ranges,\n",
    "    objective_type=\"Maximize\",\n",
    "    max_jobs=5,\n",
    "    max_parallel_jobs=2,\n",
    "    base_tuning_job_name=\"transferlearning\",\n",
    "    warm_start_config=transfer_learning_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner_transfer_learning.fit(\n",
    "    {\"train\": s3_input_train, \"validation\": s3_input_validation}, include_cls_metadata=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will be unable to successfully run the following cells until the tuning job completes.\n",
    "\n",
    "Once the tuning job finishes, we can bring in a table of metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transferlearning_tuning_job_name = tuner_transfer_learning._current_job_name\n",
    "\n",
    "tuner_transferlearning_metrics = sagemaker.HyperparameterTuningJobAnalytics(\n",
    "    transferlearning_tuning_job_name\n",
    ")\n",
    "if not tuner_transferlearning_metrics.dataframe().empty:\n",
    "    df_transfer_learning = tuner_transferlearning_metrics.dataframe().sort_values(\n",
    "        [\"FinalObjectiveValue\"], ascending=False\n",
    "    )\n",
    "\n",
    "df_transfer_learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then plot the objective metrics for the two parent jobs and the current job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import bokeh\n",
    "import bokeh.io\n",
    "\n",
    "bokeh.io.output_notebook()\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.models import HoverTool\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "df_parent_objective_value = df_parent[df_parent[\"FinalObjectiveValue\"] > -float(\"inf\")]\n",
    "df_warm_start_objective_value = df_warm_start[df_warm_start[\"FinalObjectiveValue\"] > -float(\"inf\")]\n",
    "df_transfer_learning_objective_value = df_transfer_learning[\n",
    "    df_transfer_learning[\"FinalObjectiveValue\"] > -float(\"inf\")\n",
    "]\n",
    "\n",
    "p = figure(\n",
    "    plot_width=900,\n",
    "    plot_height=400,\n",
    "    x_axis_type=\"datetime\",\n",
    "    x_axis_label=\"datetime\",\n",
    "    y_axis_label=objective_metric_name,\n",
    ")\n",
    "p.circle(\n",
    "    source=df_parent_objective_value, x=\"TrainingStartTime\", y=\"FinalObjectiveValue\", color=\"black\"\n",
    ")\n",
    "p.circle(\n",
    "    source=df_warm_start_objective_value,\n",
    "    x=\"TrainingStartTime\",\n",
    "    y=\"FinalObjectiveValue\",\n",
    "    color=\"red\",\n",
    ")\n",
    "p.circle(\n",
    "    source=df_transfer_learning_objective_value,\n",
    "    x=\"TrainingStartTime\",\n",
    "    y=\"FinalObjectiveValue\",\n",
    "    color=\"blue\",\n",
    ")\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have got the best model, we can deploy it to an endpoint. Please refer to other SageMaker sample notebooks or SageMaker documentation to see how to deploy a model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we demonstrated how to use warm start to iteratively tune your models.  Warm Start could also be used for other scenarios like: tuning additional hyperparameters, running smaller training jobs (e.g., smaller dataset or fewer epochs) in first tuning job to quickly explore search space then running second tuning job with full size training, or re-tuning a model as you've collected new data over time.\n",
    "\n",
    "For more information on using SageMaker's Automatic Model Tuning, see our other [example notebooks](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/hyperparameter_tuning) and [documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/build_and_train_models|sm-hpo_warmstart_image_classification.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Data Science 2.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/sagemaker-data-science-38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}