 archived/Amazon_JumpStart_Image_Classification_Benchmarking.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7e9f377d",
   "metadata": {},
   "source": [
    "# JumpStart - Image Classification Benchmarking and Model Selection"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "562e863c",
   "metadata": {},
   "source": [
    "***\n",
    "Welcome to Amazon [SageMaker JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html)! You can use JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through [SageMaker JumpStart API](https://sagemaker.readthedocs.io/en/stable/overview.html#use-prebuilt-models-with-sagemaker-jumpstart). \n",
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "95e01626-75c8-40a7-a336-a7e4d061e98c",
   "metadata": {},
   "source": [
    "This demo notebook demonstrates how to use the SageMaker JumpStart API to perform large-scale image classification model selection or benchmarking tasks. The SageMaker JumpStart model hub provides access to many image classification models that enable transfer learning and fine-tuning on custom datasets. Many types of models have been developed to accomplish the image classification task, and the SageMaker JumpStart model hub now contains many of these popular model architectures, to include \n",
    "Residual Networks ([ResNet](https://arxiv.org/abs/1512.03385)), \n",
    "[MobileNet](https://arxiv.org/abs/1704.04861), \n",
    "[EfficientNet](https://arxiv.org/abs/1905.11946), \n",
    "[Inception](https://arxiv.org/abs/1409.4842), \n",
    "Neural Architecture Search Networks ([NASNet](https://arxiv.org/abs/1707.07012)), \n",
    "Big Transfer ([BiT](https://arxiv.org/abs/1912.11370)), \n",
    "Shifted Window ([Swin](https://arxiv.org/abs/2103.14030)) Transformers,\n",
    "Class-Attention in Image Transformers ([CaiT](https://arxiv.org/abs/2103.17239)), and \n",
    "Data-Efficient Image Transformers ([DeiT](https://arxiv.org/pdf/2012.12877.pdf)).\n",
    "Vastly different internal structures comprise each model architectures. For instance, ResNet models utilize skip connections to allow for substantially deeper networks while transformer-based models use self-attention mechanisms that eliminate the intrinsic locality of convolution operations in favor of more global receptive fields.\n",
    "In addition to the diverse feature sets that each of these different structures provide, each model architecture has several model configurations that adjust the model size, shape, and complexity within that architecture. This results in hundreds of unique image classification models available on the SageMaker JumpStart model hub.\n",
    "\n",
    "A machine learning practitioner may therefore ask questions like: \"what model should I deploy to achieve the best performance on my dataset?\" And a machine learning researcher may ask questions like: \"how can I produce my own fair comparison of multiple model architectures against a specified dataset while controlling training hyperparameters and machine architectures?\" The former question addresses _model selection_ across model architectures while the latter question concerns _benchmarking_ trained models against a test dataset.\n",
    "\n",
    "This notebook demonstrates a methodology to perform either model selection or benchmarking tasks with the SageMaker JumpStart API by asynchronously launching [SageMaker Automatic Model Tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html) jobs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7037d554",
   "metadata": {},
   "source": [
    "***\n",
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Identify models and datasets](#2.-Identify-models-and-datasets)\n",
    "3. [Create helper functions for training](#3.-Create-helper-functions-for-training)\n",
    "4. [Asynchronously execute hyperparameter tuning jobs](#4.-Asynchronously-execute-hyperparameter-tuning-jobs)\n",
    "5. [Analyze results](#5.-Analyze-results)\n",
    "***"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cf570f82",
   "metadata": {},
   "source": [
    "## 1. Set Up\n",
    "***\n",
    "First, perform necessary imports.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b0dfd09",
   "metadata": {},
   "outputs": [],
   "source": [
    "import concurrent.futures as cf\n",
    "import json\n",
    "import itertools\n",
    "import queue\n",
    "import time\n",
    "import traceback\n",
    "from pathlib import Path\n",
    "from pprint import pprint\n",
    "from typing import Any, Callable, Dict, List, NamedTuple, Optional, Tuple\n",
    "\n",
    "import boto3\n",
    "import pandas as pd\n",
    "import sagemaker\n",
    "import sagemaker.hyperparameters\n",
    "import sagemaker.model_uris\n",
    "import sagemaker.script_uris\n",
    "import sagemaker.image_uris\n",
    "from botocore.config import Config\n",
    "from sagemaker.tuner import CategoricalParameter"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "355786c5",
   "metadata": {},
   "source": [
    "***\n",
    "Now you can identify the top-level constants and adjustable parameters that will be utilized throughout this notebook. These values are gathered at the top of this notebook, so commonly needed adjustments can be made in one place.\n",
    "\n",
    "The first set of constants control SageMaker training job behaviors:\n",
    "- __SM_TRAINING_INSTANCE_TYPE__: EC2 instance type used for training. The default here is set to `ml.p3.2xlarge`. If your [SageMaker service quota](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html) does not support this many simultaneous training jobs, please do one of the following: 1) request an increase to your SageMaker service quota, 2) adjust the constants `SM_AMT_MAX_PARALLEL_TRAINING_JOBS_PER_TUNER` and `SM_AMT_MAX_PARALLEL_TUNING_JOBS` to comply with your SageMaker service quota, or 3) change this instance type to an instance with a larger default SageMaker service quota, such as `ml.m5.4xlarge`.\n",
    "- __SM_AMT_MAX_JOBS__: Maximum total number of training jobs to start per hyperparameter tuning job.\n",
    "- __SM_AMT_MAX_PARALLEL_TRAINING_JOBS_PER_TUNER__: Maximum number of parallel training jobs per hyperparameter tuning job.\n",
    "- __SM_AMT_MAX_PARALLEL_TUNING_JOBS__: Maximum number of parallel hyperparameter tuning jobs. The resulting maximum number of parallel _training_ jobs is `SM_AMT_MAX_PARALLEL_TRAINING_JOBS_PER_TUNER * SM_AMT_MAX_PARALLEL_TUNING_JOBS`. Please ensure this quantity is less than your SageMaker Training service quota for the specified instance type in your current region.\n",
    "- __SM_AMT_OBJECTIVE_METRIC_NAME__: Name of the metric for evaluating training jobs during model selection. This value is used for early stopping criteria and hyperparameter tuning model selection.\n",
    "- __SM_AMT_OBJECTIVE_TYPE__: The type of the objective metric for evaluating training jobs. This value can be either \u2018Minimize\u2019 or \u2018Maximize\u2019.\n",
    "- __SM_SESSION__: SageMaker Session object with custom configuration to resolve [SDK rate exceeded and throttling exceptions](https://aws.amazon.com/premiumsupport/knowledge-center/sagemaker-python-throttlingexception/).\n",
    "- __SM_AMT_HYPERPARAMETER_RANGE_LAMBDA_DICT__: Dictionary of callables that accept a hyperparameter dictionary and return a `sagemaker.parameter.ParameterRange`. These parameter ranges can be one of three types: Continuous, Integer, or Categorical. The keys of the dictionary are the names of the hyperparameter, and the values are callables that produce the appropriate parameter range class to represent the range given a single hyperparameter dictionary argument. This allows AMT hyperparameter range defitions to depend on default model hyperparameter values.\n",
    "- __SM_TRAINING_METRIC_DEFINITIONS__: A list of dictionaries that defines the metric(s) used to evaluate the training jobs. Each dictionary contains two keys: \u2018Name\u2019 for the name of the metric, and \u2018Regex\u2019 for the regular expression used to extract the metric from the logs. SageMaker JumpStart provides many pre-implemented extractable metrics.\n",
    "\n",
    "The next set of constants control the behavior of the training script:\n",
    "- __HYPERPARAMETERS__: Set of hyperparameters overriding any [default built-in value](https://docs.aws.amazon.com/sagemaker/latest/dg/IC-TF-Hyperparameter.html).\n",
    "\n",
    "Finally, this notebook provides features to re-attach previously launched training jobs and load previously saved metrics for further analysis. The following constants control this behavior:\n",
    "- __SAVE_TUNING_JOB_NAMES_FILE_PATH__: Path of the [JSON Lines](https://jsonlines.org/) file that keeps track of the tuning job name associated with a unique model name and dataset name.\n",
    "- __SAVE_METRICS_FILE_PATH__: Path of the JSON Lines file that records metrics associated with each tuning job.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52791961",
   "metadata": {},
   "outputs": [],
   "source": [
    "SM_TRAINING_INSTANCE_TYPE = \"ml.p3.2xlarge\"\n",
    "SM_AMT_MAX_JOBS = 2\n",
    "SM_AMT_MAX_PARALLEL_TRAINING_JOBS_PER_TUNER = 2\n",
    "SM_AMT_MAX_PARALLEL_TUNING_JOBS = 8\n",
    "SM_AMT_OBJECTIVE_METRIC_NAME = \"val_accuracy\"\n",
    "SM_AMT_OBJECTIVE_TYPE = \"Maximize\"\n",
    "SM_SESSION = sagemaker.Session(\n",
    "    sagemaker_client=boto3.client(\n",
    "        \"sagemaker\",\n",
    "        config=Config(connect_timeout=5, read_timeout=60, retries={\"max_attempts\": 20}),\n",
    "    )\n",
    ")\n",
    "SM_AMT_HYPERPARAMETER_RANGE_LAMBDA_DICT = {\n",
    "    \"learning_rate\": lambda x: CategoricalParameter(\n",
    "        [float(x[\"learning_rate\"]), float(x[\"learning_rate\"]) / 5]\n",
    "    )\n",
    "}\n",
    "_METRICS_MULTICLASS = (\"top_5_accuracy\",)\n",
    "_METRICS_BINARY = (\"precision\", \"recall\", \"auc\", \"prc\")\n",
    "_METRICS = (\"accuracy\", \"loss\", *_METRICS_MULTICLASS, *_METRICS_BINARY)\n",
    "_NAME, _RE = \"Name\", \"Regex\"\n",
    "SM_TRAINING_METRIC_DEFINITIONS = [\n",
    "    *({_NAME: f\"train_{metric}\", _RE: f\"- {metric}: ([0-9\\\\.]+)\"} for metric in _METRICS),\n",
    "    *({_NAME: f\"val_{metric}\", _RE: f\"- val_{metric}: ([0-9\\\\.]+)\"} for metric in _METRICS),\n",
    "    *({_NAME: f\"test_{metric}\", _RE: f\"- Test {metric}: ([0-9\\\\.]+)\"} for metric in _METRICS),\n",
    "    {_NAME: \"num_params\", _RE: \"- Number of parameters: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"num_trainable_params\", _RE: \"- Number of trainable parameters: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"num_non_trainable_params\", _RE: \"- Number of non-trainable parameters: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"train_duration\", _RE: \"- Total training duration: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"train_epoch_duration\", _RE: \"- Average training duration per epoch: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"test_evaluation_latency\", _RE: \"- Test evaluation latency: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"test_latency_per_sample\", _RE: \"- Average test latency per sample: ([0-9\\\\.]+)\"},\n",
    "    {_NAME: \"test_throughput\", _RE: \"- Average test throughput: ([0-9\\\\.]+)\"},\n",
    "]\n",
    "\n",
    "HYPERPARAMETERS = {\n",
    "    \"epochs\": 5,\n",
    "    \"early_stopping\": \"True\",\n",
    "    \"early_stopping_patience\": 3,\n",
    "    \"early_stopping_min_delta\": 0.0,\n",
    "}\n",
    "\n",
    "SAVE_TUNING_JOB_NAMES_FILE_PATH = Path.cwd() / \"benchmarking_tuning_job_names.jsonl\"\n",
    "SAVE_METRICS_FILE_PATH = Path.cwd() / \"benchmarking_metrics.jsonl\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7d45e9",
   "metadata": {},
   "source": [
    "## 2. Identify models and datasets\n",
    "***\n",
    "In this section, you will define two tuples, `MODELS` and `DATASETS`, which contain unique identifiers for all models and all datasets you wish to perform this benchmarking task on. The hyperparameter tuning jobs instantiated will be the [Cartesian product](https://docs.python.org/3/library/itertools.html#itertools.product) between these two lists.\n",
    "\n",
    "***\n",
    "First, you will identify all built-in image classification model IDs to run this benchmarking task on. Because SageMaker JumpStart maintains many models for this task, the default code in this notebook identifies only a few models by model ID. You can run a thorough benchmarking or model selection analysis on all TensorFlow image classification models made available by SageMaker Built-In Algorithms via:\n",
    "```python\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "from sagemaker.jumpstart.filters import And\n",
    "\n",
    "filter_value = And(\"task == ic\", \"framework == tensorflow\")\n",
    "MODELS = list_jumpstart_models(filter=filter_value)\n",
    "```\n",
    "This may be desired if you have a unique dataset and would like to perform large-scale benchmarking or model selection tasks on your custom dataset. However, please be cautious as a benchmarking task with this many models will require the deployment of many resources.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1a12d8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "MODELS = (\n",
    "    \"tensorflow-ic-swin-base-patch4-window7-224\",\n",
    "    \"tensorflow-ic-efficientnet-v2-imagenet21k-ft1k-m\",\n",
    "    \"tensorflow-ic-imagenet-mobilenet-v3-large-100-224\",\n",
    "    \"tensorflow-ic-imagenet-inception-v3-classification-4\",\n",
    "    \"tensorflow-ic-resnet-50-classification-1\",\n",
    "    \"tensorflow-ic-imagenet-nasnet-mobile\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5291829f",
   "metadata": {},
   "source": [
    "***\n",
    "You also need to identify the datasets to perform benchmarking on. Unlike built-in models, an API to query available datasets on S3 does not exist. It is also likely that you may have your own dataset hosted on S3 that you would like to benchmark. The following data structures provide a consistent framework to define dataset locations for the scope of this notebook. This is important because the benchmarking task is most beneficial with a training/validation/test dataset split. While possible, it is not recommended to let the model transfer learning script perform this split. Fitting a SageMaker Estimator requires channel definitions, which these objects create automatically via the `S3DatasetSplit.channels` method.\n",
    "\n",
    "*Notes on dataset channel behaviors*: Training will utilize only the data provided in the \"training\" channel, model selection across hyperparameters and epochs will use data in the \"validation\" channel, and the final evaluation of model performance will be based on data provided in the \"test\" channel. If a \"test\" channel is not provided, then training should complete successfully, but metric definitions with a name matching the pattern \"test_\\*\" will not be available in the training job logs. If a \"validation\" channel is not provided, then the default behavior of the SageMaker JumpStart TensorFlow image classification algorithm is to perform a split of the \"training\" channel dataset into training and validation datasets.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22987387",
   "metadata": {},
   "outputs": [],
   "source": [
    "class S3Dataset:\n",
    "    def __init__(self, bucket: str, prefix: str) -> None:\n",
    "        self.bucket = bucket\n",
    "        self.prefix = prefix\n",
    "\n",
    "    def path(self) -> str:\n",
    "        return f\"s3://{self.bucket}/{self.prefix}\"\n",
    "\n",
    "\n",
    "class S3DatasetSplit:\n",
    "    def __init__(\n",
    "        self,\n",
    "        name: str,\n",
    "        train: S3Dataset,\n",
    "        validation: Optional[S3Dataset] = None,\n",
    "        test: Optional[S3Dataset] = None,\n",
    "        hyperparameters: Optional[Dict[str, str]] = None,\n",
    "    ) -> None:\n",
    "        self.name = name\n",
    "        self.train = train\n",
    "        self.validation = validation\n",
    "        self.test = test\n",
    "        self.hyperparameters = hyperparameters\n",
    "\n",
    "    @classmethod\n",
    "    def from_prefixes(\n",
    "        cls,\n",
    "        name: str,\n",
    "        bucket: str,\n",
    "        prefix_train: str,\n",
    "        prefix_validation: Optional[str] = None,\n",
    "        prefix_test: Optional[str] = None,\n",
    "        hyperparameters: Optional[Dict[str, str]] = None,\n",
    "    ) -> \"S3DatasetSplit\":\n",
    "        train = S3Dataset(bucket, prefix_train)\n",
    "        validation = S3Dataset(bucket, prefix_validation) if prefix_validation is not None else None\n",
    "        test = S3Dataset(bucket, prefix_test) if prefix_test is not None else None\n",
    "        return cls(name, train, validation, test, hyperparameters)\n",
    "\n",
    "    def channels(self) -> Dict[str, str]:\n",
    "        res = {\"training\": self.train.path()}\n",
    "        if self.validation is not None:\n",
    "            res[\"validation\"] = self.validation.path()\n",
    "        if self.test is not None:\n",
    "            res[\"test\"] = self.test.path()\n",
    "        return res"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb36df43-4c42-4b18-8837-4ae6256ba8d8",
   "metadata": {},
   "source": [
    "***\n",
    "Next, a dictionary of available datasets is created and one of these datasets is selected to perform analysis. To get a feel for the performance of different models with respect to different datasets, simply run this notebook for a different selected list of datasets! If you have your own dataset, just create a new entry that specifies the bucket along with prefixes for the train, validation, and test datasets. The dataset should be structured according to the [built-in algorithm training data input format](https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification-tensorflow.html).\n",
    "\n",
    "Two example datasets are populated. These are both small datasets and used here for demonstration purposes. One of these datasets, ants-and-bees, only has two class labels, ants and bees, thereby allowing you to train the models with a `binary_mode` hyperparameter set to True. With this setting, the SageMaker JumpStart built-in algorithm will create a model that returns a single probability number for the positive class and can use additional eval_metric options.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30f1eaf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_DICT = {\n",
    "    \"tf-flowers\": S3DatasetSplit.from_prefixes(\n",
    "        name=\"tf-flowers\",\n",
    "        bucket=f\"jumpstart-cache-prod-{SM_SESSION.boto_region_name}\",\n",
    "        prefix_train=\"training-datasets/cv/tf-flowers-split/train/\",\n",
    "        prefix_validation=\"training-datasets/cv/tf-flowers-split/val/\",\n",
    "        prefix_test=\"training-datasets/cv/tf-flowers-split/test/\",\n",
    "    ),\n",
    "    \"ants-and-bees\": S3DatasetSplit.from_prefixes(\n",
    "        name=\"ants-and-bees\",\n",
    "        bucket=f\"jumpstart-cache-prod-{SM_SESSION.boto_region_name}\",\n",
    "        prefix_train=\"training-datasets/cv/ants-and-bees-split/train/\",\n",
    "        prefix_validation=\"training-datasets/cv/ants-and-bees-split/val/\",\n",
    "        prefix_test=\"training-datasets/cv/ants-and-bees-split/test/\",\n",
    "        hyperparameters={\"binary_mode\": \"True\"},\n",
    "    ),\n",
    "}\n",
    "\n",
    "# This tuple specifies which datasets will be used\n",
    "DATASETS = (\n",
    "    \"tf-flowers\",\n",
    "    \"ants-and-bees\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc883f99",
   "metadata": {},
   "source": [
    "## 3. Create helper functions for training\n",
    "\n",
    "***\n",
    "This section contains a variety of helper functions that will be utilized for this SageMaker TensorFlow image classification benchmarking task, including functions to:\n",
    "1) create a SageMaker `Estimator` object from the JumpStart model hub\n",
    "2) create a SageMaker `HyperparameterTuner` for a specified model from the JumpStart model hub\n",
    "3) re-attach a SageMaker `HyperparameterTuner` if a tuning job has already started\n",
    "4) extract metrics from `Estimator` logs\n",
    "5) save tuning job information to file to enable re-attaching jobs in new sessions\n",
    "6) save resulting benchmarking metrics to file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e27cd272",
   "metadata": {},
   "source": [
    "***\n",
    "The following block contains a helper function to obtain a SageMaker `Estimator` for a given JumpStart built-in `model_id`. This includes obtaining the appropriate URIs for the training docker image, the training script tarball, and the pre-trained model tarball to further fine-tune. This retrieval is provided by the SageMaker JumpStart built-in algorithms and allows for the creation of a SageMaker `Estimator` instance directly from these URIs.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2064c01c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_jumpstart_estimator(\n",
    "    model_id: str,\n",
    "    role: str,\n",
    "    job_name: str,\n",
    "    s3_output_location: str,\n",
    "    model_version: str,\n",
    "    instance_type: str = SM_TRAINING_INSTANCE_TYPE,\n",
    "    metric_definitions: Optional[List[Dict[str, str]]] = None,\n",
    "    hyperparameters: Optional[Dict[str, str]] = None,\n",
    ") -> sagemaker.estimator.Estimator:\n",
    "    \"\"\"Obtain a SageMaker Estimator for a given model ID.\"\"\"\n",
    "\n",
    "    # Retrieve the docker image\n",
    "    train_image_uri = sagemaker.image_uris.retrieve(\n",
    "        region=None,\n",
    "        framework=None,\n",
    "        model_id=model_id,\n",
    "        model_version=model_version,\n",
    "        image_scope=\"training\",\n",
    "        instance_type=instance_type,\n",
    "    )\n",
    "\n",
    "    # Retrieve the training script\n",
    "    train_source_uri = sagemaker.script_uris.retrieve(\n",
    "        model_id=model_id, model_version=model_version, script_scope=\"training\"\n",
    "    )\n",
    "\n",
    "    # Retrieve the pre-trained model tarball to further fine-tune\n",
    "    train_model_uri = sagemaker.model_uris.retrieve(\n",
    "        model_id=model_id, model_version=model_version, model_scope=\"training\"\n",
    "    )\n",
    "\n",
    "    # Create and return SageMaker Estimator instance\n",
    "    return sagemaker.estimator.Estimator(\n",
    "        role=role,\n",
    "        image_uri=train_image_uri,\n",
    "        source_dir=train_source_uri,\n",
    "        model_uri=train_model_uri,\n",
    "        entry_point=\"transfer_learning.py\",\n",
    "        instance_count=1,\n",
    "        instance_type=instance_type,\n",
    "        max_run=360000,\n",
    "        hyperparameters=hyperparameters,\n",
    "        output_path=s3_output_location,\n",
    "        base_job_name=job_name,\n",
    "        metric_definitions=metric_definitions,\n",
    "        sagemaker_session=SM_SESSION,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57659785-5979-494f-babc-36ee8ab5ed9f",
   "metadata": {},
   "source": [
    "***\n",
    "While you now have a means to create a SageMaker `Estimator`, default hyperparameter values may not be sufficient for the considered task. Therefore, to obtain the best benchmarking results, you provide this `Estimator` to a SageMaker hyperparameter tuning job. The following function uses `create_jumpstart_estimator` to obtain a SageMaker `HyperparameterTuner` for a given JumpStart build-in model_id with properties for this benchmarking task. Because tuning jobs have a 32-character name length limit and this benchmarking task can create many tuning jobs with similar (or identical) names after truncation, a unique-id is provided for each model to enforce unique tuning job names.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caf1d189",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_benchmarking_tuner(\n",
    "    model_id: str,\n",
    "    unique_id: int,\n",
    "    session: sagemaker.session.Session = SM_SESSION,\n",
    "    model_version: str = \"*\",\n",
    "    hyperparameters: Optional[Dict[str, str]] = HYPERPARAMETERS,\n",
    "    dataset_hyperparameters: Optional[Dict[str, str]] = None,\n",
    "    hyperparameter_range_lambda_dict: Dict[str, Callable] = SM_AMT_HYPERPARAMETER_RANGE_LAMBDA_DICT,\n",
    "    metric_definitions: List[Dict[str, str]] = SM_TRAINING_METRIC_DEFINITIONS,\n",
    ") -> sagemaker.tuner.HyperparameterTuner:\n",
    "    \"\"\"Obtain a SageMaker HyperparameterTuner with properties for this benchmarking task.\n",
    "\n",
    "    A unique ID is helpful to distinguish names of benchmarking jobs.\n",
    "    \"\"\"\n",
    "    role = session.get_caller_identity_arn()\n",
    "    output_bucket = session.default_bucket()\n",
    "    output_prefix = \"jumpstart-example-tf-ic-benchmarking\"\n",
    "    job_name = sagemaker.utils.name_from_base(\n",
    "        f\"bm-{unique_id}-{model_id.replace('tensorflow-ic-', '')}\"\n",
    "    )\n",
    "    s3_output_location = f\"s3://{output_bucket}/{output_prefix}/output\"\n",
    "\n",
    "    _hyperparameters = sagemaker.hyperparameters.retrieve_default(\n",
    "        model_id=model_id, model_version=model_version\n",
    "    )\n",
    "    if dataset_hyperparameters is not None:\n",
    "        _hyperparameters.update(dataset_hyperparameters)\n",
    "    if hyperparameters is not None:\n",
    "        _hyperparameters.update(hyperparameters)\n",
    "\n",
    "    estimator = create_jumpstart_estimator(\n",
    "        model_id,\n",
    "        role,\n",
    "        job_name,\n",
    "        s3_output_location,\n",
    "        model_version=model_version,\n",
    "        metric_definitions=metric_definitions,\n",
    "        hyperparameters=_hyperparameters,\n",
    "    )\n",
    "\n",
    "    hyperparameter_ranges = {\n",
    "        name: func(_hyperparameters) for name, func in hyperparameter_range_lambda_dict.items()\n",
    "    }\n",
    "\n",
    "    tuner = sagemaker.tuner.HyperparameterTuner(\n",
    "        estimator,\n",
    "        SM_AMT_OBJECTIVE_METRIC_NAME,\n",
    "        hyperparameter_ranges,\n",
    "        metric_definitions,\n",
    "        max_jobs=SM_AMT_MAX_JOBS,\n",
    "        max_parallel_jobs=SM_AMT_MAX_PARALLEL_TRAINING_JOBS_PER_TUNER,\n",
    "        objective_type=SM_AMT_OBJECTIVE_TYPE,\n",
    "        base_tuning_job_name=job_name,\n",
    "    )\n",
    "    return tuner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fc3fa53",
   "metadata": {},
   "source": [
    "***\n",
    "With these helper functions established, you can now create a `HyperparameterTuner` object for each specified model. But what happens if there is an error or the kernel for this script is terminated? The hyperparameter tuning jobs would still run to completion, and you do would not want to re-launch these jobs in order to obtain our results. Therefore, you need yet another helper function that will either re-attach the hyperparameter tuning job if it exists or create a new one via `create_benchmarking_tuner`. To accomplish this, the JSON Lines file specified in `SAVE_TUNING_JOB_NAMES_FILE_PATH` is read and checked for whether a tuning job already exists for `model_name` and `dataset_name`. If it does exist, then the job is re-attached and returned. Otherwise, a new tuner is created and the `fit()` method is invoked with the `wait=False` argument and channels specified per the previously defined `S3DatasetSplit` object you used to store our dataset S3 location. You will have the thread wait for this job to complete later, but you first need to put this job information on the `queue_save_tuning_job` queue, which will indicate to the primary thread to append this job information to `SAVE_TUNING_JOB_NAMES_FILE_PATH`. Writing to this file needs to be done by the primary thread listener because multiple threads simultaneously writing to a file is not thread safe.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25ab06b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class JobInformation(NamedTuple):\n",
    "    model_name: str\n",
    "    dataset_name: str\n",
    "    tuning_job_name: Optional[str] = None\n",
    "\n",
    "\n",
    "def create_or_attach_tuner(\n",
    "    model_name: str,\n",
    "    dataset: S3DatasetSplit,\n",
    "    unique_id: int,\n",
    "    queue_save_tuning_job: queue.Queue,\n",
    "    tuning_job_names_file_path: Path = SAVE_TUNING_JOB_NAMES_FILE_PATH,\n",
    "    session: sagemaker.Session = SM_SESSION,\n",
    ") -> Tuple[sagemaker.tuner.HyperparameterTuner, JobInformation]:\n",
    "    if tuning_job_names_file_path.exists():\n",
    "        tuning_jobs_df = pd.read_json(tuning_job_names_file_path, lines=True).set_index(\n",
    "            [\"model_name\", \"dataset_name\"]\n",
    "        )\n",
    "        if (model_name, dataset.name) in tuning_jobs_df.index:\n",
    "            tuning_job_name = tuning_jobs_df.loc[(model_name, dataset.name), \"tuning_job_name\"]\n",
    "            tuner = sagemaker.tuner.HyperparameterTuner.attach(tuning_job_name, session)\n",
    "            job_information = JobInformation(model_name, dataset.name, tuning_job_name)\n",
    "            print(f\"> Re-attached previous SageMaker tuning job, {job_information}\")\n",
    "            return tuner, job_information\n",
    "\n",
    "    tuner = create_benchmarking_tuner(\n",
    "        model_name, unique_id, dataset_hyperparameters=dataset.hyperparameters\n",
    "    )\n",
    "    tuner.fit(dataset.channels(), wait=False)\n",
    "    tuning_job_name = tuner.latest_tuning_job.name\n",
    "    job_information = JobInformation(model_name, dataset.name, tuning_job_name)\n",
    "    queue_save_tuning_job.put(job_information)\n",
    "    print(f\"> Starting new SageMaker tuning job, {job_information}\")\n",
    "    return tuner, job_information"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fdc26ac0-0d8d-4a5e-a747-b2ca3125cc5a",
   "metadata": {},
   "source": [
    "***\n",
    "Once a tuning job is complete, you need to obtain a description of the best training job. While the objective metric for the hyperparameter tuner is easily obtained via the `HyperparameterTuner.analytics()` method, which returns a `HyperparameterTuningJobAnalytics` object, the additional auxiliary metrics provided to the original estimator are not extracted with this object. The following function will probe the training job description in order to extract all metrics of interest to this benchmarking scenario from the key `FinalMetricDataList`. It will also extract the hyperparameters utilized from the key `HyperParameters`, which may be useful in identifying specific information about the training job. You can look into the [SageMaker DescribeTrainingJob API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeTrainingJob.html) for any additional keys you would want to extract from Amazon CloudWatch.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eb86293",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_job_summary_from_logs(\n",
    "    tuner: sagemaker.tuner.HyperparameterTuner,\n",
    "    job_information: JobInformation,\n",
    "    session: sagemaker.Session = SM_SESSION,\n",
    ") -> Dict[str, Any]:\n",
    "    description = session.describe_training_job(tuner.best_training_job())\n",
    "    metrics = {\n",
    "        metric[\"MetricName\"]: metric[\"Value\"] for metric in description[\"FinalMetricDataList\"]\n",
    "    }\n",
    "    hyperparameters = description[\"HyperParameters\"]\n",
    "    return {**metrics, **hyperparameters, **job_information._asdict()}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ea5bbfff",
   "metadata": {},
   "source": [
    "***\n",
    "Next, you will define a function that runs a single tuning job. For a given `model_id`, this function will do three things: 1) obtain a `HyperparameterTuner` object for this model, 2) launch the hyperparameter tuning job and wait for the job to complete, and 3) extract the relevant metrics from Amazon CloudWatch Logs for this training job. Additionally, this function requests access to the `queue_currently_running` queue, which has a maximum capacity and will block without timeout until there is an available spot on the queue. This allows you to cap the number of simultaneously running hyperparameter tuning jobs.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dfd7a6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_tuner(\n",
    "    model_name: str,\n",
    "    dataset: S3DatasetSplit,\n",
    "    unique_id: int,\n",
    "    queue_currently_running: queue.Queue,\n",
    "    queue_save_tuning_job: queue.Queue,\n",
    ") -> Dict[str, Any]:\n",
    "    queue_currently_running.put(None)\n",
    "    tuner, job_information = create_or_attach_tuner(\n",
    "        model_name, dataset, unique_id, queue_save_tuning_job\n",
    "    )\n",
    "    tuner.wait()\n",
    "    job_summary_dict = extract_job_summary_from_logs(tuner, job_information)\n",
    "    print(f\"> Completed SageMaker tuning job, {job_information}\")\n",
    "    return job_summary_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8513488-d922-4807-b2fe-0ccf663d4c8e",
   "metadata": {},
   "source": [
    "***\n",
    "Finally, you need a couple of helper functions to log information to file. The first is intended to be triggered whenever the `create_or_attach_tuner` function puts job information onto the `queue_save_tuning_job` queue. Because you are using multithreading in this example, and it is not thread safe to have multiple threads write to file simultaneously, you will have the primary script listening to the futures threads pass job information to be saved to this function. The second helper function here is intended to be called whenever a tuning job completes. It extracts the metrics as the return value of the future and writes a json line to file. It also prints out any exceptions generated by the future without raising an error to allow the remainder of jobs to complete. This prevents a single job failure from preventing any future analyses.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcfc9edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def append_tuning_job_to_file(\n",
    "    job_information: JobInformation, file_path: Path = SAVE_TUNING_JOB_NAMES_FILE_PATH\n",
    ") -> None:\n",
    "    with open(file_path, \"a+\") as file:\n",
    "        file.write(f\"{json.dumps(job_information._asdict())}\\n\")\n",
    "    print(f\"> Saved job information to file, {job_information}\")\n",
    "\n",
    "\n",
    "def append_metrics_to_file(\n",
    "    future: cf.Future,\n",
    "    job_information: JobInformation,\n",
    "    file_path: Path = SAVE_METRICS_FILE_PATH,\n",
    ") -> None:\n",
    "    try:\n",
    "        metrics = future.result()\n",
    "        with open(file_path, \"a+\") as file:\n",
    "            file.write(f\"{json.dumps(metrics)}\\n\")\n",
    "        print(f\"> Saved metrics to file, {job_information}\")\n",
    "    except Exception as exc:\n",
    "        print(f\"> Exception generated for {job_information}: {exc}\")\n",
    "        traceback.print_exc()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c793bd",
   "metadata": {},
   "source": [
    "## 4. Asynchronously execute hyperparameter tuning jobs\n",
    "***\n",
    "Everything is now in place to launch training jobs and aggregate performance metrics for the benchmarking evaluation. This notebook makes use of the Python standard library's [concurrent futures](https://docs.python.org/3/library/concurrent.futures.html) module, which is a high-level interface for asynchronously executing callable functions. The `run_tuner` function will be repetitively executed on a thread pool and the `queue_currently_running` queue will block any threads from launching additional training instances until the number of currently running tuning jobs is less than `SM_AMT_MAX_PARALLEL_TUNING_JOBS`. Note that this queue would not be necessary if a `ProcessPoolExecutor` was used in place of `ThreadPoolExecutor`, but a process pool cannot share global state and therefore calling the functions `append_tuning_job_to_file` and `append_metrics_to_file` would not be thread safe.\n",
    "\n",
    "Once all jobs are submitted to the executor, this script listens to the futures job pool. Until all jobs are completed, it will perform two tasks: 1) call `append_tuning_job_to_file` with any job information that gets populated into `queue_save_tuning_job`, and 2) call `append_metrics_to_file` for any future that has finished execution.\n",
    "\n",
    "__FINAL NOTE__: Depending on the number of models and datasets defined above, this block may take a long time to run and consume many resources. Please double-check your settings!\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25eee554",
   "metadata": {},
   "outputs": [],
   "source": [
    "if SAVE_METRICS_FILE_PATH.exists():\n",
    "    SAVE_METRICS_FILE_PATH.unlink()\n",
    "\n",
    "queue_save_tuning_job = queue.Queue()\n",
    "queue_currently_running = queue.Queue(maxsize=SM_AMT_MAX_PARALLEL_TUNING_JOBS)\n",
    "\n",
    "jobs = itertools.product(MODELS, DATASETS)\n",
    "\n",
    "with cf.ThreadPoolExecutor(max_workers=SM_AMT_MAX_PARALLEL_TUNING_JOBS) as executor:\n",
    "    futures_to_job_information = {}\n",
    "    for unique_id, (model_name, dataset_name) in enumerate(jobs):\n",
    "        dataset = DATASET_DICT[dataset_name]\n",
    "        future = executor.submit(\n",
    "            run_tuner,\n",
    "            model_name,\n",
    "            dataset,\n",
    "            unique_id,\n",
    "            queue_currently_running,\n",
    "            queue_save_tuning_job,\n",
    "        )\n",
    "        futures_to_job_information[future] = JobInformation(model_name, dataset_name)\n",
    "\n",
    "    while futures_to_job_information:\n",
    "        done, not_done = cf.wait(\n",
    "            futures_to_job_information, timeout=5.0, return_when=cf.FIRST_COMPLETED\n",
    "        )\n",
    "\n",
    "        while not queue_save_tuning_job.empty():\n",
    "            job_information = queue_save_tuning_job.get()\n",
    "            append_tuning_job_to_file(job_information)\n",
    "\n",
    "        for future in done:\n",
    "            queue_currently_running.get()\n",
    "            job_information_before_execution = futures_to_job_information.pop(future)\n",
    "            append_metrics_to_file(future, job_information_before_execution)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a7caaa52-082e-4fad-bfdb-e74d60abef87",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 5. Analyze results\n",
    "***\n",
    "At this point, all tuning jobs should have completed execution. Congratulations! Please check the file `SAVE_METRICS_FILE_PATH` to see that each job should have appended a JSON object to a new row in the file. You can read the contents of this file into a Pandas `DataFrame` to view results in tabular form. The following block reads the saved metrics into a `DataFrame` and then cleans the model_name column and adds a model_category column indicating the model architecture.\n",
    "\n",
    "Packages are re-imported in this section so you can perform analyses at a later time or in a different session given the saved JSON Lines file without re-running this whole notebook. You may need to manually re-define the constant `SAVE_METRICS_FILE_PATH` here.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c07e23c-84ac-43ed-96fa-60cc51cf2cbd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "metrics_df = pd.read_json(SAVE_METRICS_FILE_PATH, lines=True)\n",
    "\n",
    "print(\"Available columns in metrics_df are as follows:\", metrics_df.columns.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48397986",
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_name_clean(model_name: str):\n",
    "    model_name = model_name.replace(\"tensorflow-ic-\", \"\")\n",
    "    model_name = model_name.replace(\"imagenet-\", \"\")\n",
    "    model_name = model_name.split(\"-classification\")[0]\n",
    "    return model_name\n",
    "\n",
    "\n",
    "metrics_df[\"model_name\"] = metrics_df[\"model_name\"].apply(model_name_clean)\n",
    "metrics_df[\"model_category\"] = metrics_df[\"model_name\"].apply(\n",
    "    lambda x: x.replace(\"tf2-preview-\", \"\").split(\"-\")[0]\n",
    ")\n",
    "\n",
    "index_columns = [\"dataset_name\", \"model_category\", \"model_name\"]\n",
    "display_columns = [name for name in metrics_df.columns if \"test_\" in name]\n",
    "display(metrics_df.sort_values(by=index_columns).set_index(index_columns)[display_columns])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e931181-c3b7-4185-9a2c-b9e5c419e77b",
   "metadata": {},
   "source": [
    "***\n",
    "With a Pandas `DataFrame` of all performance metrics populated, you can easily create tables of interest. For example, here the table is pivoted to display models as rows and performance for different datasets as columns.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82f4b6fe-64dc-4622-86ef-8f2596eeb3be",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", 500)\n",
    "index_columns = [\"model_category\", \"model_name\"]\n",
    "columns = [\"dataset_name\"]\n",
    "value_columns = [\"test_accuracy\", \"test_evaluation_latency\", \"train_duration\"]\n",
    "display(metrics_df.pivot(index=index_columns, columns=columns, values=value_columns).round(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057125f2",
   "metadata": {},
   "source": [
    "***\n",
    "Finally, you can create a plotly figure illustrating the Pareto front tradeoff between validation accuracy and throughput. If using Jupyter Lab, be sure to enable the plotly Jupyter extension for best viewing results.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be6f912d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import plotly.express as px\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "\n",
    "def benchmarking_figure(\n",
    "    df: pd.DataFrame,\n",
    "    dataset_name: str,\n",
    "    x: str = \"test_throughput\",\n",
    "    y: str = \"test_accuracy\",\n",
    "    title: str = \"SageMaker JumpStart TensorFlow Image Classification Benchmarking\",\n",
    "    model_name: str = \"model_name\",\n",
    "    xaxis_title: str = \"throughput (images per second)\",\n",
    "    yaxis_title: str = \"test accuracy\",\n",
    "    size: str = \"num_params\",\n",
    "    color: str = \"model_category\",\n",
    "    width: int = 800,\n",
    "    height: int = 600,\n",
    ") -> go.Figure:\n",
    "\n",
    "    df[f\"sqrt_{size}\"] = np.sqrt(df[size])\n",
    "    df = df.sort_values(by=[model_name])\n",
    "    df = df[df[\"dataset_name\"] == dataset_name]\n",
    "\n",
    "    fig = px.scatter(\n",
    "        df,\n",
    "        x=x,\n",
    "        y=y,\n",
    "        color=color,\n",
    "        size=f\"sqrt_{size}\",\n",
    "        title=f\"{title} ({dataset_name})\",\n",
    "        hover_name=model_name,\n",
    "        log_x=True,\n",
    "        width=width,\n",
    "        height=height,\n",
    "    )\n",
    "    fig.update_layout(\n",
    "        xaxis_title=xaxis_title,\n",
    "        yaxis_title=yaxis_title,\n",
    "    )\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e998a9-fff5-4bc6-abf5-e079c9e6b5ce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for dataset_name in metrics_df[\"dataset_name\"].unique():\n",
    "    fig = benchmarking_figure(metrics_df, dataset_name)\n",
    "    fig.write_html(f\"jumpstart_tf_ic_benchmarking_pareto_{dataset_name}.html\")\n",
    "    fig.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|jumpstart_image_classification|Amazon_JumpStart_Image_Classification_Benchmarking.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}