multi-model-endpoint-tensorflow-cv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "---\n",
    "## <span style=\"color:orange\"> Host Multiple TensorFlow Computer Vision Models using SageMaker Multi-Model Endpoint </span>\n",
    "---\n",
    "## <span style=\"color:black\">Contents</span>\n",
    "1. [Background](#Background)\n",
    "1. [Setup](#Setup)\n",
    "1. [Train Model 1 - CIFAR-10 Image Classification](#Train-Model-1---CIFAR-10-Image-Classification)\n",
    "1. [Train Model 2 - Sign Language Image Classification](#Train-Model-2---Sign-Language-Image-Classification)\n",
    "1. [Create a Multi-Model Endpoint](#Create-a-Multi-Model-Endpoint)\n",
    "1. [Test Multi-Model Endpoint for Real Time Inference](#Test-Multi-Model-Endpoint-for-Real-Time-Inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Background\n",
    "In this notebook, we show how to host two computer vision models trained using the TensorFlow framework under one SageMaker multi-model endpoint.  For the first model, we train a smaller version of AlexNet CNN to classify images from the CIFAR-10 dataset. For the second model, we use a pretrained VGG16 CNN model pretrained on the ImageNet dataset and fine-tune on Sign Language Digits Dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    " #### Prerequisites \n",
    " Choose Kernel for this notebook.<br>\n",
    " Under `Kernel` tab at the top of this notebook &#8594; `Choose kernel`, select `conda_python3` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "!pip install tensorflow==2.3.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Imports "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "from sagemaker.tensorflow.serving import TensorFlowModel\n",
    "from sagemaker.multidatamodel import MultiDataModel\n",
    "from tensorflow.keras.datasets import cifar10\n",
    "from sagemaker.tensorflow import TensorFlow\n",
    "from sagemaker.inputs import TrainingInput\n",
    "from sagemaker import get_execution_role\n",
    "from tensorflow.keras import utils\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "from datetime import datetime\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import sagemaker\n",
    "import logging\n",
    "import boto3\n",
    "import time\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Setup Logger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "logger = logging.getLogger(__name__)\n",
    "logger.setLevel(logging.INFO)\n",
    "logger.addHandler(logging.StreamHandler())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "logger.info(f'[Using TensorFlow version: {tf.__version__}]')\n",
    "logger.info(f'[Using SageMaker version: {sagemaker.__version__}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Seed for Reproducability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "SEED = 123\n",
    "np.random.seed(SEED)\n",
    "tf.random.set_seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Create Roles, Sessions and Data Locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "role = get_execution_role()\n",
    "session = boto3.Session()\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "s3 = session.resource('s3')\n",
    "TF_FRAMEWORK_VERSION = '2.3.0'\n",
    "BUCKET = sagemaker.Session().default_bucket()\n",
    "PREFIX = 'cv-models'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Train Model 1 - CIFAR-10 Image Classification\n",
    "\n",
    "<p align=\"justify\">First, we will train a Convolutional Neural Network (CNN) model to classify images from the CIFAR-10 dataset. Image classification is the task of assigning a label to an image, from a predefined set of categories. CIFAR-10 is an established CV dataset used for object recognition. It is a subset of the 80 Million Tiny Images dataset and consists of 60,000 (32x32) color images containing 1 of 10 object classes, with 6,000 images per class.</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### a) Load Data\n",
    "\n",
    "The first step is to load the pre-shuffled CIFAR-10 dataset into our train and test objects. Luckily, Keras provides the CIFAR dataset for us to load using the `load_data()` method. All we have to do is import keras.datasets and then load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = cifar10.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "logger.info(f'X_train Shape: {X_train.shape}')\n",
    "logger.info(f'y_train Shape: {y_train.shape}')\n",
    "logger.info(f'X_test Shape : {X_test.shape}')\n",
    "logger.info(f'y_test Shape : {y_test.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### b) Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(20, 5))\n",
    "for i in range(36):\n",
    "    ax = fig.add_subplot(3, 12, i+1, xticks=[], yticks=[])\n",
    "    ax.imshow(np.squeeze(X_train[i].T))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### c) Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "##### Rescale \n",
    "Rescales the images by dividing the pixel values by 255: [0,255] ⇒ [0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "X_train = X_train.astype('float32')/255\n",
    "X_test = X_test.astype('float32')/255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "##### One Hot Encode Target Labels\n",
    "One-hot encoding is a process by which categorical variables are converted into a numeric form. One-hot encoding converts the (1 × n) label vector to a label matrix of dimensions (10 × n), where n is the number of sample images. So, if we have 1,000 images in our dataset, the label vector will have the dimensions (1 × 1000). After one-hot encoding, the label matrix dimensions will be (1000 × 10). That’s why, when we define our network architecture in the next step, we will make the output softmax layer contain 10 nodes, where each node represents the probability of each class we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "num_classes = len(np.unique(y_train))\n",
    "y_train = utils.to_categorical(y_train, num_classes)\n",
    "y_test = utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "##### Split Data\n",
    "Break original train set further into train and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_validation = X_train[500:], X_train[:500]\n",
    "y_train, y_validation = y_train[500:], y_train[:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "##### Save to Local"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Create a local `data/cifar_10` directory to save the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "DATASET_PATH = './data/cifar_10'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "os.makedirs(DATASET_PATH, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Save train, validation and test sets to local `data` directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "np.save(f'{DATASET_PATH}/X_train.npy', X_train)\n",
    "np.save(f'{DATASET_PATH}/y_train.npy', y_train)\n",
    "np.save(f'{DATASET_PATH}/X_validation.npy', X_validation)\n",
    "np.save(f'{DATASET_PATH}/y_validation.npy', y_validation)\n",
    "np.save(f'{DATASET_PATH}/X_test.npy', X_test)\n",
    "np.save(f'{DATASET_PATH}/y_test.npy', y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "##### Copy Datasets to S3\n",
    "Copy train, validation and test sets from the local dir to S3, since SageMaker expects datasets to be in S3 for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!aws s3 cp ./{DATASET_PATH}/X_train.npy s3://{BUCKET}/{PREFIX}/cifar_10/train/\n",
    "!aws s3 cp ./{DATASET_PATH}/y_train.npy s3://{BUCKET}/{PREFIX}/cifar_10/train/\n",
    "!aws s3 cp ./{DATASET_PATH}/X_validation.npy s3://{BUCKET}/{PREFIX}/cifar_10/validation/\n",
    "!aws s3 cp ./{DATASET_PATH}/y_validation.npy s3://{BUCKET}/{PREFIX}/cifar_10/validation/\n",
    "!aws s3 cp ./{DATASET_PATH}/X_test.npy s3://{BUCKET}/{PREFIX}/cifar_10/test/\n",
    "!aws s3 cp ./{DATASET_PATH}/y_test.npy s3://{BUCKET}/{PREFIX}/cifar_10/test/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### d) Create Training Inputs \n",
    "Using the S3 locations of the datasets we saved in the previous step, create pointers to these datasets using the `TrainingInput`class from the SageMaker SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_input = TrainingInput(s3_data=f's3://{BUCKET}/{PREFIX}/cifar_10/train', \n",
    "                            distribution='FullyReplicated', \n",
    "                            content_type='npy')\n",
    "validation_input = TrainingInput(s3_data=f's3://{BUCKET}/{PREFIX}/cifar_10/validation', \n",
    "                                 distribution='FullyReplicated', \n",
    "                                 content_type='npy')\n",
    "test_input = TrainingInput(s3_data=f's3://{BUCKET}/{PREFIX}/cifar_10/test', \n",
    "                           distribution='FullyReplicated', \n",
    "                           content_type='npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "inputs = {'train': train_input, 'val': validation_input, 'test': test_input}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### e) Define Model Architecture & create Training Script\n",
    "\n",
    "We will build a small CNN consisting of three convolutional layers and two dense layers.<br>\n",
    "<b>Note:</b> We will use the ReLU activation function for all the hidden layers. In the last dense layer, we will use a softmax activation function with 10 nodes to return an array of 10 probability scores (summing to 1). Each score will be the probability that the current image belongs to our 10 image classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!pygmentize cifar_train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### f) Create a TensorFlow Estimator & fit the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "model_name = 'cifar-10'\n",
    "hyperparameters = {'epochs': 30}\n",
    "estimator_parameters = {'entry_point':'cifar_train.py',\n",
    "                        'instance_type': 'ml.m5.2xlarge',\n",
    "                        'instance_count': 1,\n",
    "                        'model_dir': f'/opt/ml/model',\n",
    "                        'role': role,\n",
    "                        'hyperparameters': hyperparameters,\n",
    "                        'output_path': f's3://{BUCKET}/{PREFIX}/cifar_10/out',\n",
    "                        'base_job_name': f'mme-cv-{model_name}',\n",
    "                        'framework_version': TF_FRAMEWORK_VERSION,\n",
    "                        'py_version': 'py37',\n",
    "                        'script_mode': True}\n",
    "estimator_1 = TensorFlow(**estimator_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "estimator_1.fit(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Train Model 2 - Sign Language Image Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### a) Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_path  = './data/sign_language/train'\n",
    "validation_path  = './data/sign_language/validation'\n",
    "test_path  = './data/sign_language/test'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### b) Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "img = mpimg.imread(f'{train_path}/0/IMG_1118.JPG')\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "img.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<p>ImageDataGenerator generates batches of tensor image data with real-time data augmentation. \n",
    "The data will be looped over (in batches)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_batches = ImageDataGenerator().flow_from_directory(train_path, \n",
    "                                                         target_size=(224, 224), \n",
    "                                                         batch_size=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_batches.next()[0][0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Visualize random sample of images from `train_batches`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "fig, rows = plt.subplots(nrows=1, ncols=4, figsize=(224, 224))\n",
    "for row in rows:\n",
    "    row.imshow(train_batches.next()[0][0].astype('int').T)\n",
    "    row.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### c) Copy local data to S3 for SageMaker training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!aws s3 cp ./data/sign_language/ s3://{BUCKET}/{PREFIX}/sign_language/ --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### d) Define Model Architecture & create Training Script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!pygmentize sign_language_train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### f) Create a TensorFlow Estimator & fit the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_input = TrainingInput(s3_data=f's3://{BUCKET}/{PREFIX}/sign_language/train', \n",
    "                            distribution='ShardedByS3Key')\n",
    "val_input = TrainingInput(s3_data=f's3://{BUCKET}/{PREFIX}/sign_language/validation', \n",
    "                          distribution='ShardedByS3Key')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "model_name = 'sign-language'\n",
    "\n",
    "hyperparameters = {'epochs': 1}\n",
    "estimator_parameters = {'entry_point':'sign_language_train.py',\n",
    "                        'instance_type': 'ml.m5.2xlarge',\n",
    "                        'instance_count': 1,\n",
    "                        'hyperparameters': hyperparameters,\n",
    "                        'model_dir': f'/opt/ml/model',\n",
    "                        'role': role,\n",
    "                        'output_path': f's3://{BUCKET}/{PREFIX}/sign_language/out',\n",
    "                        'base_job_name': f'cv-{model_name}',\n",
    "                        'framework_version': TF_FRAMEWORK_VERSION,\n",
    "                        'py_version': 'py37',\n",
    "                        'script_mode': True}\n",
    "\n",
    "estimator_2 = TensorFlow(**estimator_parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "estimator_2.fit({'train': train_input, 'val': val_input})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Create a Multi-Model Endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### a) Copy Trained Models to a common S3 Prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "tf_model_1 = estimator_1.model_data\n",
    "output_1 = f's3://{BUCKET}/{PREFIX}/mme/cifar.tar.gz'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "tf_model_2 = estimator_2.model_data\n",
    "output_2 = f's3://{BUCKET}/{PREFIX}/mme/sign-language.tar.gz'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "tf_model_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!aws s3 cp {tf_model_1} {output_1}\n",
    "!aws s3 cp {tf_model_2} {output_2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### b) Essentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "current_time = datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d-%H-%M-%S')\n",
    "current_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "IMAGE_URI = '763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.3.1-cpu-py37-ubuntu18.04'\n",
    "model_data_prefix = f's3://{BUCKET}/{PREFIX}/mme/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### c) Create a MultiDataModel using Model 1 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "model_1 = TensorFlowModel(model_data=output_1, \n",
    "                          role=role, \n",
    "                          image_uri=IMAGE_URI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "mme = MultiDataModel(name=f'mme-tensorflow-{current_time}',\n",
    "                     model_data_prefix=model_data_prefix,\n",
    "                     model=model_1,\n",
    "                     sagemaker_session=sagemaker_session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### d) Deploy Multi-Model Endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "predictor = mme.deploy(initial_instance_count=1,\n",
    "                       instance_type='ml.m5.2xlarge',\n",
    "                       endpoint_name=f'mme-tensorflow-{current_time}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "list(mme.list_models())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Test Multi-Model Endpoint for Real Time Inference "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### a) Test Model-1 CIFAR Image Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from tensorflow.keras.preprocessing.image import img_to_array\n",
    "from tensorflow.keras.preprocessing.image import load_img\n",
    "from tensorflow.keras.preprocessing import image\n",
    "from IPython.display import Image\n",
    "import matplotlib.image as mpimg \n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "CIFAR10_LABELS = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "Image('./data/cifar_10/raw_images/airplane.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "img = load_img('./data/cifar_10/raw_images/airplane.png', target_size=(32, 32))\n",
    "data = img_to_array(img)\n",
    "data = data.astype('float32')\n",
    "data = data / 255.0\n",
    "data = data.reshape(1, 32, 32, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "payload = {'instances': data}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "y_pred = predictor.predict(data=payload, initial_args={'TargetModel': 'cifar.tar.gz'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "predicted_label = CIFAR10_LABELS[np.argmax(y_pred)]\n",
    "print(f'Predicted Label: [{predicted_label}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### b) Test Model-2 Sign Language Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "test_path  = './data/sign_language/test'\n",
    "img = mpimg.imread(f'{test_path}/0/IMG_4159.JPG')\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def path_to_tensor(img_path):\n",
    "    # loads RGB image as PIL.Image.Image type\n",
    "    img = image.load_img(img_path, target_size=(224, 224))\n",
    "    # convert PIL.Image.Image type to 3D tensor with shape (224, 224, 3)\n",
    "    x = image.img_to_array(img)\n",
    "    # convert 3D tensor to 4D tensor with shape (1, 224, 224, 3) and return 4D tensor\n",
    "    return np.expand_dims(x, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "data = path_to_tensor(f'{test_path}/0/IMG_4159.JPG')\n",
    "payload = {'instances': data}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "y_pred = predictor.predict(data=payload, initial_args={'TargetModel': 'sign-language.tar.gz'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "predicted_label = np.argmax(y_pred)\n",
    "print(f'Predicted Label: [{predicted_label}]')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}