advanced_functionality/multi_model_xgboost_home_value/xgboost_multi_model_endpoint_home_value.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon SageMaker Multi-Model Endpoints using XGBoost\n",
    "With [Amazon SageMaker multi-model endpoints](https://docs.aws.amazon.com/sagemaker/latest/dg/multi-model-endpoints.html), customers can create an endpoint that seamlessly hosts up to thousands of models. These endpoints are well suited to use cases where any one of a large number of models, which can be served from a common inference container, needs to be invokable on-demand and where it is acceptable for infrequently invoked models to incur some additional latency. For applications which require consistently low inference latency, a traditional endpoint is still the best choice.\n",
    "\n",
    "At a high level, Amazon SageMaker manages the loading and unloading of models for a multi-model endpoint, as they are needed. When an invocation request is made for a particular model, Amazon SageMaker routes the request to an instance assigned to that model, downloads the model artifacts from S3 onto that instance, and initiates loading of the model into the memory of the container. As soon as the loading is complete, Amazon SageMaker performs the requested invocation and returns the result. If the model is already loaded in memory on the selected instance, the downloading and loading steps are skipped and the invocation is performed immediately.\n",
    "\n",
    "To demonstrate how multi-model endpoints are created and used, this notebook provides an example using a set of XGBoost models that each predict housing prices for a single location. This domain is used as a simple example to easily experiment with multi-model endpoints.\n",
    "\n",
    "The Amazon SageMaker multi-model endpoint capability is designed to work across all machine learning frameworks and algorithms including those where you bring your own container."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Contents\n",
    "\n",
    "1. [Build and register an XGBoost container that can serve multiple models](#Build-and-register-an-XGBoost-container-that-can-serve-multiple-models)\n",
    "1. [Generate synthetic data for housing models](#Generate-synthetic-data-for-housing-models)\n",
    "1. [Train multiple house value prediction models](#Train-multiple-house-value-prediction-models)\n",
    "1. [Import models into hosting](#Import-models-into-hosting)\n",
    "  1. [Deploy model artifacts to be found by the endpoint](#Deploy-model-artifacts-to-be-found-by-the-endpoint)\n",
    "  1. [Create the Amazon SageMaker model entity](#Create-the-Amazon-SageMaker-model-entity)\n",
    "  1. [Create the multi-model endpoint](#Create-the-multi-model-endpoint)\n",
    "1. [Exercise the multi-model endpoint](#Exercise-the-multi-model-endpoint)\n",
    "  1. [Dynamically deploy another model](#Dynamically-deploy-another-model)\n",
    "  1. [Invoke the newly deployed model](#Invoke-the-newly-deployed-model)\n",
    "  1. [Updating a model](#Updating-a-model)\n",
    "1. [Clean up](#Clean-up)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build and register an XGBoost container that can serve multiple models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -qU awscli boto3 sagemaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the inference container to serve multiple models in a multi-model endpoint, it must implement [additional APIs](https://docs.aws.amazon.com/sagemaker/latest/dg/build-multi-model-build-container.html) in order to load, list, get, unload and invoke specific models.\n",
    "\n",
    "The ['mme' branch of the SageMaker XGBoost Container repository](https://github.com/aws/sagemaker-xgboost-container/tree/mme) is an example implementation on how to adapt SageMaker's XGBoost framework container to use [Multi Model Server](https://github.com/awslabs/multi-model-server), a framework that provides an HTTP frontend that implements the additional container APIs required by multi-model endpoints, and also provides a pluggable backend handler for serving models using a custom framework, in this case the XGBoost framework.\n",
    "\n",
    "Using this branch, below we will build an XGBoost container that fulfills all of the multi-model endpoint container requirements, and then upload that image to Amazon Elastic Container Registry (ECR). Because uploading the image to ECR may create a new ECR repository, this notebook requires permissions in addition to the regular SageMakerFullAccess permissions. The easiest way to add these permissions is simply to add the managed policy AmazonEC2ContainerRegistryFullAccess to the role that you used to start your notebook instance. There's no need to restart your notebook instance when you do this, the new permissions will be available immediately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ALGORITHM_NAME = 'multi-model-xgboost'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh -s $ALGORITHM_NAME\n",
    "\n",
    "algorithm_name=$1\n",
    "\n",
    "account=$(aws sts get-caller-identity --query Account --output text)\n",
    "\n",
    "# Get the region defined in the current configuration\n",
    "region=$(aws configure get region)\n",
    "\n",
    "ecr_image=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n",
    "\n",
    "# If the repository doesn't exist in ECR, create it.\n",
    "aws ecr describe-repositories --repository-names \"${algorithm_name}\" > /dev/null 2>&1\n",
    "\n",
    "if [ $? -ne 0 ]\n",
    "then\n",
    "    aws ecr create-repository --repository-name \"${algorithm_name}\" > /dev/null\n",
    "fi\n",
    "\n",
    "# Get the login command from ECR and execute it directly\n",
    "$(aws ecr get-login --region ${region} --no-include-email --registry-ids ${account})\n",
    "\n",
    "# Build the docker image locally with the image name and then push it to ECR\n",
    "# with the full name.\n",
    "\n",
    "# First clear out any prior version of the cloned repo\n",
    "rm -rf sagemaker-xgboost-container/\n",
    "\n",
    "# Clone the xgboost container repo\n",
    "git clone --single-branch --branch mme https://github.com/aws/sagemaker-xgboost-container.git\n",
    "cd sagemaker-xgboost-container/\n",
    "\n",
    "# Build the \"base\" container image that encompasses the installation of the\n",
    "# XGBoost framework and all of the dependencies needed.\n",
    "docker build -q -t xgboost-container-base:0.90-2-cpu-py3 -f docker/0.90-2/base/Dockerfile.cpu .\n",
    "\n",
    "# Create the SageMaker XGBoost Container Python package.\n",
    "python setup.py bdist_wheel --universal\n",
    "\n",
    "# Build the \"final\" container image that encompasses the installation of the\n",
    "# code that implements the SageMaker multi-model container requirements.\n",
    "docker build -q -t ${algorithm_name} -f docker/0.90-2/final/Dockerfile.cpu .\n",
    "\n",
    "docker tag ${algorithm_name} ${ecr_image}\n",
    "\n",
    "docker push ${ecr_image}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate synthetic data for housing models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import json\n",
    "import datetime\n",
    "import time\n",
    "from time import gmtime, strftime\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_HOUSES_PER_LOCATION = 1000\n",
    "LOCATIONS  = ['NewYork_NY',    'LosAngeles_CA',   'Chicago_IL',    'Houston_TX',   'Dallas_TX',\n",
    "              'Phoenix_AZ',    'Philadelphia_PA', 'SanAntonio_TX', 'SanDiego_CA',  'SanFrancisco_CA']\n",
    "PARALLEL_TRAINING_JOBS = 4 # len(LOCATIONS) if your account limits can handle it\n",
    "MAX_YEAR = 2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_price(house):\n",
    "    _base_price = int(house['SQUARE_FEET'] * 150)\n",
    "    _price = int(_base_price + (10000 * house['NUM_BEDROOMS']) + \\\n",
    "                               (15000 * house['NUM_BATHROOMS']) + \\\n",
    "                               (15000 * house['LOT_ACRES']) + \\\n",
    "                               (15000 * house['GARAGE_SPACES']) - \\\n",
    "                               (5000 * (MAX_YEAR - house['YEAR_BUILT'])))\n",
    "    return _price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_random_house():\n",
    "    _house = {'SQUARE_FEET':   int(np.random.normal(3000, 750)),\n",
    "              'NUM_BEDROOMS':  np.random.randint(2, 7),\n",
    "              'NUM_BATHROOMS': np.random.randint(2, 7) / 2,\n",
    "              'LOT_ACRES':     round(np.random.normal(1.0, 0.25), 2),\n",
    "              'GARAGE_SPACES': np.random.randint(0, 4),\n",
    "              'YEAR_BUILT':    min(MAX_YEAR, int(np.random.normal(1995, 10)))}\n",
    "    _price = gen_price(_house)\n",
    "    return [_price, _house['YEAR_BUILT'],   _house['SQUARE_FEET'], \n",
    "                    _house['NUM_BEDROOMS'], _house['NUM_BATHROOMS'], \n",
    "                    _house['LOT_ACRES'],    _house['GARAGE_SPACES']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_houses(num_houses):\n",
    "    _house_list = []\n",
    "    for i in range(num_houses):\n",
    "        _house_list.append(gen_random_house())\n",
    "    _df = pd.DataFrame(_house_list, \n",
    "                       columns=['PRICE',        'YEAR_BUILT',    'SQUARE_FEET',  'NUM_BEDROOMS',\n",
    "                                'NUM_BATHROOMS','LOT_ACRES',     'GARAGE_SPACES'])\n",
    "    return _df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train multiple house value prediction models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.predictor import csv_serializer\n",
    "import boto3\n",
    "\n",
    "sm_client = boto3.client(service_name='sagemaker')\n",
    "runtime_sm_client = boto3.client(service_name='sagemaker-runtime')\n",
    "\n",
    "s3 = boto3.resource('s3')\n",
    "s3_client = boto3.client('s3')\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "role = get_execution_role()\n",
    "\n",
    "ACCOUNT_ID = boto3.client('sts').get_caller_identity()['Account']\n",
    "REGION     = boto3.Session().region_name\n",
    "BUCKET     = sagemaker_session.default_bucket()\n",
    "\n",
    "MULTI_MODEL_XGBOOST_IMAGE = '{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(ACCOUNT_ID, REGION, \n",
    "                                                                           ALGORITHM_NAME)\n",
    "\n",
    "DATA_PREFIX            = 'DEMO_MME_XGBOOST'\n",
    "HOUSING_MODEL_NAME     = 'housing'\n",
    "MULTI_MODEL_ARTIFACTS  = 'multi_model_artifacts'\n",
    "\n",
    "TRAIN_INSTANCE_TYPE    = 'ml.m4.xlarge'\n",
    "ENDPOINT_INSTANCE_TYPE = 'ml.m4.xlarge'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split a given dataset into train, validation, and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "SEED = 7\n",
    "SPLIT_RATIOS = [0.6, 0.3, 0.1]\n",
    "\n",
    "def split_data(df):\n",
    "    # split data into train and test sets\n",
    "    seed      = SEED\n",
    "    val_size  = SPLIT_RATIOS[1]\n",
    "    test_size = SPLIT_RATIOS[2]\n",
    "    \n",
    "    num_samples = df.shape[0]\n",
    "    X1 = df.values[:num_samples, 1:] # keep only the features, skip the target, all rows\n",
    "    Y1 = df.values[:num_samples, :1] # keep only the target, all rows\n",
    "\n",
    "    # Use split ratios to divide up into train/val/test\n",
    "    X_train, X_val, y_train, y_val = \\\n",
    "        train_test_split(X1, Y1, test_size=(test_size + val_size), random_state=seed)\n",
    "    # Of the remaining non-training samples, give proper ratio to validation and to test\n",
    "    X_test, X_test, y_test, y_test = \\\n",
    "        train_test_split(X_val, y_val, test_size=(test_size / (test_size + val_size)), \n",
    "                         random_state=seed)\n",
    "    # reassemble the datasets with target in first column and features after that\n",
    "    _train = np.concatenate([y_train, X_train], axis=1)\n",
    "    _val   = np.concatenate([y_val,   X_val],   axis=1)\n",
    "    _test  = np.concatenate([y_test,  X_test],  axis=1)\n",
    "\n",
    "    return _train, _val, _test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch a single training job for a given housing location\n",
    "There is nothing specific to multi-model endpoints in terms of the models it will host. They are trained in the same way as all other SageMaker models. Here we are using the XGBoost estimator and not waiting for the job to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def launch_training_job(location):\n",
    "    # clear out old versions of the data\n",
    "    s3_bucket = s3.Bucket(BUCKET)\n",
    "    full_input_prefix = '{}/model_prep/{}'.format(DATA_PREFIX, location)\n",
    "    s3_bucket.objects.filter(Prefix=full_input_prefix + '/').delete()\n",
    "\n",
    "    # upload the entire set of data for all three channels\n",
    "    local_folder = 'data/{}'.format(location)\n",
    "    inputs = sagemaker_session.upload_data(path=local_folder, key_prefix=full_input_prefix)\n",
    "    print('Training data uploaded: {}'.format(inputs))\n",
    "    \n",
    "    _job = 'xgb-{}'.format(location.replace('_', '-'))\n",
    "    full_output_prefix = '{}/model_artifacts/{}'.format(DATA_PREFIX, location)\n",
    "    s3_output_path = 's3://{}/{}'.format(BUCKET, full_output_prefix)\n",
    "\n",
    "    xgb = sagemaker.estimator.Estimator(MULTI_MODEL_XGBOOST_IMAGE, role, \n",
    "                                        train_instance_count=1, train_instance_type=TRAIN_INSTANCE_TYPE,\n",
    "                                        output_path=s3_output_path, base_job_name=_job,\n",
    "                                        sagemaker_session=sagemaker_session)\n",
    "    xgb.set_hyperparameters(max_depth=5, eta=0.2, gamma=4, min_child_weight=6, subsample=0.8, silent=0, \n",
    "                            early_stopping_rounds=5, objective='reg:linear', num_round=25) \n",
    "    \n",
    "    DISTRIBUTION_MODE = 'FullyReplicated'\n",
    "    train_input = sagemaker.s3_input(s3_data=inputs+'/train', \n",
    "                                     distribution=DISTRIBUTION_MODE, content_type='csv')\n",
    "    val_input   = sagemaker.s3_input(s3_data=inputs+'/val', \n",
    "                                     distribution=DISTRIBUTION_MODE, content_type='csv')\n",
    "    remote_inputs = {'train': train_input, 'validation': val_input}\n",
    "\n",
    "    xgb.fit(remote_inputs, wait=False)\n",
    "    \n",
    "    return xgb.latest_training_job.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kick off a model training job for each housing location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_data_locally(location, train, val, test):\n",
    "    os.makedirs('data/{}/train'.format(location))\n",
    "    np.savetxt( 'data/{0}/train/{0}_train.csv'.format(location), train, delimiter=',', fmt='%.2f')\n",
    "    \n",
    "    os.makedirs('data/{}/val'.format(location))\n",
    "    np.savetxt( 'data/{0}/val/{0}_val.csv'.format(location),     val, delimiter=',', fmt='%.2f')\n",
    "    \n",
    "    os.makedirs('data/{}/test'.format(location))\n",
    "    np.savetxt( 'data/{0}/test/{0}_test.csv'.format(location),   test, delimiter=',', fmt='%.2f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "import os\n",
    "\n",
    "training_jobs = []\n",
    "\n",
    "shutil.rmtree('data', ignore_errors=True)\n",
    "\n",
    "for loc in LOCATIONS[:PARALLEL_TRAINING_JOBS]:\n",
    "    _houses = gen_houses(NUM_HOUSES_PER_LOCATION)\n",
    "    _train, _val, _test = split_data(_houses)\n",
    "    save_data_locally(loc, _train, _val, _test)\n",
    "    _job = launch_training_job(loc)\n",
    "    training_jobs.append(_job)\n",
    "print('{} training jobs launched: {}'.format(len(training_jobs), training_jobs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait for all model training to finish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wait_for_training_job_to_complete(job_name):\n",
    "    print('Waiting for job {} to complete...'.format(job_name))\n",
    "    resp = sm_client.describe_training_job(TrainingJobName=job_name)\n",
    "    status = resp['TrainingJobStatus']\n",
    "    while status=='InProgress':\n",
    "        time.sleep(60)\n",
    "        resp = sm_client.describe_training_job(TrainingJobName=job_name)\n",
    "        status = resp['TrainingJobStatus']\n",
    "        if status == 'InProgress':\n",
    "            print('{} job status: {}'.format(job_name, status))\n",
    "    print('DONE. Status for {} is {}\\n'.format(job_name, status))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wait for the jobs to finish\n",
    "for j in training_jobs:\n",
    "    wait_for_training_job_to_complete(j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import models into hosting\n",
    "A big difference for multi-model endpoints is that when creating the Model entity, the container's `ModelDataUrl` is the S3 prefix where the model artifacts that are invokable by the endpoint are located. The rest of the S3 path will be specified when actually invoking the model. Remember to close the location with a trailing slash.\n",
    "\n",
    "The `Mode` of container is specified as `MultiModel` to signify that the container will host multiple models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy model artifacts to be found by the endpoint\n",
    "As described above, the multi-model endpoint is configured to find its model artifacts in a specific location in S3. For each trained model, we make a copy of its model artifacts into that location.\n",
    "\n",
    "In our example, we are storing all the models within a single folder. The implementation of multi-model endpoints is flexible enough to permit an arbitrary folder structure. For a set of housing models for example, you could have a top level folder for each region, and the model artifacts would be copied to those regional folders. The target model referenced when invoking such a model would include the folder path. For example, `northeast/Boston_MA.tar.gz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def parse_model_artifacts(model_data_url):\n",
    "    # extract the s3 key from the full url to the model artifacts\n",
    "    _s3_key = model_data_url.split('s3://{}/'.format(BUCKET))[1]\n",
    "    # get the part of the key that identifies the model within the model artifacts folder\n",
    "    _model_name_plus = _s3_key[_s3_key.find('model_artifacts') + len('model_artifacts') + 1:]\n",
    "    # finally, get the unique model name (e.g., \"NewYork_NY\")\n",
    "    _model_name = re.findall('^(.*?)/', _model_name_plus)[0]\n",
    "    return _s3_key, _model_name "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make a copy of the model artifacts from the original output of the training job to the place in\n",
    "# s3 where the multi model endpoint will dynamically load individual models\n",
    "def deploy_artifacts_to_mme(job_name):\n",
    "    _resp = sm_client.describe_training_job(TrainingJobName=job_name)\n",
    "    _source_s3_key, _model_name = parse_model_artifacts(_resp['ModelArtifacts']['S3ModelArtifacts'])\n",
    "    _copy_source = {'Bucket': BUCKET, 'Key': _source_s3_key}\n",
    "    _key = '{}/{}/{}.tar.gz'.format(DATA_PREFIX, MULTI_MODEL_ARTIFACTS, _model_name)\n",
    "    \n",
    "    print('Copying {} model\\n   from: {}\\n     to: {}...'.format(_model_name, _source_s3_key, _key))\n",
    "    s3_client.copy_object(Bucket=BUCKET, CopySource=_copy_source, Key=_key)\n",
    "    return _key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we are purposely *not* copying the first model. This will be copied later in the notebook to demonstrate how to dynamically add new models to an already running endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, clear out old versions of the model artifacts from previous runs of this notebook\n",
    "s3 = boto3.resource('s3')\n",
    "s3_bucket = s3.Bucket(BUCKET)\n",
    "full_input_prefix = '{}/multi_model_artifacts'.format(DATA_PREFIX)\n",
    "print('Removing old model artifacts from {}'.format(full_input_prefix))\n",
    "s3_bucket.objects.filter(Prefix=full_input_prefix + '/').delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# copy every model except the first one\n",
    "for job in training_jobs[1:]:\n",
    "    deploy_artifacts_to_mme(job)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Amazon SageMaker model entity\n",
    "Here we use `boto3` to create the model entity. Instead of describing a single model, it will indicate the use of multi-model semantics and will identify the source location of all specific model artifacts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_multi_model_entity(multi_model_name, role):\n",
    "    # establish the place in S3 from which the endpoint will pull individual models\n",
    "    _model_url  = 's3://{}/{}/{}/'.format(BUCKET, DATA_PREFIX, MULTI_MODEL_ARTIFACTS)\n",
    "    _container = {\n",
    "        'Image':        MULTI_MODEL_XGBOOST_IMAGE,\n",
    "        'ModelDataUrl': _model_url,\n",
    "        'Mode':         'MultiModel'\n",
    "    }\n",
    "    create_model_response = sm_client.create_model(\n",
    "        ModelName = multi_model_name,\n",
    "        ExecutionRoleArn = role,\n",
    "        Containers = [_container])\n",
    "    \n",
    "    return _model_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "multi_model_name = '{}-{}'.format(HOUSING_MODEL_NAME, strftime('%Y-%m-%d-%H-%M-%S', gmtime()))\n",
    "model_url = create_multi_model_entity(multi_model_name, role)\n",
    "print('Multi model name: {}'.format(multi_model_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the multi-model endpoint\n",
    "There is nothing special about the SageMaker endpoint config for a multi-model endpoint. You need to consider the appropriate instance type and number of instances for the projected prediction workload. The number and size of the individual models will drive memory requirements.\n",
    "\n",
    "Once the endpoint config is in place, the endpoint creation is straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_config_name = multi_model_name\n",
    "print('Endpoint config name: ' + endpoint_config_name)\n",
    "\n",
    "create_endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName = endpoint_config_name,\n",
    "    ProductionVariants=[{\n",
    "        'InstanceType': ENDPOINT_INSTANCE_TYPE,\n",
    "        'InitialInstanceCount': 1,\n",
    "        'InitialVariantWeight': 1,\n",
    "        'ModelName': multi_model_name,\n",
    "        'VariantName': 'AllTraffic'}])\n",
    "\n",
    "endpoint_name = multi_model_name\n",
    "print('Endpoint name: ' + endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    EndpointConfigName=endpoint_config_name)\n",
    "print('Endpoint Arn: ' + create_endpoint_response['EndpointArn'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Waiting for {} endpoint to be in service...'.format(endpoint_name))\n",
    "waiter = sm_client.get_waiter('endpoint_in_service')\n",
    "waiter.wait(EndpointName=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise the multi-model endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke multiple individual models hosted behind a single endpoint\n",
    "Here we iterate through a set of housing predictions, choosing the specific location-based housing model at random. Notice the cold start price paid for the first invocation of any given model. Subsequent invocations of the same model take advantage of the model already being loaded into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_one_house_value(features, model_name):\n",
    "    print('Using model {} to predict price of this house: {}'.format(full_model_name,\n",
    "                                                                     features))\n",
    "    body = ','.join(map(str, features)) + '\\n'\n",
    "    start_time = time.time()\n",
    "\n",
    "    response = runtime_sm_client.invoke_endpoint(\n",
    "                        EndpointName=endpoint_name,\n",
    "                        ContentType='text/csv',\n",
    "                        TargetModel=full_model_name,\n",
    "                        Body=body)\n",
    "    predicted_value = json.loads(response['Body'].read())[0]\n",
    "\n",
    "    duration = time.time() - start_time\n",
    "    \n",
    "    print('${:,.2f}, took {:,d} ms\\n'.format(predicted_value, int(duration * 1000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Here are the models that the endpoint has at its disposal:')\n",
    "!aws s3 ls --human-readable --summarize $model_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iterate through invocations with random inputs against a random model showing results and latency\n",
    "for i in range(10):\n",
    "    model_name = LOCATIONS[np.random.randint(1, len(LOCATIONS[:PARALLEL_TRAINING_JOBS]))]\n",
    "    full_model_name = '{}.tar.gz'.format(model_name)\n",
    "    predict_one_house_value(gen_random_house()[1:], full_model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dynamically deploy another model\n",
    "Here we demonstrate the power of dynamic loading of new models. We purposely did not copy the first model when deploying models earlier. Now we deploy an additional model and can immediately invoke it through the multi-model endpoint. As with the earlier models, the first invocation to the new model takes longer, as the endpoint takes time to download the model and load it into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add another model to the endpoint and exercise it\n",
    "deploy_artifacts_to_mme(training_jobs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke the newly deployed model\n",
    "Exercise the newly deployed model without the need for any endpoint update or restart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Here are the models that the endpoint has at its disposal:')\n",
    "!aws s3 ls $model_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = LOCATIONS[0]\n",
    "full_model_name = '{}.tar.gz'.format(model_name)\n",
    "for i in range(5):\n",
    "    features = gen_random_house()\n",
    "    predict_one_house_value(gen_random_house()[1:], full_model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Updating a model\n",
    "To update a model, you would follow the same approach as above and add it as a new model. For example, if you have retrained the `NewYork_NY.tar.gz` model and wanted to start invoking it, you would upload the updated model artifacts behind the S3 prefix with a new name such as `NewYork_NY_v2.tar.gz`, and then change the `TargetModel` field to invoke `NewYork_NY_v2.tar.gz` instead of `NewYork_NY.tar.gz`. You do not want to overwrite the model artifacts in Amazon S3, because the old version of the model might still be loaded in the containers or on the storage volume of the instances on the endpoint. Invocations to the new model could then invoke the old version of the model.\n",
    "\n",
    "Alternatively, you could stop the endpoint and re-deploy a fresh set of models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "Here, to be sure we are not billed for endpoints we are no longer using, we clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shut down the endpoint\n",
    "sm_client.delete_endpoint(EndpointName=endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and the endpoint config\n",
    "sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete model too\n",
    "sm_client.delete_model(ModelName=multi_model_name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}