 archived/xgboost_multi_model_endpoint_home_value.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adolescent-lloyd",
   "metadata": {
    "papermill": {
     "duration": 0.017999,
     "end_time": "2021-05-26T15:52:21.666811",
     "exception": false,
     "start_time": "2021-05-26T15:52:21.648812",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Amazon SageMaker Multi-Model Endpoints using XGBoost\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ed3421f5",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a155f75d",
   "metadata": {
    "papermill": {
     "duration": 0.017999,
     "end_time": "2021-05-26T15:52:21.666811",
     "exception": false,
     "start_time": "2021-05-26T15:52:21.648812",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "With [Amazon SageMaker multi-model endpoints](https://docs.aws.amazon.com/sagemaker/latest/dg/multi-model-endpoints.html), customers can create an endpoint that seamlessly hosts up to thousands of models. These endpoints are well suited to use cases where any one of a large number of models, which can be served from a common inference container to save inference costs, needs to be invokable on-demand and where it is acceptable for infrequently invoked models to incur some additional latency. For applications which require consistently low inference latency, an endpoint deploying a single model is still the best choice.\n",
    "\n",
    "At a high level, Amazon SageMaker manages the loading and unloading of models for a multi-model endpoint, as they are needed. When an invocation request is made for a particular model, Amazon SageMaker routes the request to an instance assigned to that model, downloads the model artifacts from S3 onto that instance, and initiates loading of the model into the memory of the container. As soon as the loading is complete, Amazon SageMaker performs the requested invocation and returns the result. If the model is already loaded in memory on the selected instance, the downloading and loading steps are skipped and the invocation is performed immediately.\n",
    "\n",
    "To demonstrate how multi-model endpoints are created and used, this notebook provides an example using a set of XGBoost models that each predict housing prices for a single location. This domain is used as a simple example to easily experiment with multi-model endpoints.\n",
    "\n",
    "The Amazon SageMaker multi-model endpoint capability is designed to work across with Mxnet, PyTorch and Scikit-Learn machine learning frameworks (TensorFlow coming soon), SageMaker XGBoost, KNN, and Linear Learner algorithms.\n",
    "\n",
    "In addition, Amazon SageMaker multi-model endpoints are also designed to work with cases where you bring your own container that integrates with the multi-model server library. An example of this can be found [here](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/advanced_functionality/multi_model_bring_your_own) and documentation [here.](https://docs.aws.amazon.com/sagemaker/latest/dg/build-multi-model-build-container.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "extra-institute",
   "metadata": {
    "papermill": {
     "duration": 0.017728,
     "end_time": "2021-05-26T15:52:21.702317",
     "exception": false,
     "start_time": "2021-05-26T15:52:21.684589",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Contents\n",
    "\n",
    "1. [Generate synthetic data for housing models](#Generate-synthetic-data-for-housing-models)\n",
    "1. [Train multiple house value prediction models](#Train-multiple-house-value-prediction-models)\n",
    "1. [Create the Amazon SageMaker MultiDataModel entity](#Create-the-Amazon-SageMaker-MultiDataModel-entity)\n",
    "  1. [Create the Multi-Model Endpoint](#Create-the-multi-model-endpoint)\n",
    "  1. [Deploy the Multi-Model Endpoint](#deploy-the-multi-model-endpoint)\n",
    "1. [Get Predictions from the endpoint](#Get-predictions-from-the-endpoint)\n",
    "1. [Additional Information](#Additional-information)\n",
    "1. [Clean up](#Clean-up)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "macro-treatment",
   "metadata": {
    "papermill": {
     "duration": 0.017802,
     "end_time": "2021-05-26T15:52:21.738017",
     "exception": false,
     "start_time": "2021-05-26T15:52:21.720215",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Generate synthetic data\n",
    "\n",
    "The code below contains helper functions to generate synthetic data in the form of a `1x7` numpy array representing the features of a house.\n",
    "\n",
    "The first entry in the array is the randomly generated price of a house. The remaining entries are the features (i.e. number of bedroom, square feet, number of bathrooms, etc.).\n",
    "\n",
    "These functions will be used to generate synthetic data for training, validation, and testing. It will also allow us to submit synthetic payloads for inference to test our multi-model endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "trained-georgia",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -Uq sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "significant-brighton",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.42.1\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "\n",
    "print(sagemaker.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "synthetic-decline",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:21.777417Z",
     "iopub.status.busy": "2021-05-26T15:52:21.776697Z",
     "iopub.status.idle": "2021-05-26T15:52:21.987003Z",
     "shell.execute_reply": "2021-05-26T15:52:21.986510Z"
    },
    "papermill": {
     "duration": 0.231274,
     "end_time": "2021-05-26T15:52:21.987124",
     "exception": false,
     "start_time": "2021-05-26T15:52:21.755850",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "endless-stroke",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:22.030013Z",
     "iopub.status.busy": "2021-05-26T15:52:22.029557Z",
     "iopub.status.idle": "2021-05-26T15:52:22.031372Z",
     "shell.execute_reply": "2021-05-26T15:52:22.031715Z"
    },
    "papermill": {
     "duration": 0.024093,
     "end_time": "2021-05-26T15:52:22.031835",
     "exception": false,
     "start_time": "2021-05-26T15:52:22.007742",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "NUM_HOUSES_PER_LOCATION = 1000\n",
    "LOCATIONS = [\n",
    "    \"NewYork_NY\",\n",
    "    \"LosAngeles_CA\",\n",
    "    \"Chicago_IL\",\n",
    "    \"Houston_TX\",\n",
    "    \"Dallas_TX\",\n",
    "    \"Phoenix_AZ\",\n",
    "    \"Philadelphia_PA\",\n",
    "    \"SanAntonio_TX\",\n",
    "    \"SanDiego_CA\",\n",
    "    \"SanFrancisco_CA\",\n",
    "]\n",
    "PARALLEL_TRAINING_JOBS = 4  # len(LOCATIONS) if your account limits can handle it\n",
    "MAX_YEAR = 2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "grateful-swedish",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:22.073304Z",
     "iopub.status.busy": "2021-05-26T15:52:22.072704Z",
     "iopub.status.idle": "2021-05-26T15:52:22.074317Z",
     "shell.execute_reply": "2021-05-26T15:52:22.074648Z"
    },
    "papermill": {
     "duration": 0.024762,
     "end_time": "2021-05-26T15:52:22.074768",
     "exception": false,
     "start_time": "2021-05-26T15:52:22.050006",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def gen_price(house):\n",
    "    _base_price = int(house[\"SQUARE_FEET\"] * 150)\n",
    "    _price = int(\n",
    "        _base_price\n",
    "        + (10000 * house[\"NUM_BEDROOMS\"])\n",
    "        + (15000 * house[\"NUM_BATHROOMS\"])\n",
    "        + (15000 * house[\"LOT_ACRES\"])\n",
    "        + (15000 * house[\"GARAGE_SPACES\"])\n",
    "        - (5000 * (MAX_YEAR - house[\"YEAR_BUILT\"]))\n",
    "    )\n",
    "    return _price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "flying-tooth",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:22.116765Z",
     "iopub.status.busy": "2021-05-26T15:52:22.116281Z",
     "iopub.status.idle": "2021-05-26T15:52:22.118367Z",
     "shell.execute_reply": "2021-05-26T15:52:22.118004Z"
    },
    "papermill": {
     "duration": 0.025511,
     "end_time": "2021-05-26T15:52:22.118465",
     "exception": false,
     "start_time": "2021-05-26T15:52:22.092954",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def gen_random_house():\n",
    "    _house = {\n",
    "        \"SQUARE_FEET\": int(np.random.normal(3000, 750)),\n",
    "        \"NUM_BEDROOMS\": np.random.randint(2, 7),\n",
    "        \"NUM_BATHROOMS\": np.random.randint(2, 7) / 2,\n",
    "        \"LOT_ACRES\": round(np.random.normal(1.0, 0.25), 2),\n",
    "        \"GARAGE_SPACES\": np.random.randint(0, 4),\n",
    "        \"YEAR_BUILT\": min(MAX_YEAR, int(np.random.normal(1995, 10))),\n",
    "    }\n",
    "    _price = gen_price(_house)\n",
    "    return [\n",
    "        _price,\n",
    "        _house[\"YEAR_BUILT\"],\n",
    "        _house[\"SQUARE_FEET\"],\n",
    "        _house[\"NUM_BEDROOMS\"],\n",
    "        _house[\"NUM_BATHROOMS\"],\n",
    "        _house[\"LOT_ACRES\"],\n",
    "        _house[\"GARAGE_SPACES\"],\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "freelance-september",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:22.160044Z",
     "iopub.status.busy": "2021-05-26T15:52:22.159581Z",
     "iopub.status.idle": "2021-05-26T15:52:22.161558Z",
     "shell.execute_reply": "2021-05-26T15:52:22.161195Z"
    },
    "papermill": {
     "duration": 0.024632,
     "end_time": "2021-05-26T15:52:22.161655",
     "exception": false,
     "start_time": "2021-05-26T15:52:22.137023",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def gen_houses(num_houses):\n",
    "    _house_list = []\n",
    "    for i in range(num_houses):\n",
    "        _house_list.append(gen_random_house())\n",
    "    _df = pd.DataFrame(\n",
    "        _house_list,\n",
    "        columns=[\n",
    "            \"PRICE\",\n",
    "            \"YEAR_BUILT\",\n",
    "            \"SQUARE_FEET\",\n",
    "            \"NUM_BEDROOMS\",\n",
    "            \"NUM_BATHROOMS\",\n",
    "            \"LOT_ACRES\",\n",
    "            \"GARAGE_SPACES\",\n",
    "        ],\n",
    "    )\n",
    "    return _df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "addressed-headquarters",
   "metadata": {
    "papermill": {
     "duration": 0.018767,
     "end_time": "2021-05-26T15:52:22.198955",
     "exception": false,
     "start_time": "2021-05-26T15:52:22.180188",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Train multiple house value prediction models\n",
    "\n",
    "In the follow section, we are setting up the code to train a house price prediction model for each of 4 different cities.\n",
    "\n",
    "As such, we will launch multiple training jobs asynchronously, using the XGBoost algorithm.\n",
    "\n",
    "In this notebook, we will be using the AWS Managed XGBoost Image for both training and inference - this image provides native support for launching multi-model endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "informal-peeing",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:22.242238Z",
     "iopub.status.busy": "2021-05-26T15:52:22.241516Z",
     "iopub.status.idle": "2021-05-26T15:52:23.317862Z",
     "shell.execute_reply": "2021-05-26T15:52:23.318235Z"
    },
    "papermill": {
     "duration": 1.100112,
     "end_time": "2021-05-26T15:52:23.318383",
     "exception": false,
     "start_time": "2021-05-26T15:52:22.218271",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.amazon.amazon_estimator import image_uris\n",
    "import boto3\n",
    "from time import gmtime, strftime\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "s3 = boto3.resource(\"s3\")\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "role = get_execution_role()\n",
    "\n",
    "BUCKET = sagemaker_session.default_bucket()\n",
    "\n",
    "# This is references the AWS managed XGBoost container\n",
    "XGBOOST_IMAGE = image_uris.retrieve(\n",
    "    region=boto3.Session().region_name, framework=\"xgboost\", version=\"1.5-1\"\n",
    ")\n",
    "\n",
    "DATA_PREFIX = \"XGBOOST_BOSTON_HOUSING\"\n",
    "MULTI_MODEL_ARTIFACTS = \"multi_model_artifacts\"\n",
    "\n",
    "TRAIN_INSTANCE_TYPE = \"ml.m4.xlarge\"\n",
    "ENDPOINT_INSTANCE_TYPE = \"ml.m4.xlarge\"\n",
    "\n",
    "ENDPOINT_NAME = f'mme-xgboost-housing-{strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())}'\n",
    "\n",
    "MODEL_NAME = ENDPOINT_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liquid-canyon",
   "metadata": {
    "papermill": {
     "duration": 0.018486,
     "end_time": "2021-05-26T15:52:23.355516",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.337030",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Split a given dataset into train, validation, and test\n",
    "\n",
    "The code below will generate 3 sets of data. 1 set to train, 1 set for validation and 1 for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ordered-member",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:23.399413Z",
     "iopub.status.busy": "2021-05-26T15:52:23.398931Z",
     "iopub.status.idle": "2021-05-26T15:52:23.400640Z",
     "shell.execute_reply": "2021-05-26T15:52:23.400973Z"
    },
    "papermill": {
     "duration": 0.026957,
     "end_time": "2021-05-26T15:52:23.401092",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.374135",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "SEED = 7\n",
    "SPLIT_RATIOS = [0.6, 0.3, 0.1]\n",
    "\n",
    "\n",
    "def split_data(df):\n",
    "    # split data into train and test sets\n",
    "    seed = SEED\n",
    "    val_size = SPLIT_RATIOS[1]\n",
    "    test_size = SPLIT_RATIOS[2]\n",
    "\n",
    "    num_samples = df.shape[0]\n",
    "    X1 = df.values[:num_samples, 1:]  # keep only the features, skip the target, all rows\n",
    "    Y1 = df.values[:num_samples, :1]  # keep only the target, all rows\n",
    "\n",
    "    # Use split ratios to divide up into train/val/test\n",
    "    X_train, X_val, y_train, y_val = train_test_split(\n",
    "        X1, Y1, test_size=(test_size + val_size), random_state=seed\n",
    "    )\n",
    "    # Of the remaining non-training samples, give proper ratio to validation and to test\n",
    "    X_test, X_test, y_test, y_test = train_test_split(\n",
    "        X_val, y_val, test_size=(test_size / (test_size + val_size)), random_state=seed\n",
    "    )\n",
    "    # reassemble the datasets with target in first column and features after that\n",
    "    _train = np.concatenate([y_train, X_train], axis=1)\n",
    "    _val = np.concatenate([y_val, X_val], axis=1)\n",
    "    _test = np.concatenate([y_test, X_test], axis=1)\n",
    "\n",
    "    return _train, _val, _test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "equipped-belarus",
   "metadata": {
    "papermill": {
     "duration": 0.018452,
     "end_time": "2021-05-26T15:52:23.437956",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.419504",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Launch a single training job for a given housing location\n",
    "There is nothing specific to multi-model endpoints in terms of the models it will host. They are trained in the same way as all other SageMaker models. Here we are using the XGBoost estimator and not waiting for the job to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "junior-owner",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:23.492036Z",
     "iopub.status.busy": "2021-05-26T15:52:23.491505Z",
     "iopub.status.idle": "2021-05-26T15:52:23.493365Z",
     "shell.execute_reply": "2021-05-26T15:52:23.493704Z"
    },
    "papermill": {
     "duration": 0.03668,
     "end_time": "2021-05-26T15:52:23.493826",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.457146",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def launch_training_job(location):\n",
    "    # clear out old versions of the data\n",
    "    s3_bucket = s3.Bucket(BUCKET)\n",
    "    full_input_prefix = f\"{DATA_PREFIX}/model_prep/{location}\"\n",
    "    s3_bucket.objects.filter(Prefix=full_input_prefix + \"/\").delete()\n",
    "\n",
    "    # upload the entire set of data for all three channels\n",
    "    local_folder = f\"data/{location}\"\n",
    "    inputs = sagemaker_session.upload_data(path=local_folder, key_prefix=full_input_prefix)\n",
    "    print(f\"Training data uploaded: {inputs}\")\n",
    "\n",
    "    _job = \"xgb-{}\".format(location.replace(\"_\", \"-\"))\n",
    "    full_output_prefix = f\"{DATA_PREFIX}/model_artifacts/{location}\"\n",
    "    s3_output_path = f\"s3://{BUCKET}/{full_output_prefix}\"\n",
    "\n",
    "    xgb = sagemaker.estimator.Estimator(\n",
    "        XGBOOST_IMAGE,\n",
    "        role,\n",
    "        instance_count=1,\n",
    "        instance_type=TRAIN_INSTANCE_TYPE,\n",
    "        output_path=s3_output_path,\n",
    "        base_job_name=_job,\n",
    "        sagemaker_session=sagemaker_session,\n",
    "    )\n",
    "\n",
    "    xgb.set_hyperparameters(\n",
    "        max_depth=5,\n",
    "        eta=0.2,\n",
    "        gamma=4,\n",
    "        min_child_weight=6,\n",
    "        subsample=0.8,\n",
    "        verbosity=0,\n",
    "        early_stopping_rounds=5,\n",
    "        objective=\"reg:squarederror\",\n",
    "        num_round=25,\n",
    "    )\n",
    "\n",
    "    DISTRIBUTION_MODE = \"FullyReplicated\"\n",
    "\n",
    "    train_input = sagemaker.inputs.TrainingInput(\n",
    "        s3_data=inputs + \"/train\", distribution=DISTRIBUTION_MODE, content_type=\"csv\"\n",
    "    )\n",
    "\n",
    "    val_input = sagemaker.inputs.TrainingInput(\n",
    "        s3_data=inputs + \"/val\", distribution=DISTRIBUTION_MODE, content_type=\"csv\"\n",
    "    )\n",
    "\n",
    "    remote_inputs = {\"train\": train_input, \"validation\": val_input}\n",
    "\n",
    "    xgb.fit(remote_inputs, wait=False)\n",
    "\n",
    "    # Return the estimator object\n",
    "    return xgb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "false-lawrence",
   "metadata": {
    "papermill": {
     "duration": 0.018533,
     "end_time": "2021-05-26T15:52:23.531195",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.512662",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Kick off a model training job for each housing location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "worse-lying",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:23.573376Z",
     "iopub.status.busy": "2021-05-26T15:52:23.572906Z",
     "iopub.status.idle": "2021-05-26T15:52:23.574613Z",
     "shell.execute_reply": "2021-05-26T15:52:23.574948Z"
    },
    "papermill": {
     "duration": 0.025113,
     "end_time": "2021-05-26T15:52:23.575068",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.549955",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def save_data_locally(location, train, val, test):\n",
    "    os.makedirs(f\"data/{location}/train\")\n",
    "    np.savetxt(f\"data/{location}/train/{location}_train.csv\", train, delimiter=\",\", fmt=\"%.2f\")\n",
    "\n",
    "    os.makedirs(f\"data/{location}/val\")\n",
    "    np.savetxt(f\"data/{location}/val/{location}_val.csv\", val, delimiter=\",\", fmt=\"%.2f\")\n",
    "\n",
    "    os.makedirs(f\"data/{location}/test\")\n",
    "    np.savetxt(f\"data/{location}/test/{location}_test.csv\", test, delimiter=\",\", fmt=\"%.2f\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "classified-thunder",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:23.650728Z",
     "iopub.status.busy": "2021-05-26T15:52:23.636764Z",
     "iopub.status.idle": "2021-05-26T15:52:27.820539Z",
     "shell.execute_reply": "2021-05-26T15:52:27.820130Z"
    },
    "papermill": {
     "duration": 4.227067,
     "end_time": "2021-05-26T15:52:27.820645",
     "exception": false,
     "start_time": "2021-05-26T15:52:23.593578",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data uploaded: s3://sagemaker-us-west-2-688520471316/XGBOOST_BOSTON_HOUSING/model_prep/NewYork_NY\n",
      "Training data uploaded: s3://sagemaker-us-west-2-688520471316/XGBOOST_BOSTON_HOUSING/model_prep/LosAngeles_CA\n",
      "Training data uploaded: s3://sagemaker-us-west-2-688520471316/XGBOOST_BOSTON_HOUSING/model_prep/Chicago_IL\n",
      "Training data uploaded: s3://sagemaker-us-west-2-688520471316/XGBOOST_BOSTON_HOUSING/model_prep/Houston_TX\n",
      "\n",
      "4 training jobs launched: ['xgb-NewYork-NY-2021-05-28-20-27-47-850', 'xgb-LosAngeles-CA-2021-05-28-20-27-48-350', 'xgb-Chicago-IL-2021-05-28-20-27-51-370', 'xgb-Houston-TX-2021-05-28-20-27-53-744']\n"
     ]
    }
   ],
   "source": [
    "import shutil\n",
    "import os\n",
    "\n",
    "estimators = []\n",
    "\n",
    "shutil.rmtree(\"data\", ignore_errors=True)\n",
    "\n",
    "for loc in LOCATIONS[:PARALLEL_TRAINING_JOBS]:\n",
    "    _houses = gen_houses(NUM_HOUSES_PER_LOCATION)\n",
    "    _train, _val, _test = split_data(_houses)\n",
    "    save_data_locally(loc, _train, _val, _test)\n",
    "    estimator = launch_training_job(loc)\n",
    "    estimators.append(estimator)\n",
    "\n",
    "print()\n",
    "print(\n",
    "    f\"{len(estimators)} training jobs launched: {[x.latest_training_job.job_name for x in estimators]}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pregnant-individual",
   "metadata": {
    "papermill": {
     "duration": 0.019897,
     "end_time": "2021-05-26T15:52:27.860903",
     "exception": false,
     "start_time": "2021-05-26T15:52:27.841006",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Wait for all model training to finish"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "gross-society",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:27.904842Z",
     "iopub.status.busy": "2021-05-26T15:52:27.904334Z",
     "iopub.status.idle": "2021-05-26T15:52:27.906372Z",
     "shell.execute_reply": "2021-05-26T15:52:27.905978Z"
    },
    "papermill": {
     "duration": 0.025948,
     "end_time": "2021-05-26T15:52:27.906472",
     "exception": false,
     "start_time": "2021-05-26T15:52:27.880524",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def wait_for_training_job_to_complete(estimator):\n",
    "    job = estimator.latest_training_job.job_name\n",
    "    print(f\"Waiting for job: {job}\")\n",
    "    status = estimator.latest_training_job.describe()[\"TrainingJobStatus\"]\n",
    "    while status == \"InProgress\":\n",
    "        time.sleep(45)\n",
    "        status = estimator.latest_training_job.describe()[\"TrainingJobStatus\"]\n",
    "        if status == \"InProgress\":\n",
    "            print(f\"{job} job status: {status}\")\n",
    "    print(f\"DONE. Status for {job} is {status}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "specified-flood",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:52:27.950994Z",
     "iopub.status.busy": "2021-05-26T15:52:27.950570Z",
     "iopub.status.idle": "2021-05-26T15:56:58.599409Z",
     "shell.execute_reply": "2021-05-26T15:56:58.599823Z"
    },
    "papermill": {
     "duration": 270.673568,
     "end_time": "2021-05-26T15:56:58.599958",
     "exception": false,
     "start_time": "2021-05-26T15:52:27.926390",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for job: xgb-NewYork-NY-2021-05-28-20-27-47-850\n",
      "xgb-NewYork-NY-2021-05-28-20-27-47-850 job status: InProgress\n",
      "xgb-NewYork-NY-2021-05-28-20-27-47-850 job status: InProgress\n",
      "xgb-NewYork-NY-2021-05-28-20-27-47-850 job status: InProgress\n",
      "xgb-NewYork-NY-2021-05-28-20-27-47-850 job status: InProgress\n",
      "DONE. Status for xgb-NewYork-NY-2021-05-28-20-27-47-850 is Completed\n",
      "\n",
      "Waiting for job: xgb-LosAngeles-CA-2021-05-28-20-27-48-350\n",
      "DONE. Status for xgb-LosAngeles-CA-2021-05-28-20-27-48-350 is Completed\n",
      "\n",
      "Waiting for job: xgb-Chicago-IL-2021-05-28-20-27-51-370\n",
      "DONE. Status for xgb-Chicago-IL-2021-05-28-20-27-51-370 is Completed\n",
      "\n",
      "Waiting for job: xgb-Houston-TX-2021-05-28-20-27-53-744\n",
      "DONE. Status for xgb-Houston-TX-2021-05-28-20-27-53-744 is Completed\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for est in estimators:\n",
    "    wait_for_training_job_to_complete(est)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wrong-performer",
   "metadata": {
    "papermill": {
     "duration": 0.021394,
     "end_time": "2021-05-26T15:56:58.642731",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.621337",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Create the multi-model endpoint with the SageMaker SDK"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "interracial-knowing",
   "metadata": {
    "papermill": {
     "duration": 0.021378,
     "end_time": "2021-05-26T15:56:58.685363",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.663985",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Create a SageMaker Model from one of the Estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "forbidden-verse",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:56:58.732555Z",
     "iopub.status.busy": "2021-05-26T15:56:58.732087Z",
     "iopub.status.idle": "2021-05-26T15:56:58.755286Z",
     "shell.execute_reply": "2021-05-26T15:56:58.754836Z"
    },
    "papermill": {
     "duration": 0.048791,
     "end_time": "2021-05-26T15:56:58.755390",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.706599",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator = estimators[0]\n",
    "model = estimator.create_model(role=role, image_uri=XGBOOST_IMAGE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "incomplete-construction",
   "metadata": {
    "papermill": {
     "duration": 0.021606,
     "end_time": "2021-05-26T15:56:58.798545",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.776939",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Create the Amazon SageMaker MultiDataModel entity\n",
    "\n",
    "We create the multi-model endpoint using the [```MultiDataModel```](https://sagemaker.readthedocs.io/en/stable/api/inference/multi_data_model.html) class.\n",
    "\n",
    "You can create a MultiDataModel by directly passing in a `sagemaker.model.Model` object - in which case, the Endpoint will inherit information about the image to use, as well as any environmental variables, network isolation, etc., once the MultiDataModel is deployed.\n",
    "\n",
    "In addition, a MultiDataModel can also be created without explictly passing a `sagemaker.model.Model` object. Please refer to the documentation for additional details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "numeric-berry",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:56:58.844822Z",
     "iopub.status.busy": "2021-05-26T15:56:58.844241Z",
     "iopub.status.idle": "2021-05-26T15:56:58.846787Z",
     "shell.execute_reply": "2021-05-26T15:56:58.846422Z"
    },
    "papermill": {
     "duration": 0.026963,
     "end_time": "2021-05-26T15:56:58.846889",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.819926",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.multidatamodel import MultiDataModel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "electronic-kelly",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:56:58.892947Z",
     "iopub.status.busy": "2021-05-26T15:56:58.892485Z",
     "iopub.status.idle": "2021-05-26T15:56:58.894266Z",
     "shell.execute_reply": "2021-05-26T15:56:58.894606Z"
    },
    "papermill": {
     "duration": 0.026415,
     "end_time": "2021-05-26T15:56:58.894726",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.868311",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is where our MME will read models from on S3.\n",
    "model_data_prefix = f\"s3://{BUCKET}/{DATA_PREFIX}/{MULTI_MODEL_ARTIFACTS}/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "freelance-richmond",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:56:58.941336Z",
     "iopub.status.busy": "2021-05-26T15:56:58.940878Z",
     "iopub.status.idle": "2021-05-26T15:56:58.946592Z",
     "shell.execute_reply": "2021-05-26T15:56:58.946220Z"
    },
    "papermill": {
     "duration": 0.03048,
     "end_time": "2021-05-26T15:56:58.946693",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.916213",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "mme = MultiDataModel(\n",
    "    name=MODEL_NAME,\n",
    "    model_data_prefix=model_data_prefix,\n",
    "    model=model,  # passing our model - passes container image needed for the endpoint\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "thorough-oxide",
   "metadata": {
    "papermill": {
     "duration": 0.021504,
     "end_time": "2021-05-26T15:56:58.989865",
     "exception": false,
     "start_time": "2021-05-26T15:56:58.968361",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Deploy the Multi Model Endpoint\n",
    "\n",
    "You need to consider the appropriate instance type and number of instances for the projected prediction workload across all the models you plan to host behind your multi-model endpoint. The number and size of the individual models will also drive memory requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "wound-camping",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T15:56:59.038240Z",
     "iopub.status.busy": "2021-05-26T15:56:59.037801Z",
     "iopub.status.idle": "2021-05-26T16:04:31.002278Z",
     "shell.execute_reply": "2021-05-26T16:04:31.002630Z"
    },
    "papermill": {
     "duration": 451.991333,
     "end_time": "2021-05-26T16:04:31.002783",
     "exception": false,
     "start_time": "2021-05-26T15:56:59.011450",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------!"
     ]
    }
   ],
   "source": [
    "predictor = mme.deploy(\n",
    "    initial_instance_count=1, instance_type=ENDPOINT_INSTANCE_TYPE, endpoint_name=ENDPOINT_NAME\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dutch-culture",
   "metadata": {
    "papermill": {
     "duration": 0.025076,
     "end_time": "2021-05-26T16:04:31.052719",
     "exception": false,
     "start_time": "2021-05-26T16:04:31.027643",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Our endpoint has launched! Let's look at what models are available to the endpoint!\n",
    "\n",
    "By 'available', what we mean is, what model artfiacts are currently stored under the S3 prefix we defined when setting up the `MultiDataModel` above i.e. `model_data_prefix`.\n",
    "\n",
    "Currently, since we have no artifacts (i.e. `tar.gz` files) stored under  our defined S3 prefix, our endpoint, will have no models 'available' to serve inference requests.\n",
    "\n",
    "We will demonstrate how to make models 'available' to our endpoint below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "accepted-synthetic",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:31.113092Z",
     "iopub.status.busy": "2021-05-26T16:04:31.112617Z",
     "iopub.status.idle": "2021-05-26T16:04:31.236757Z",
     "shell.execute_reply": "2021-05-26T16:04:31.236290Z"
    },
    "papermill": {
     "duration": 0.159187,
     "end_time": "2021-05-26T16:04:31.236876",
     "exception": false,
     "start_time": "2021-05-26T16:04:31.077689",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# No models visible!\n",
    "list(mme.list_models())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "divided-portable",
   "metadata": {
    "papermill": {
     "duration": 0.025501,
     "end_time": "2021-05-26T16:04:31.287713",
     "exception": false,
     "start_time": "2021-05-26T16:04:31.262212",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Lets deploy model artifacts to be found by the endpoint\n",
    "\n",
    "We are now using the `.add_model()` method of the `MultiDataModel` to copy over our model artifacts from where they were initially stored, during training, to where our endpoint will source model artifacts for inference requests.\n",
    "\n",
    "`model_data_source` refers to the location of our model artifact (i.e. where it was deposited on S3 after training completed)\n",
    "\n",
    "`model_data_path` is the **relative** path to the S3 prefix we specified above (i.e. `model_data_prefix`) where our endpoint will source models for inference requests.\n",
    "\n",
    "Since this is a **relative** path, we can simply pass the name of what we wish to call the model artifact at inference time (i.e. `Chicago_IL.tar.gz`)\n",
    "\n",
    "### Dynamically deploying additional models\n",
    "\n",
    "It is also important to note, that we can always use the `.add_model()` method, as shown below, to dynamically deploy more models to the endpoint, to serve up inference requests as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "spare-jacket",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:31.343754Z",
     "iopub.status.busy": "2021-05-26T16:04:31.343299Z",
     "iopub.status.idle": "2021-05-26T16:04:32.094101Z",
     "shell.execute_reply": "2021-05-26T16:04:32.094471Z"
    },
    "papermill": {
     "duration": 0.78157,
     "end_time": "2021-05-26T16:04:32.094607",
     "exception": false,
     "start_time": "2021-05-26T16:04:31.313037",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "for est in estimators:\n",
    "    artifact_path = est.latest_training_job.describe()[\"ModelArtifacts\"][\"S3ModelArtifacts\"]\n",
    "    model_name = artifact_path.split(\"/\")[-4] + \".tar.gz\"\n",
    "    # This is copying over the model artifact to the S3 location for the MME.\n",
    "    mme.add_model(model_data_source=artifact_path, model_data_path=model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "general-tractor",
   "metadata": {
    "papermill": {
     "duration": 0.025434,
     "end_time": "2021-05-26T16:04:32.145605",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.120171",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## We have added the 4 model artifacts from our training jobs!\n",
    "\n",
    "We can see that the S3 prefix we specified when setting up `MultiDataModel` now has 4 model artifacts. As such, the endpoint can now serve up inference requests for these models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "small-disclosure",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:32.206383Z",
     "iopub.status.busy": "2021-05-26T16:04:32.205867Z",
     "iopub.status.idle": "2021-05-26T16:04:32.335273Z",
     "shell.execute_reply": "2021-05-26T16:04:32.335641Z"
    },
    "papermill": {
     "duration": 0.164864,
     "end_time": "2021-05-26T16:04:32.335776",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.170912",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Chicago_IL.tar.gz',\n",
       " 'Houston_TX.tar.gz',\n",
       " 'LosAngeles_CA.tar.gz',\n",
       " 'NewYork_NY.tar.gz']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(mme.list_models())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unusual-bradford",
   "metadata": {
    "papermill": {
     "duration": 0.025694,
     "end_time": "2021-05-26T16:04:32.387201",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.361507",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Get predictions from the endpoint\n",
    "\n",
    "Recall that ```mme.deploy()``` returns a [RealTimePredictor](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/predictor.py#L35) that we saved in a variable called ```predictor```.\n",
    "\n",
    "We will use ```predictor``` to submit requests to the endpoint.\n",
    "\n",
    "XGBoost supports ```text/csv``` for the content type and accept type. For more information on XGBoost Input/Output Interface, please see [here.](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html#InputOutput-XGBoost)\n",
    "\n",
    "Since the default RealTimePredictor does not have a serializer or deserializer set for requests, we will also set these.\n",
    "\n",
    "This will allow us to submit a python list for inference, and get back a float response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "devoted-complexity",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:32.442050Z",
     "iopub.status.busy": "2021-05-26T16:04:32.441607Z",
     "iopub.status.idle": "2021-05-26T16:04:32.443360Z",
     "shell.execute_reply": "2021-05-26T16:04:32.443669Z"
    },
    "papermill": {
     "duration": 0.030971,
     "end_time": "2021-05-26T16:04:32.443803",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.412832",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "opened-macedonia",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:32.498958Z",
     "iopub.status.busy": "2021-05-26T16:04:32.498390Z",
     "iopub.status.idle": "2021-05-26T16:04:32.500088Z",
     "shell.execute_reply": "2021-05-26T16:04:32.500474Z"
    },
    "papermill": {
     "duration": 0.031118,
     "end_time": "2021-05-26T16:04:32.500600",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.469482",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor.serializer = CSVSerializer()\n",
    "predictor.deserializer = JSONDeserializer()\n",
    "# predictor.content_type =predictor.content_type , removed as mentioned https://github.com/aws/sagemaker-python-sdk/blob/e8d16f8bc4c570f763f1129afc46ba3e0b98cdad/src/sagemaker/predictor.py#L82\n",
    "# predictor.accept = \"text/csv\" # removed also : https://github.com/aws/sagemaker-python-sdk/blob/e8d16f8bc4c570f763f1129afc46ba3e0b98cdad/src/sagemaker/predictor.py#L83"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "external-velvet",
   "metadata": {
    "papermill": {
     "duration": 0.025813,
     "end_time": "2021-05-26T16:04:32.552226",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.526413",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Invoking models on a multi-model endpoint\n",
    "Notice the higher latencies on the first invocation of any given model. This is due to the time it takes SageMaker to download the model to the Endpoint instance and then load the model into the inference container. Subsequent invocations of the same model take advantage of the model already being loaded into the inference container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "approximate-anthony",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:32.610285Z",
     "iopub.status.busy": "2021-05-26T16:04:32.609828Z",
     "iopub.status.idle": "2021-05-26T16:04:34.138639Z",
     "shell.execute_reply": "2021-05-26T16:04:34.138249Z"
    },
    "papermill": {
     "duration": 1.560633,
     "end_time": "2021-05-26T16:04:34.138746",
     "exception": false,
     "start_time": "2021-05-26T16:04:32.578113",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$395,909.25, took 1,469 ms\n",
      "\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "predicted_value = predictor.predict(data=gen_random_house()[1:], target_model=\"Chicago_IL.tar.gz\")\n",
    "\n",
    "duration = time.time() - start_time\n",
    "print(\"${:,.2f}, took {:,d} ms\\n\".format(predicted_value[0], int(duration * 1000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "smart-foundation",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:34.197277Z",
     "iopub.status.busy": "2021-05-26T16:04:34.196781Z",
     "iopub.status.idle": "2021-05-26T16:04:34.222623Z",
     "shell.execute_reply": "2021-05-26T16:04:34.222989Z"
    },
    "papermill": {
     "duration": 0.058109,
     "end_time": "2021-05-26T16:04:34.223125",
     "exception": false,
     "start_time": "2021-05-26T16:04:34.165016",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$372,641.38, took 23 ms\n",
      "\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "# Invoke endpoint\n",
    "predicted_value = predictor.predict(data=gen_random_house()[1:], target_model=\"Chicago_IL.tar.gz\")\n",
    "\n",
    "duration = time.time() - start_time\n",
    "print(\"${:,.2f}, took {:,d} ms\\n\".format(predicted_value[0], int(duration * 1000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "biblical-underwear",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:34.282025Z",
     "iopub.status.busy": "2021-05-26T16:04:34.281584Z",
     "iopub.status.idle": "2021-05-26T16:04:35.344346Z",
     "shell.execute_reply": "2021-05-26T16:04:35.344752Z"
    },
    "papermill": {
     "duration": 1.095017,
     "end_time": "2021-05-26T16:04:35.344889",
     "exception": false,
     "start_time": "2021-05-26T16:04:34.249872",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$344,676.03, took 1,145 ms\n",
      "\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "# Invoke endpoint\n",
    "predicted_value = predictor.predict(data=gen_random_house()[1:], target_model=\"Houston_TX.tar.gz\")\n",
    "\n",
    "duration = time.time() - start_time\n",
    "print(\"${:,.2f}, took {:,d} ms\\n\".format(predicted_value[0], int(duration * 1000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "nonprofit-ranking",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:35.404113Z",
     "iopub.status.busy": "2021-05-26T16:04:35.403635Z",
     "iopub.status.idle": "2021-05-26T16:04:35.430302Z",
     "shell.execute_reply": "2021-05-26T16:04:35.429895Z"
    },
    "papermill": {
     "duration": 0.058896,
     "end_time": "2021-05-26T16:04:35.430412",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.371516",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$479,065.41, took 19 ms\n",
      "\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "# Invoke endpoint\n",
    "predicted_value = predictor.predict(data=gen_random_house()[1:], target_model=\"Houston_TX.tar.gz\")\n",
    "\n",
    "duration = time.time() - start_time\n",
    "print(\"${:,.2f}, took {:,d} ms\\n\".format(predicted_value[0], int(duration * 1000)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "brown-competition",
   "metadata": {
    "papermill": {
     "duration": 0.027249,
     "end_time": "2021-05-26T16:04:35.484898",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.457649",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Updating a model\n",
    "To update a model, you would follow the same approach as above and add it as a new model. For example, if you have retrained the `NewYork_NY.tar.gz` model and wanted to start invoking it, you would upload the updated model artifacts behind the S3 prefix with a new name such as `NewYork_NY_v2.tar.gz`, and then change the `target_model` field to invoke `NewYork_NY_v2.tar.gz` instead of `NewYork_NY.tar.gz`. You do not want to overwrite the model artifacts in Amazon S3, because the old version of the model might still be loaded in the containers or on the storage volume of the instances on the endpoint. Invocations to the new model could then invoke the old version of the model.\n",
    "\n",
    "Alternatively, you could stop the endpoint and re-deploy a fresh set of models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funny-sunset",
   "metadata": {
    "papermill": {
     "duration": 0.027094,
     "end_time": "2021-05-26T16:04:35.540083",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.512989",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Using Boto APIs to invoke the endpoint\n",
    "\n",
    "While developing interactively within a Jupyter notebook, since `.deploy()` returns a `RealTimePredictor` it is a more seamless experience to start invoking your endpoint using the SageMaker SDK. You have more fine grained control over the serialization and deserialization protocols to shape your request and response payloads to/from the endpoint.\n",
    "\n",
    "This is great for iterative experimentation within a notebook. Furthermore, should you have an application that has access to the SageMaker SDK, you can always import `RealTimePredictor` and attach it to an existing endpoint - this allows you to stick to using the high level SDK if preferable.\n",
    "\n",
    "Additional documentation on `RealTimePredictor` can be found [here.](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html?highlight=RealTimePredictor#sagemaker.predictor.RealTimePredictor)\n",
    "\n",
    "The lower level Boto3 SDK may be preferable if you are attempting to invoke the endpoint as a part of a broader architecture.\n",
    "\n",
    "Imagine an API gateway frontend that uses a Lambda Proxy in order to transform request payloads before hitting a SageMaker Endpoint - in this example, Lambda does not have access to the SageMaker Python SDK, and as such, Boto3 can still allow you to interact with your endpoint and serve inference requests.\n",
    "\n",
    "Boto3 allows for quick injection of ML intelligence via SageMaker Endpoints into existing applications with minimal/no refactoring to existing code.\n",
    "\n",
    "Boto3 will submit your requests as a binary payload, while still allowing you to supply your desired `Content-Type` and `Accept` headers with serialization being handled by the inference container in the SageMaker Endpoint.\n",
    "\n",
    "Additional documentation on `.invoke_endpoint()` can be found [here.](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker-runtime.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "indirect-albania",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:35.602663Z",
     "iopub.status.busy": "2021-05-26T16:04:35.602023Z",
     "iopub.status.idle": "2021-05-26T16:04:35.603848Z",
     "shell.execute_reply": "2021-05-26T16:04:35.604194Z"
    },
    "papermill": {
     "duration": 0.037229,
     "end_time": "2021-05-26T16:04:35.604327",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.567098",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "\n",
    "runtime_sm_client = boto3.client(service_name=\"sagemaker-runtime\")\n",
    "\n",
    "\n",
    "def predict_one_house_value(features, model_name):\n",
    "    print(f\"Using model {model_name} to predict price of this house: {features}\")\n",
    "\n",
    "    # Notice how we alter the list into a string as the payload\n",
    "    body = \",\".join(map(str, features)) + \"\\n\"\n",
    "\n",
    "    start_time = time.time()\n",
    "\n",
    "    response = runtime_sm_client.invoke_endpoint(\n",
    "        EndpointName=ENDPOINT_NAME, ContentType=\"text/csv\", TargetModel=model_name, Body=body\n",
    "    )\n",
    "\n",
    "    predicted_value = json.loads(response[\"Body\"].read())[0]\n",
    "\n",
    "    duration = time.time() - start_time\n",
    "\n",
    "    print(\"${:,.2f}, took {:,d} ms\\n\".format(predicted_value, int(duration * 1000)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "infrared-shaft",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:35.664075Z",
     "iopub.status.busy": "2021-05-26T16:04:35.663613Z",
     "iopub.status.idle": "2021-05-26T16:04:35.714098Z",
     "shell.execute_reply": "2021-05-26T16:04:35.714450Z"
    },
    "papermill": {
     "duration": 0.082837,
     "end_time": "2021-05-26T16:04:35.714585",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.631748",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "predict_one_house_value(gen_random_house()[1:], \"Chicago_IL.tar.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cleared-peripheral",
   "metadata": {
    "papermill": {
     "duration": 0.027686,
     "end_time": "2021-05-26T16:04:35.769782",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.742096",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Clean up\n",
    "Here, to be sure we are not billed for endpoints we are no longer using, we clean up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "innocent-frequency",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-26T16:04:35.829381Z",
     "iopub.status.busy": "2021-05-26T16:04:35.828888Z",
     "iopub.status.idle": "2021-05-26T16:04:35.984391Z",
     "shell.execute_reply": "2021-05-26T16:04:35.983963Z"
    },
    "papermill": {
     "duration": 0.187229,
     "end_time": "2021-05-26T16:04:35.984538",
     "exception": false,
     "start_time": "2021-05-26T16:04:35.797309",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictor.delete_model()\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ffb0f6dd",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/advanced_functionality|multi_model_xgboost_home_value|xgboost_multi_model_endpoint_home_value.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 736.016806,
   "end_time": "2021-05-26T16:04:36.807261",
   "environment_variables": {},
   "exception": true,
   "input_path": "xgboost_multi_model_endpoint_home_value.ipynb",
   "output_path": "/opt/ml/processing/output/xgboost_multi_model_endpoint_home_value-2021-05-26-15-47-56.ipynb",
   "parameters": {
    "kms_key": "arn:aws:kms:us-west-2:521695447989:key/6e9984db-50cf-4c7e-926c-877ec47a8b25"
   },
   "start_time": "2021-05-26T15:52:20.790455",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}