resources/Shadow-deployment-async-process-demo-breast-cancer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "42b5e80b-ad1d-4335-a1f7-10a91127e3dc"
    }
   },
   "source": [
    "# Shadow Deployment for Asynchronous process, with Breast Cancer Prediction Model  \n",
    "\n",
    "A **Shadow deployment** consists of releasing version B alongside version A, fork version A's incoming requests and send them to version B as well without impacting production traffic. This is particularly useful to test production load on a new feature. \n",
    "\n",
    "In this notebook, we will utilize the **Data Capture** utility in ***[SageMaker Model Monitor](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html)***. Model Monitor continuously monitors the quality of Amazon SageMaker machine learning models in production. With Model Monitor, you can set alerts that notify you when there are deviations in the model quality. Early and proactive detection of these deviations enables you to take corrective actions, such as retraining models, auditing upstream systems, or fixing quality issues without having to monitor models manually or build additional tooling.\n",
    "\n",
    "For **Shadow deployment** we will enable data capture and turn on the model monitor for real-time inference endpoint for model version 1 to capture data from requests and responses and store the captured data in Amazon S3 bucket. Using the file that data capture generates (input data), use batch transform to get inference for model version 2. Optionally, we can use Amazon Athena and Amazon Quicksight to prepare dashboard and gain insights from the inferences or simply run a hash compare between the two-inference data to show the differences.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by specifying:\n",
    "\n",
    "- The SageMaker role arn used to give learning and hosting access to your data. The snippet below will use the same role used by your SageMaker notebook instance, if you're using other. Otherwise, specify the full ARN of a role with the SageMakerFullAccess policy attached.\n",
    "- The S3 bucket that you want to use for training and storing model objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "nbpresent": {
     "id": "6427e831-8f89-45c0-b150-0b134397d79a"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import re\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import io\n",
    "import time\n",
    "import json\n",
    "import csv\n",
    "import math\n",
    "import sagemaker.amazon.common as smac\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "# S3 bucket for saving code and model artifacts.\n",
    "# Feel free to specify a different bucket and prefix\n",
    "sess = sagemaker.Session()\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "\n",
    "prefix = 'sagemaker/shadow-breast-cancer-prediction' # place to upload training files within the bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "142777ae-c072-448e-b941-72bc75735d01"
    }
   },
   "source": [
    "---\n",
    "## Data\n",
    "\n",
    "Data Source: https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data\n",
    "        https://www.kaggle.com/uciml/breast-cancer-wisconsin-data\n",
    "\n",
    "Let's download the data and save it in the local folder with the name data.csv and take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "f8976dad-6897-4c7e-8c95-ae2f53070ef5"
    }
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data', header = None)\n",
    "\n",
    "# specify columns extracted from wbdc.names\n",
    "data.columns = [\"id\",\"diagnosis\",\"radius_mean\",\"texture_mean\",\"perimeter_mean\",\"area_mean\",\"smoothness_mean\",\n",
    "                \"compactness_mean\",\"concavity_mean\",\"concave points_mean\",\"symmetry_mean\",\"fractal_dimension_mean\",\n",
    "                \"radius_se\",\"texture_se\",\"perimeter_se\",\"area_se\",\"smoothness_se\",\"compactness_se\",\"concavity_se\",\n",
    "                \"concave points_se\",\"symmetry_se\",\"fractal_dimension_se\",\"radius_worst\",\"texture_worst\",\n",
    "                \"perimeter_worst\",\"area_worst\",\"smoothness_worst\",\"compactness_worst\",\"concavity_worst\",\n",
    "                \"concave points_worst\",\"symmetry_worst\",\"fractal_dimension_worst\"] \n",
    "\n",
    "# save the data\n",
    "data.to_csv(\"data.csv\", sep=',', index=False)\n",
    "\n",
    "# print the shape of the data file\n",
    "print(data.shape)\n",
    "\n",
    "# show the top few rows\n",
    "display(data.head())\n",
    "\n",
    "# describe the data object\n",
    "display(data.describe())\n",
    "\n",
    "# we will also summarize the categorical field diganosis \n",
    "display(data.diagnosis.value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Key observations:\n",
    "* Data has 569 observations and 32 columns.\n",
    "* First field is 'id'.\n",
    "* Second field, 'diagnosis', is an indicator of the actual diagnosis ('M' = Malignant; 'B' = Benign).\n",
    "* There are 30 other numeric features available for prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Features and Labels\n",
    "#### Split the data into 80% training, 10% validation and 10% testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rand_split = np.random.rand(len(data))\n",
    "train_list = rand_split < 0.8\n",
    "val_list = (rand_split >= 0.8) & (rand_split < 0.9)\n",
    "test_list = rand_split >= 0.9\n",
    "\n",
    "data_train = data[train_list]\n",
    "data_val = data[val_list]\n",
    "data_test = data[test_list]\n",
    "\n",
    "train_y = ((data_train.iloc[:,1] == 'M') +0).to_numpy();\n",
    "train_X = data_train.iloc[:,2:].to_numpy();\n",
    "\n",
    "val_y = ((data_val.iloc[:,1] == 'M') +0).to_numpy();\n",
    "val_X = data_val.iloc[:,2:].to_numpy();\n",
    "\n",
    "test_y = ((data_test.iloc[:,1] == 'M') +0).to_numpy();\n",
    "test_X = data_test.iloc[:,2:].to_numpy();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write test data to a csv file\n",
    "\n",
    "pd.DataFrame(test_X).to_csv(\"test_sample.csv\",header=None, index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "ff9d10f9-b611-423b-80da-6dcdafd1c8b9"
    }
   },
   "source": [
    "Now, we'll convert the datasets to the recordIO-wrapped protobuf format used by the Amazon SageMaker algorithms, and then upload this data to S3.  We'll start with training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "cd8e3431-79d9-40b6-91d1-d67cd61894e7"
    }
   },
   "outputs": [],
   "source": [
    "train_file = 'linear_train.data'\n",
    "\n",
    "f = io.BytesIO()\n",
    "smac.write_numpy_to_dense_tensor(f, train_X.astype('float32'), train_y.astype('float32'))\n",
    "f.seek(0)\n",
    "\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train', train_file)).upload_fileobj(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "71cbcebd-a2a5-419e-8e50-b2bc0909f564"
    }
   },
   "source": [
    "Next we'll convert and upload the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "bd113b8e-adc1-4091-a26f-a426149fe604"
    }
   },
   "outputs": [],
   "source": [
    "validation_file = 'linear_validation.data'\n",
    "\n",
    "f = io.BytesIO()\n",
    "smac.write_numpy_to_dense_tensor(f, val_X.astype('float32'), val_y.astype('float32'))\n",
    "f.seek(0)\n",
    "\n",
    "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'validation', validation_file)).upload_fileobj(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "f3b125ad-a2d5-464c-8cfa-bd203034eee4"
    }
   },
   "source": [
    "---\n",
    "## Train\n",
    "\n",
    "Now we can begin to specify our linear model.  Amazon SageMaker's Linear Learner actually fits many models in parallel, each with slightly different hyperparameters, and then returns the one with the best fit.  This functionality is automatically enabled.  We can influence this using parameters like:\n",
    "\n",
    "- `num_models` to increase to total number of models run.  The specified parameters will always be one of those models, but the algorithm also chooses models with nearby parameter values in order to find a solution nearby that may be more optimal.  In this case, we're going to use the max of 32.\n",
    "- `loss` which controls how we penalize mistakes in our model estimates.  For this case, let's use absolute loss as we haven't spent much time cleaning the data, and absolute loss will be less sensitive to outliers.\n",
    "- `wd` or `l1` which control regularization.  Regularization can prevent model overfitting by preventing our estimates from becoming too finely tuned to the training data, which can actually hurt generalizability.  In this case, we'll leave these parameters as their default \"auto\" though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify container images used for training and hosting SageMaker's linear-learner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# See 'Algorithms Provided by Amazon SageMaker: Common Parameters' in the SageMaker documentation for an explanation of these values.\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "container = get_image_uri(boto3.Session().region_name, 'linear-learner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup shadow model v1 training params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "397fb60a-c48b-453f-88ea-4d832b70c919"
    }
   },
   "outputs": [],
   "source": [
    "linear_job_v1 = 'breast-cancer-model-v1-' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "\n",
    "print(\"Job name is:\", linear_job_v1)\n",
    "\n",
    "linear_training_params_v1 = {\n",
    "    \"RoleArn\": role,\n",
    "    \"TrainingJobName\": linear_job_v1,\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": container,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,\n",
    "        \"InstanceType\": \"ml.c4.2xlarge\",\n",
    "        \"VolumeSizeInGB\": 10\n",
    "    },\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": \"s3://{}/{}/train/\".format(bucket, prefix),\n",
    "                    \"S3DataDistributionType\": \"ShardedByS3Key\"\n",
    "                }\n",
    "            },\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"RecordWrapperType\": \"None\"\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": \"s3://{}/{}/validation/\".format(bucket, prefix),\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"RecordWrapperType\": \"None\"\n",
    "        }\n",
    "\n",
    "    ],\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": \"s3://{}/{}/\".format(bucket, prefix)\n",
    "    },\n",
    "    \"HyperParameters\": {\n",
    "        \"feature_dim\": \"30\",\n",
    "        \"mini_batch_size\": \"100\",\n",
    "        \"predictor_type\": \"regressor\",\n",
    "        \"epochs\": \"10\",\n",
    "        \"num_models\": \"32\",\n",
    "        \"loss\": \"absolute_loss\"\n",
    "    },\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 60 * 60\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup shadow model v2 training params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_job_v2 = 'breast-cancer-model-v2-' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "print(\"Job name is:\", linear_job_v2)\n",
    "\n",
    "linear_training_params_v2 = {\n",
    "    \"RoleArn\": role,\n",
    "    \"TrainingJobName\": linear_job_v2,\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": container,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,\n",
    "        \"InstanceType\": \"ml.c4.2xlarge\",\n",
    "        \"VolumeSizeInGB\": 10\n",
    "    },\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": \"s3://{}/{}/train/\".format(bucket, prefix),\n",
    "                    \"S3DataDistributionType\": \"ShardedByS3Key\"\n",
    "                }\n",
    "            },\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"RecordWrapperType\": \"None\"\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": \"s3://{}/{}/validation/\".format(bucket, prefix),\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"RecordWrapperType\": \"None\"\n",
    "        }\n",
    "\n",
    "    ],\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": \"s3://{}/{}/\".format(bucket, prefix)\n",
    "    },\n",
    "    \"HyperParameters\": {\n",
    "        \"feature_dim\": \"30\",\n",
    "        \"mini_batch_size\": \"50\",\n",
    "        \"predictor_type\": \"regressor\",\n",
    "        \"epochs\": \"5\",\n",
    "        \"num_models\": \"16\",\n",
    "        \"loss\": \"huber_loss\"\n",
    "    },\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 60 * 60\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's kick off our training job in SageMaker's distributed, managed training, using the parameters we just created.  Because training is managed, we don't have to wait for our job to finish to continue, but for this case, let's use boto3's 'training_job_completed_or_stopped' waiter so we can ensure that the job has been started."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create shadow model v1 training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "sm = boto3.client('sagemaker')\n",
    "\n",
    "sm.create_training_job(**linear_training_params_v1)\n",
    "\n",
    "status = sm.describe_training_job(TrainingJobName=linear_job_v1)['TrainingJobStatus']\n",
    "print(status)\n",
    "sm.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=linear_job_v1)\n",
    "if status == 'Failed':\n",
    "    message = sm.describe_training_job(TrainingJobName=linear_job_v1)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))\n",
    "    raise Exception('Training job failed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create shadow model v2 training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "sm = boto3.client('sagemaker')\n",
    "\n",
    "sm.create_training_job(**linear_training_params_v2)\n",
    "\n",
    "status = sm.describe_training_job(TrainingJobName=linear_job_v2)['TrainingJobStatus']\n",
    "print(status)\n",
    "sm.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=linear_job_v2)\n",
    "if status == 'Failed':\n",
    "    message = sm.describe_training_job(TrainingJobName=linear_job_v2)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))\n",
    "    raise Exception('Training job failed')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "2adcc348-9ab5-4a8a-8139-d0ecd740208a"
    }
   },
   "source": [
    "---\n",
    "## Host\n",
    "\n",
    "Now that we've trained the linear algorithm on our data, let's setup a model which can later be hosted.  We will:\n",
    "1. Point to the scoring container\n",
    "1. Point to the model.tar.gz that came from training\n",
    "1. Create the hosting model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Deploy the model-1 to Amazon Sagemaker\n",
    "\n",
    "from time import gmtime, strftime\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "model1_url = 'https://{}.s3-{}.amazonaws.com/{}/{}/output/model.tar.gz'.format(bucket, region, prefix,linear_job_v1)\n",
    "image_uri1 = get_image_uri(boto3.Session().region_name, 'linear-learner')\n",
    "model1 = Model(image_uri1, model_data=model1_url, role=role)\n",
    "\n",
    "print(model1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Deploy the model-2 to Amazon Sagemaker\n",
    "\n",
    "model2_url = 'https://{}.s3-{}.amazonaws.com/{}/{}/output/model.tar.gz'.format(bucket, region, prefix,linear_job_v2)\n",
    "image_uri2 = get_image_uri(boto3.Session().region_name, 'linear-learner')\n",
    "model2 = Model(image_uri2, model_data=model2_url, role=role)\n",
    "\n",
    "print(model2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_capture_prefix = '{}/datacapture'.format(prefix)\n",
    "s3_capture_upload_path = 's3://{}/{}'.format(bucket, data_capture_prefix)\n",
    "reports_prefix = '{}/reports'.format(prefix)\n",
    "s3_report_path = 's3://{}/{}'.format(bucket,reports_prefix)\n",
    "code_prefix = '{}/code'.format(prefix)\n",
    "\n",
    "print(\"Capture path: {}\".format(s3_capture_upload_path))\n",
    "print(\"Report path: {}\".format(s3_report_path))\n",
    "print(\"Default bucket is...\", bucket)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enable Model Monitoring with Data Capture for model v1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create DataCaptureConfig and Enable DataCapture for model v1\n",
    "\n",
    "from sagemaker.model_monitor import DataCaptureConfig\n",
    "\n",
    "endpoint_name1 = 'Shadow-breast-cancer-1-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(\"EndpointName={}\".format(endpoint_name1))\n",
    "\n",
    "data_capture_config = DataCaptureConfig(\n",
    "                        enable_capture=True,\n",
    "                        sampling_percentage=100,\n",
    "                        destination_s3_uri=s3_capture_upload_path)\n",
    "\n",
    "predictor = model1.deploy(initial_instance_count=1,\n",
    "                    instance_type='ml.m4.xlarge',\n",
    "                    endpoint_name=endpoint_name1,\n",
    "                    data_capture_config=data_capture_config)\n",
    "\n",
    "print(model1.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy model v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Deploy mmodel v2\n",
    "\n",
    "endpoint_name2 = 'Shadow-breast-cancer-2-' + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "print(\"EndpointName={}\".format(endpoint_name2))\n",
    "\n",
    "predictor2 = model2.deploy(initial_instance_count=1,\n",
    "                    instance_type='ml.m4.xlarge',\n",
    "                    endpoint_name=endpoint_name2)\n",
    "\n",
    "print(model2.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.serializers import CSVSerializer\n",
    "from sagemaker.deserializers import JSONDeserializer\n",
    "from sagemaker.predictor import Predictor\n",
    "\n",
    "predictor = Predictor(endpoint_name=endpoint_name1, \n",
    "                        sagemaker_session=sess,\n",
    "                        serializer=CSVSerializer(),            \n",
    "                        deserializers=JSONDeserializer())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import Predictor\n",
    "    \n",
    "print(\"Sending data to the endpoint\")\n",
    "with open('test_sample.csv', 'r') as f:\n",
    "    for row in f:\n",
    "        payload = row.rstrip('\\n')\n",
    "        response = predictor.predict(payload)\n",
    "        time.sleep(1)\n",
    "        \n",
    "print(\"Done!\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View captured data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Note: It takes a few seconds for the capture data to appear in S3\n",
    "\n",
    "s3_client = boto3.Session().client('s3')\n",
    "current_endpoint_capture_prefix = '{}/{}'.format(data_capture_prefix, endpoint_name1)\n",
    "print(current_endpoint_capture_prefix)\n",
    "result = s3_client.list_objects(Bucket=bucket, Prefix=current_endpoint_capture_prefix)\n",
    "capture_files = [capture_file.get(\"Key\") for capture_file in result.get('Contents')]\n",
    "print(\"Found Capture Files:\")\n",
    "print(\"\\n \".join(capture_files))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_obj_body(obj_key):\n",
    "    return s3_client.get_object(Bucket=bucket, Key=obj_key).get('Body').read().decode(\"utf-8\")\n",
    "\n",
    "capture_file = get_obj_body(capture_files[-1])\n",
    "print(capture_file[:2000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "print(json.dumps(json.loads(capture_file.split('\\n')[0]), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the Captured Input from Real time Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Save the input data from DataCapture to send to Batch Prediction \n",
    "\n",
    "with open('Captured_input.csv','w') as file:\n",
    "    for i in range(20):\n",
    "        data = json.loads(capture_file.split('\\n')[i])\n",
    "        file.write(data[\"captureData\"][\"endpointInput\"][\"data\"])\n",
    "        file.write('\\n')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the output from Real time Prediction for comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('RealTimePrediction_output.csv','w') as file:\n",
    "    for i in range(20):\n",
    "        data = json.loads(capture_file.split('\\n')[i])\n",
    "        file.write(data[\"captureData\"][\"endpointOutput\"][\"data\"])\n",
    "        file.write('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!printf \"\\n\\nShowing first five lines\\n\\n\"    \n",
    "!head -n 5 Captured_input.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Upload Captured Input file to S3 bucket\n",
    "batch_input_loc = sess.upload_data(path='Captured_input.csv', bucket=bucket, key_prefix='batch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import gmtime, strftime\n",
    "import time\n",
    "\n",
    "timestamp_prefix = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "\n",
    "input_data_path = 's3://{}/{}/{}'.format(bucket, 'batch', 'Captured_input.csv')\n",
    "output_prefix = '{}/{}'.format('batch_output', timestamp_prefix)\n",
    "output_data_path = 's3://{}/{}'.format(bucket,output_prefix)\n",
    "\n",
    "#output_data_path = 's3://{}/{}/{}'.format(bucket, 'batch_output', timestamp_prefix)\n",
    "job_name = 'shadow-deployment-job-' + timestamp_prefix\n",
    "\n",
    "print(input_data_path)\n",
    "print(output_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Transform on model v2 using the captured data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create Batch transform function with model v2\n",
    "\n",
    "transformer = sagemaker.transformer.Transformer(\n",
    "    model_name = model2.name,\n",
    "    instance_count = 1,\n",
    "    instance_type = 'ml.m5.xlarge',\n",
    "    strategy = 'SingleRecord',\n",
    "    assemble_with = 'Line',\n",
    "    output_path = output_data_path,\n",
    "    base_transform_job_name='serial-inference-batch ',\n",
    "    sagemaker_session=sess,\n",
    "    accept = 'text/csv'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Calling Batch transform with Captured data from RealTime Prediction\n",
    "\n",
    "transformer.transform(data = input_data_path,\n",
    "                      job_name = job_name,\n",
    "                      content_type = 'text/csv',\n",
    "                      split_type = 'Line')\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Download the output file from Batch Transform\n",
    "\n",
    "s3_file_path= '{}/Captured_input.csv.out'.format(output_prefix)\n",
    "save_as = 'BatchPrediction_output.csv'\n",
    "\n",
    "try:\n",
    "    s3_client.download_file(bucket,s3_file_path, save_as)\n",
    "    print(\"File downloaded successfully!\")\n",
    "except:\n",
    "    print(\"File not found!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare the difference between Realtime and Batch Inference outputs for the two models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Realtime_predictions = []\n",
    "Batch_predictions = []\n",
    "\n",
    "with open('RealTimePrediction_output.csv') as read_obj:\n",
    "    csv_reader = csv.reader(read_obj)\n",
    "    for row in csv_reader:\n",
    "        data = json.loads(row[0])\n",
    "        Realtime_predictions.append(data[\"predictions\"][0][\"score\"])\n",
    "        \n",
    "print(\"Realtime predictions - \",Realtime_predictions)\n",
    "\n",
    "with open('BatchPrediction_output.csv') as read_obj:\n",
    "    csv_reader = csv.reader(read_obj)\n",
    "    for row in csv_reader:\n",
    "        Batch_predictions.append(float(row[0]))\n",
    "        \n",
    "print(\"Batch predictions - \",Batch_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_values(list1,list2):\n",
    "    difference = []\n",
    "    if len(list1) != len(list2):\n",
    "        print(\"Predictions are not of the same sample size\")\n",
    "    else:\n",
    "        print(\"The sample size is same..let's proceed with comparison..\")\n",
    "        zip_object = zip(list1, list2)\n",
    "        \n",
    "        for list1_i, list2_i in zip_object:\n",
    "            difference.append(abs(list1_i-list2_i))\n",
    "            \n",
    "    return difference \n",
    "\n",
    "       \n",
    "diff = compare_values(Realtime_predictions,Batch_predictions)\n",
    "print(diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can use Amazon Athena and Amazon Quicksight to prepare dashboard and gain insights from the inferences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the resources\n",
    "\n",
    "You can keep your endpoint running to continue capturing data. If you do not plan to collect more data or use this endpoint further, you should delete the endpoint to avoid incurring additional charges. Note that deleting your endpoint does not delete the data that was captured during the model invocations. That data persists in Amazon S3 until you delete it yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sm.delete_endpoint(EndpointName=endpoint_name1)\n",
    "#sm.delete_endpoint(EndpointName=endpoint_name2)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the License). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the license file accompanying this file. This file is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}