 archived/Batch Transform - breast cancer prediction with lowel level SDK.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "42b5e80b-ad1d-4335-a1f7-10a91127e3dc"
    }
   },
   "source": [
    "# Amazon SageMaker Batch Transform: Associate prediction results with their corresponding input records\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "42b5e80b-ad1d-4335-a1f7-10a91127e3dc"
    }
   },
   "source": [
    "_**Use SageMaker's XGBoost to train a binary classification model and for a list of tumors in batch file, predict if each is malignant**_\n",
    "\n",
    "_**It also shows how to use the input output joining / filter feature in Batch transform in details**_\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "## Background\n",
    "This purpose of this notebook is to train a model using SageMaker's XGBoost and UCI's breast cancer diagnostic data set to illustrate at how to run batch inferences and how to use the Batch Transform I/O join feature. UCI's breast cancer diagnostic data set is available at https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29. The data set is also available on Kaggle at https://www.kaggle.com/uciml/breast-cancer-wisconsin-data. The purpose here is to use this data set to build a predictve model of whether a breast mass image indicates benign or malignant tumor. \n",
    "\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Setup\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "* The SageMaker role arn used to give training and batch transform access to your data. The snippet below will use the same role used by your SageMaker notebook instance. Otherwise, specify the full ARN of a role with the SageMakerFullAccess policy attached.\n",
    "* The S3 bucket that you want to use for training and storing model objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "isConfigCell": true,
    "nbpresent": {
     "id": "6427e831-8f89-45c0-b150-0b134397d79a"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "prefix = \"DEMO-breast-cancer-prediction-xgboost-lowlevel\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "142777ae-c072-448e-b941-72bc75735d01"
    }
   },
   "source": [
    "---\n",
    "## Data sources\n",
    "\n",
    "> Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.\n",
    "\n",
    "> Breast Cancer Wisconsin (Diagnostic) Data Set [https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)].\n",
    "\n",
    "> _Also see:_ Breast Cancer Wisconsin (Diagnostic) Data Set [https://www.kaggle.com/uciml/breast-cancer-wisconsin-data].\n",
    "\n",
    "## Data preparation\n",
    "\n",
    "\n",
    "Let's download the data and save it in the local folder with the name data.csv and take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "nbpresent": {
     "id": "f8976dad-6897-4c7e-8c95-ae2f53070ef5"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>861799</td>\n",
       "      <td>M</td>\n",
       "      <td>15.370</td>\n",
       "      <td>22.76</td>\n",
       "      <td>100.20</td>\n",
       "      <td>728.2</td>\n",
       "      <td>0.09200</td>\n",
       "      <td>0.10360</td>\n",
       "      <td>0.11220</td>\n",
       "      <td>0.07483</td>\n",
       "      <td>...</td>\n",
       "      <td>16.43</td>\n",
       "      <td>25.84</td>\n",
       "      <td>107.50</td>\n",
       "      <td>830.9</td>\n",
       "      <td>0.1257</td>\n",
       "      <td>0.1997</td>\n",
       "      <td>0.2846</td>\n",
       "      <td>0.14760</td>\n",
       "      <td>0.2556</td>\n",
       "      <td>0.06828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>395</th>\n",
       "      <td>903811</td>\n",
       "      <td>B</td>\n",
       "      <td>14.060</td>\n",
       "      <td>17.18</td>\n",
       "      <td>89.75</td>\n",
       "      <td>609.1</td>\n",
       "      <td>0.08045</td>\n",
       "      <td>0.05361</td>\n",
       "      <td>0.02681</td>\n",
       "      <td>0.03251</td>\n",
       "      <td>...</td>\n",
       "      <td>14.92</td>\n",
       "      <td>25.34</td>\n",
       "      <td>96.42</td>\n",
       "      <td>684.5</td>\n",
       "      <td>0.1066</td>\n",
       "      <td>0.1231</td>\n",
       "      <td>0.0846</td>\n",
       "      <td>0.07911</td>\n",
       "      <td>0.2523</td>\n",
       "      <td>0.06609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>8813129</td>\n",
       "      <td>B</td>\n",
       "      <td>13.270</td>\n",
       "      <td>17.02</td>\n",
       "      <td>84.55</td>\n",
       "      <td>546.4</td>\n",
       "      <td>0.08445</td>\n",
       "      <td>0.04994</td>\n",
       "      <td>0.03554</td>\n",
       "      <td>0.02456</td>\n",
       "      <td>...</td>\n",
       "      <td>15.14</td>\n",
       "      <td>23.60</td>\n",
       "      <td>98.84</td>\n",
       "      <td>708.8</td>\n",
       "      <td>0.1276</td>\n",
       "      <td>0.1311</td>\n",
       "      <td>0.1786</td>\n",
       "      <td>0.09678</td>\n",
       "      <td>0.2506</td>\n",
       "      <td>0.07623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>406</th>\n",
       "      <td>905189</td>\n",
       "      <td>B</td>\n",
       "      <td>16.140</td>\n",
       "      <td>14.86</td>\n",
       "      <td>104.30</td>\n",
       "      <td>800.0</td>\n",
       "      <td>0.09495</td>\n",
       "      <td>0.08501</td>\n",
       "      <td>0.05500</td>\n",
       "      <td>0.04528</td>\n",
       "      <td>...</td>\n",
       "      <td>17.71</td>\n",
       "      <td>19.58</td>\n",
       "      <td>115.90</td>\n",
       "      <td>947.9</td>\n",
       "      <td>0.1206</td>\n",
       "      <td>0.1722</td>\n",
       "      <td>0.2310</td>\n",
       "      <td>0.11290</td>\n",
       "      <td>0.2778</td>\n",
       "      <td>0.07012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>388</th>\n",
       "      <td>903011</td>\n",
       "      <td>B</td>\n",
       "      <td>11.270</td>\n",
       "      <td>15.50</td>\n",
       "      <td>73.38</td>\n",
       "      <td>392.0</td>\n",
       "      <td>0.08365</td>\n",
       "      <td>0.11140</td>\n",
       "      <td>0.10070</td>\n",
       "      <td>0.02757</td>\n",
       "      <td>...</td>\n",
       "      <td>12.04</td>\n",
       "      <td>18.93</td>\n",
       "      <td>79.73</td>\n",
       "      <td>450.0</td>\n",
       "      <td>0.1102</td>\n",
       "      <td>0.2809</td>\n",
       "      <td>0.3021</td>\n",
       "      <td>0.08272</td>\n",
       "      <td>0.2157</td>\n",
       "      <td>0.10430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>868682</td>\n",
       "      <td>B</td>\n",
       "      <td>11.430</td>\n",
       "      <td>15.39</td>\n",
       "      <td>73.06</td>\n",
       "      <td>399.8</td>\n",
       "      <td>0.09639</td>\n",
       "      <td>0.06889</td>\n",
       "      <td>0.03503</td>\n",
       "      <td>0.02875</td>\n",
       "      <td>...</td>\n",
       "      <td>12.32</td>\n",
       "      <td>22.02</td>\n",
       "      <td>79.93</td>\n",
       "      <td>462.0</td>\n",
       "      <td>0.1190</td>\n",
       "      <td>0.1648</td>\n",
       "      <td>0.1399</td>\n",
       "      <td>0.08476</td>\n",
       "      <td>0.2676</td>\n",
       "      <td>0.06765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>859471</td>\n",
       "      <td>B</td>\n",
       "      <td>9.029</td>\n",
       "      <td>17.33</td>\n",
       "      <td>58.79</td>\n",
       "      <td>250.5</td>\n",
       "      <td>0.10660</td>\n",
       "      <td>0.14130</td>\n",
       "      <td>0.31300</td>\n",
       "      <td>0.04375</td>\n",
       "      <td>...</td>\n",
       "      <td>10.31</td>\n",
       "      <td>22.65</td>\n",
       "      <td>65.50</td>\n",
       "      <td>324.7</td>\n",
       "      <td>0.1482</td>\n",
       "      <td>0.4365</td>\n",
       "      <td>1.2520</td>\n",
       "      <td>0.17500</td>\n",
       "      <td>0.4228</td>\n",
       "      <td>0.11750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>329</th>\n",
       "      <td>895633</td>\n",
       "      <td>M</td>\n",
       "      <td>16.260</td>\n",
       "      <td>21.88</td>\n",
       "      <td>107.50</td>\n",
       "      <td>826.8</td>\n",
       "      <td>0.11650</td>\n",
       "      <td>0.12830</td>\n",
       "      <td>0.17990</td>\n",
       "      <td>0.07981</td>\n",
       "      <td>...</td>\n",
       "      <td>17.73</td>\n",
       "      <td>25.21</td>\n",
       "      <td>113.70</td>\n",
       "      <td>975.2</td>\n",
       "      <td>0.1426</td>\n",
       "      <td>0.2116</td>\n",
       "      <td>0.3344</td>\n",
       "      <td>0.10470</td>\n",
       "      <td>0.2736</td>\n",
       "      <td>0.07953</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "91    861799         M       15.370         22.76          100.20      728.2   \n",
       "395   903811         B       14.060         17.18           89.75      609.1   \n",
       "224  8813129         B       13.270         17.02           84.55      546.4   \n",
       "406   905189         B       16.140         14.86          104.30      800.0   \n",
       "388   903011         B       11.270         15.50           73.38      392.0   \n",
       "137   868682         B       11.430         15.39           73.06      399.8   \n",
       "68    859471         B        9.029         17.33           58.79      250.5   \n",
       "329   895633         M       16.260         21.88          107.50      826.8   \n",
       "\n",
       "     smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "91           0.09200           0.10360         0.11220              0.07483   \n",
       "395          0.08045           0.05361         0.02681              0.03251   \n",
       "224          0.08445           0.04994         0.03554              0.02456   \n",
       "406          0.09495           0.08501         0.05500              0.04528   \n",
       "388          0.08365           0.11140         0.10070              0.02757   \n",
       "137          0.09639           0.06889         0.03503              0.02875   \n",
       "68           0.10660           0.14130         0.31300              0.04375   \n",
       "329          0.11650           0.12830         0.17990              0.07981   \n",
       "\n",
       "     ...  radius_worst  texture_worst  perimeter_worst  area_worst  \\\n",
       "91   ...         16.43          25.84           107.50       830.9   \n",
       "395  ...         14.92          25.34            96.42       684.5   \n",
       "224  ...         15.14          23.60            98.84       708.8   \n",
       "406  ...         17.71          19.58           115.90       947.9   \n",
       "388  ...         12.04          18.93            79.73       450.0   \n",
       "137  ...         12.32          22.02            79.93       462.0   \n",
       "68   ...         10.31          22.65            65.50       324.7   \n",
       "329  ...         17.73          25.21           113.70       975.2   \n",
       "\n",
       "     smoothness_worst  compactness_worst  concavity_worst  \\\n",
       "91             0.1257             0.1997           0.2846   \n",
       "395            0.1066             0.1231           0.0846   \n",
       "224            0.1276             0.1311           0.1786   \n",
       "406            0.1206             0.1722           0.2310   \n",
       "388            0.1102             0.2809           0.3021   \n",
       "137            0.1190             0.1648           0.1399   \n",
       "68             0.1482             0.4365           1.2520   \n",
       "329            0.1426             0.2116           0.3344   \n",
       "\n",
       "     concave points_worst  symmetry_worst  fractal_dimension_worst  \n",
       "91                0.14760          0.2556                  0.06828  \n",
       "395               0.07911          0.2523                  0.06609  \n",
       "224               0.09678          0.2506                  0.07623  \n",
       "406               0.11290          0.2778                  0.07012  \n",
       "388               0.08272          0.2157                  0.10430  \n",
       "137               0.08476          0.2676                  0.06765  \n",
       "68                0.17500          0.4228                  0.11750  \n",
       "329               0.10470          0.2736                  0.07953  \n",
       "\n",
       "[8 rows x 32 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "s3 = boto3.client(\"s3\")\n",
    "\n",
    "filename = \"wdbc.csv\"\n",
    "s3.download_file(\n",
    "    f\"sagemaker-example-files-prod-{boto3.session.Session().region_name}\",\n",
    "    \"datasets/tabular/breast_cancer/wdbc.csv\",\n",
    "    filename,\n",
    ")\n",
    "data = pd.read_csv(filename, header=None)\n",
    "\n",
    "# specify columns extracted from wbdc.names\n",
    "data.columns = [\n",
    "    \"id\",\n",
    "    \"diagnosis\",\n",
    "    \"radius_mean\",\n",
    "    \"texture_mean\",\n",
    "    \"perimeter_mean\",\n",
    "    \"area_mean\",\n",
    "    \"smoothness_mean\",\n",
    "    \"compactness_mean\",\n",
    "    \"concavity_mean\",\n",
    "    \"concave points_mean\",\n",
    "    \"symmetry_mean\",\n",
    "    \"fractal_dimension_mean\",\n",
    "    \"radius_se\",\n",
    "    \"texture_se\",\n",
    "    \"perimeter_se\",\n",
    "    \"area_se\",\n",
    "    \"smoothness_se\",\n",
    "    \"compactness_se\",\n",
    "    \"concavity_se\",\n",
    "    \"concave points_se\",\n",
    "    \"symmetry_se\",\n",
    "    \"fractal_dimension_se\",\n",
    "    \"radius_worst\",\n",
    "    \"texture_worst\",\n",
    "    \"perimeter_worst\",\n",
    "    \"area_worst\",\n",
    "    \"smoothness_worst\",\n",
    "    \"compactness_worst\",\n",
    "    \"concavity_worst\",\n",
    "    \"concave points_worst\",\n",
    "    \"symmetry_worst\",\n",
    "    \"fractal_dimension_worst\",\n",
    "]\n",
    "\n",
    "# save the data\n",
    "data.to_csv(\"data.csv\", sep=\",\", index=False)\n",
    "\n",
    "data.sample(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Key observations:\n",
    "* The data has 569 observations and 32 columns.\n",
    "* The first field is the 'id' attribute that we will want to drop before batch inference and add to the final inference output next to the probability of malignancy.\n",
    "* Second field, 'diagnosis', is an indicator of the actual diagnosis ('M' = Malignant; 'B' = Benign).\n",
    "* There are 30 other numeric features that we will use for training and inferencing."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's replace the M/B diagnosis with a 1/0 boolean value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>843786</td>\n",
       "      <td>1</td>\n",
       "      <td>12.45</td>\n",
       "      <td>15.70</td>\n",
       "      <td>82.57</td>\n",
       "      <td>477.1</td>\n",
       "      <td>0.12780</td>\n",
       "      <td>0.17000</td>\n",
       "      <td>0.15780</td>\n",
       "      <td>0.08089</td>\n",
       "      <td>...</td>\n",
       "      <td>15.47</td>\n",
       "      <td>23.75</td>\n",
       "      <td>103.40</td>\n",
       "      <td>741.6</td>\n",
       "      <td>0.1791</td>\n",
       "      <td>0.52490</td>\n",
       "      <td>0.53550</td>\n",
       "      <td>0.17410</td>\n",
       "      <td>0.3985</td>\n",
       "      <td>0.12440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>311</th>\n",
       "      <td>89382601</td>\n",
       "      <td>0</td>\n",
       "      <td>14.61</td>\n",
       "      <td>15.69</td>\n",
       "      <td>92.68</td>\n",
       "      <td>664.9</td>\n",
       "      <td>0.07618</td>\n",
       "      <td>0.03515</td>\n",
       "      <td>0.01447</td>\n",
       "      <td>0.01877</td>\n",
       "      <td>...</td>\n",
       "      <td>16.46</td>\n",
       "      <td>21.75</td>\n",
       "      <td>103.70</td>\n",
       "      <td>840.8</td>\n",
       "      <td>0.1011</td>\n",
       "      <td>0.07087</td>\n",
       "      <td>0.04746</td>\n",
       "      <td>0.05813</td>\n",
       "      <td>0.2530</td>\n",
       "      <td>0.05695</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>87281702</td>\n",
       "      <td>1</td>\n",
       "      <td>16.46</td>\n",
       "      <td>20.11</td>\n",
       "      <td>109.30</td>\n",
       "      <td>832.9</td>\n",
       "      <td>0.09831</td>\n",
       "      <td>0.15560</td>\n",
       "      <td>0.17930</td>\n",
       "      <td>0.08866</td>\n",
       "      <td>...</td>\n",
       "      <td>17.79</td>\n",
       "      <td>28.45</td>\n",
       "      <td>123.50</td>\n",
       "      <td>981.2</td>\n",
       "      <td>0.1415</td>\n",
       "      <td>0.46670</td>\n",
       "      <td>0.58620</td>\n",
       "      <td>0.20350</td>\n",
       "      <td>0.3054</td>\n",
       "      <td>0.09519</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>866674</td>\n",
       "      <td>1</td>\n",
       "      <td>19.79</td>\n",
       "      <td>25.12</td>\n",
       "      <td>130.40</td>\n",
       "      <td>1192.0</td>\n",
       "      <td>0.10150</td>\n",
       "      <td>0.15890</td>\n",
       "      <td>0.25450</td>\n",
       "      <td>0.11490</td>\n",
       "      <td>...</td>\n",
       "      <td>22.63</td>\n",
       "      <td>33.58</td>\n",
       "      <td>148.70</td>\n",
       "      <td>1589.0</td>\n",
       "      <td>0.1275</td>\n",
       "      <td>0.38610</td>\n",
       "      <td>0.56730</td>\n",
       "      <td>0.17320</td>\n",
       "      <td>0.3305</td>\n",
       "      <td>0.08465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>515</th>\n",
       "      <td>916221</td>\n",
       "      <td>0</td>\n",
       "      <td>11.34</td>\n",
       "      <td>18.61</td>\n",
       "      <td>72.76</td>\n",
       "      <td>391.2</td>\n",
       "      <td>0.10490</td>\n",
       "      <td>0.08499</td>\n",
       "      <td>0.04302</td>\n",
       "      <td>0.02594</td>\n",
       "      <td>...</td>\n",
       "      <td>12.47</td>\n",
       "      <td>23.03</td>\n",
       "      <td>79.15</td>\n",
       "      <td>478.6</td>\n",
       "      <td>0.1483</td>\n",
       "      <td>0.15740</td>\n",
       "      <td>0.16240</td>\n",
       "      <td>0.08542</td>\n",
       "      <td>0.3060</td>\n",
       "      <td>0.06783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>325</th>\n",
       "      <td>89511502</td>\n",
       "      <td>0</td>\n",
       "      <td>12.67</td>\n",
       "      <td>17.30</td>\n",
       "      <td>81.25</td>\n",
       "      <td>489.9</td>\n",
       "      <td>0.10280</td>\n",
       "      <td>0.07664</td>\n",
       "      <td>0.03193</td>\n",
       "      <td>0.02107</td>\n",
       "      <td>...</td>\n",
       "      <td>13.71</td>\n",
       "      <td>21.10</td>\n",
       "      <td>88.70</td>\n",
       "      <td>574.4</td>\n",
       "      <td>0.1384</td>\n",
       "      <td>0.12120</td>\n",
       "      <td>0.10200</td>\n",
       "      <td>0.05602</td>\n",
       "      <td>0.2688</td>\n",
       "      <td>0.06888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>279</th>\n",
       "      <td>8911834</td>\n",
       "      <td>0</td>\n",
       "      <td>13.85</td>\n",
       "      <td>15.18</td>\n",
       "      <td>88.99</td>\n",
       "      <td>587.4</td>\n",
       "      <td>0.09516</td>\n",
       "      <td>0.07688</td>\n",
       "      <td>0.04479</td>\n",
       "      <td>0.03711</td>\n",
       "      <td>...</td>\n",
       "      <td>14.98</td>\n",
       "      <td>21.74</td>\n",
       "      <td>98.37</td>\n",
       "      <td>670.0</td>\n",
       "      <td>0.1185</td>\n",
       "      <td>0.17240</td>\n",
       "      <td>0.14560</td>\n",
       "      <td>0.09993</td>\n",
       "      <td>0.2955</td>\n",
       "      <td>0.06912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>323</th>\n",
       "      <td>895100</td>\n",
       "      <td>1</td>\n",
       "      <td>20.34</td>\n",
       "      <td>21.51</td>\n",
       "      <td>135.90</td>\n",
       "      <td>1264.0</td>\n",
       "      <td>0.11700</td>\n",
       "      <td>0.18750</td>\n",
       "      <td>0.25650</td>\n",
       "      <td>0.15040</td>\n",
       "      <td>...</td>\n",
       "      <td>25.30</td>\n",
       "      <td>31.86</td>\n",
       "      <td>171.10</td>\n",
       "      <td>1938.0</td>\n",
       "      <td>0.1592</td>\n",
       "      <td>0.44920</td>\n",
       "      <td>0.53440</td>\n",
       "      <td>0.26850</td>\n",
       "      <td>0.5558</td>\n",
       "      <td>0.10240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id  diagnosis  radius_mean  texture_mean  perimeter_mean  \\\n",
       "5      843786          1        12.45         15.70           82.57   \n",
       "311  89382601          0        14.61         15.69           92.68   \n",
       "177  87281702          1        16.46         20.11          109.30   \n",
       "129    866674          1        19.79         25.12          130.40   \n",
       "515    916221          0        11.34         18.61           72.76   \n",
       "325  89511502          0        12.67         17.30           81.25   \n",
       "279   8911834          0        13.85         15.18           88.99   \n",
       "323    895100          1        20.34         21.51          135.90   \n",
       "\n",
       "     area_mean  smoothness_mean  compactness_mean  concavity_mean  \\\n",
       "5        477.1          0.12780           0.17000         0.15780   \n",
       "311      664.9          0.07618           0.03515         0.01447   \n",
       "177      832.9          0.09831           0.15560         0.17930   \n",
       "129     1192.0          0.10150           0.15890         0.25450   \n",
       "515      391.2          0.10490           0.08499         0.04302   \n",
       "325      489.9          0.10280           0.07664         0.03193   \n",
       "279      587.4          0.09516           0.07688         0.04479   \n",
       "323     1264.0          0.11700           0.18750         0.25650   \n",
       "\n",
       "     concave points_mean  ...  radius_worst  texture_worst  perimeter_worst  \\\n",
       "5                0.08089  ...         15.47          23.75           103.40   \n",
       "311              0.01877  ...         16.46          21.75           103.70   \n",
       "177              0.08866  ...         17.79          28.45           123.50   \n",
       "129              0.11490  ...         22.63          33.58           148.70   \n",
       "515              0.02594  ...         12.47          23.03            79.15   \n",
       "325              0.02107  ...         13.71          21.10            88.70   \n",
       "279              0.03711  ...         14.98          21.74            98.37   \n",
       "323              0.15040  ...         25.30          31.86           171.10   \n",
       "\n",
       "     area_worst  smoothness_worst  compactness_worst  concavity_worst  \\\n",
       "5         741.6            0.1791            0.52490          0.53550   \n",
       "311       840.8            0.1011            0.07087          0.04746   \n",
       "177       981.2            0.1415            0.46670          0.58620   \n",
       "129      1589.0            0.1275            0.38610          0.56730   \n",
       "515       478.6            0.1483            0.15740          0.16240   \n",
       "325       574.4            0.1384            0.12120          0.10200   \n",
       "279       670.0            0.1185            0.17240          0.14560   \n",
       "323      1938.0            0.1592            0.44920          0.53440   \n",
       "\n",
       "     concave points_worst  symmetry_worst  fractal_dimension_worst  \n",
       "5                 0.17410          0.3985                  0.12440  \n",
       "311               0.05813          0.2530                  0.05695  \n",
       "177               0.20350          0.3054                  0.09519  \n",
       "129               0.17320          0.3305                  0.08465  \n",
       "515               0.08542          0.3060                  0.06783  \n",
       "325               0.05602          0.2688                  0.06888  \n",
       "279               0.09993          0.2955                  0.06912  \n",
       "323               0.26850          0.5558                  0.10240  \n",
       "\n",
       "[8 rows x 32 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"diagnosis\"] = data[\"diagnosis\"].apply(lambda x: ((x == \"M\")) + 0)\n",
    "data.sample(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the data as follows: 80% for training, 10% for validation and let's set 10% aside for our batch inference job. In addition, let's drop the 'id' field on the training set and validation set as 'id' is not a training feature. For our batch set however, we keep the 'id' feature. We'll want to filter it out prior to running our inferences so that the input data features match the ones of training set and then ultimately, we'll want to join it with inference result. We are however dropping the diagnosis attribute for the batch set since this is what we'll try to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data split in three sets, training, validation and batch inference\n",
    "rand_split = np.random.rand(len(data))\n",
    "train_list = rand_split < 0.8\n",
    "val_list = (rand_split >= 0.8) & (rand_split < 0.9)\n",
    "batch_list = rand_split >= 0.9\n",
    "\n",
    "data_train = data[train_list].drop([\"id\"], axis=1)\n",
    "data_val = data[val_list].drop([\"id\"], axis=1)\n",
    "data_batch = data[batch_list].drop([\"diagnosis\"], axis=1)\n",
    "data_batch_noID = data_batch.drop([\"id\"], axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "ff9d10f9-b611-423b-80da-6dcdafd1c8b9"
    }
   },
   "source": [
    "Let's upload those data sets in S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "nbpresent": {
     "id": "cd8e3431-79d9-40b6-91d1-d67cd61894e7"
    }
   },
   "outputs": [],
   "source": [
    "s3_resource = boto3.Session().resource(\"s3\")\n",
    "\n",
    "train_file = \"train_data.csv\"\n",
    "data_train.to_csv(train_file, index=False, header=False)\n",
    "with open(train_file, \"rb\") as data:\n",
    "    s3_resource.Bucket(bucket).upload_fileobj(data, os.path.join(prefix, \"train\", train_file))\n",
    "\n",
    "validation_file = \"validation_data.csv\"\n",
    "data_val.to_csv(validation_file, index=False, header=False)\n",
    "with open(validation_file, \"rb\") as data:\n",
    "    s3_resource.Bucket(bucket).upload_fileobj(\n",
    "        data, os.path.join(prefix, \"validation\", validation_file)\n",
    "    )\n",
    "\n",
    "batch_file = \"batch_data.csv\"\n",
    "data_batch.to_csv(batch_file, index=False, header=False)\n",
    "with open(batch_file, \"rb\") as data:\n",
    "    s3_resource.Bucket(bucket).upload_fileobj(data, os.path.join(prefix, \"batch\", batch_file))\n",
    "\n",
    "batch_file_noID = \"batch_data_noID.csv\"\n",
    "data_batch_noID.to_csv(batch_file_noID, index=False, header=False)\n",
    "with open(batch_file_noID, \"rb\") as data:\n",
    "    s3_resource.Bucket(bucket).upload_fileobj(data, os.path.join(prefix, \"batch\", batch_file_noID))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "71cbcebd-a2a5-419e-8e50-b2bc0909f564"
    }
   },
   "source": [
    "---\n",
    "\n",
    "## Training job and model creation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bd113b8e-adc1-4091-a26f-a426149fe604"
    }
   },
   "source": [
    "The below cell uses the [Boto3 SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_training_job) to kick off the training job using both our training set and validation set. Not that the objective is set to 'binary:logistic' which trains a model to output a probability between 0 and 1 (here the probability of a tumor being malignant)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "f3b125ad-a2d5-464c-8cfa-bd203034eee4"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "from time import gmtime, strftime\n",
    "from sagemaker.image_uris import retrieve\n",
    "\n",
    "job_name = \"xgb-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "print(\"Training job\", job_name)\n",
    "\n",
    "image = retrieve(framework=\"xgboost\", region=boto3.Session().region_name, version=\"latest\")\n",
    "\n",
    "output_location = \"s3://{}/{}/output/{}\".format(bucket, prefix, job_name)\n",
    "print(\"Training artifacts will be uploaded to: {}\".format(output_location))\n",
    "\n",
    "create_training_params = {\n",
    "    \"AlgorithmSpecification\": {\"TrainingImage\": image, \"TrainingInputMode\": \"File\"},\n",
    "    \"RoleArn\": role,\n",
    "    \"OutputDataConfig\": {\"S3OutputPath\": output_location},\n",
    "    \"ResourceConfig\": {\"InstanceCount\": 1, \"InstanceType\": \"ml.m5.4xlarge\", \"VolumeSizeInGB\": 50},\n",
    "    \"TrainingJobName\": job_name,\n",
    "    \"HyperParameters\": {\n",
    "        \"objective\": \"binary:logistic\",\n",
    "        \"max_depth\": \"5\",\n",
    "        \"eta\": \"0.2\",\n",
    "        \"gamma\": \"4\",\n",
    "        \"min_child_weight\": \"6\",\n",
    "        \"subsample\": \"0.8\",\n",
    "        \"silent\": \"0\",\n",
    "        \"num_round\": \"100\",\n",
    "    },\n",
    "    \"StoppingCondition\": {\"MaxRuntimeInSeconds\": 60 * 60},\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"train\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": \"s3://{}/{}/train\".format(bucket, prefix),\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\",\n",
    "                }\n",
    "            },\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"RecordWrapperType\": \"None\",\n",
    "            \"ContentType\": \"text/csv\",\n",
    "        },\n",
    "        {\n",
    "            \"ChannelName\": \"validation\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": \"s3://{}/{}/validation\".format(bucket, prefix),\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\",\n",
    "                }\n",
    "            },\n",
    "            \"CompressionType\": \"None\",\n",
    "            \"RecordWrapperType\": \"None\",\n",
    "            \"ContentType\": \"text/csv\",\n",
    "        },\n",
    "    ],\n",
    "}\n",
    "\n",
    "sagemaker = boto3.client(\"sagemaker\")\n",
    "sagemaker.create_training_job(**create_training_params)\n",
    "status = sagemaker.describe_training_job(TrainingJobName=job_name)[\"TrainingJobStatus\"]\n",
    "print(status)\n",
    "\n",
    "try:\n",
    "    sagemaker.get_waiter(\"training_job_completed_or_stopped\").wait(TrainingJobName=job_name)\n",
    "finally:\n",
    "    status = sagemaker.describe_training_job(TrainingJobName=job_name)[\"TrainingJobStatus\"]\n",
    "    print(\"Training job ended with status: \" + status)\n",
    "    if status == \"Failed\":\n",
    "        message = sagemaker.describe_training_job(TrainingJobName=job_name)[\"FailureReason\"]\n",
    "        print(\"Training failed with the following error: {}\".format(message))\n",
    "        raise Exception(\"Training job failed\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's create a model based on our training job. \n",
    "\n",
    "The below cell creates a model in SageMaker based on the training job we just executed. The model can later be deployed using the SageMaker hosting services or in our case used in a Batch Transform job.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model_name = job_name\n",
    "print(model_name)\n",
    "\n",
    "info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "model_data = info[\"ModelArtifacts\"][\"S3ModelArtifacts\"]\n",
    "\n",
    "primary_container = {\"Image\": image, \"ModelDataUrl\": model_data}\n",
    "\n",
    "create_model_response = sagemaker.create_model(\n",
    "    ModelName=model_name, ExecutionRoleArn=role, PrimaryContainer=primary_container\n",
    ")\n",
    "\n",
    "print(create_model_response[\"ModelArn\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "397fb60a-c48b-453f-88ea-4d832b70c919"
    }
   },
   "source": [
    "---\n",
    "\n",
    "## Batch Transform\n",
    "\n",
    "In SageMaker Batch Transform, we introduced a new attribute called __DataProcessing__.In the below cell, we use the [Boto3 SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_transform_job) to kick-off several Batch Transform jobs using different configurations of DataProcessing. Please refer to [Associate Prediction Results with Input Records](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform-data-processing.html) to learn more about how to utilize the __DataProcessing__ attribute.\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Without data processing\n",
    "Let's first set the data processing fields to null and inspect the inference results. We'll use it as a baseline to compare to the results with data processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "from time import gmtime, strftime\n",
    "\n",
    "batch_job_name = \"Batch-Transform-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "input_location = \"s3://{}/{}/batch/{}\".format(\n",
    "    bucket, prefix, batch_file_noID\n",
    ")  # use input data without ID column\n",
    "output_location = \"s3://{}/{}/output/{}\".format(bucket, prefix, batch_job_name)\n",
    "\n",
    "request = {\n",
    "    \"TransformJobName\": batch_job_name,\n",
    "    \"ModelName\": job_name,\n",
    "    \"TransformOutput\": {\n",
    "        \"S3OutputPath\": output_location,\n",
    "        \"Accept\": \"text/csv\",\n",
    "        \"AssembleWith\": \"Line\",\n",
    "    },\n",
    "    \"TransformInput\": {\n",
    "        \"DataSource\": {\"S3DataSource\": {\"S3DataType\": \"S3Prefix\", \"S3Uri\": input_location}},\n",
    "        \"ContentType\": \"text/csv\",\n",
    "        \"SplitType\": \"Line\",\n",
    "        \"CompressionType\": \"None\",\n",
    "    },\n",
    "    \"TransformResources\": {\"InstanceType\": \"ml.m4.xlarge\", \"InstanceCount\": 1},\n",
    "}\n",
    "\n",
    "sagemaker.create_transform_job(**request)\n",
    "print(\"Created Transform job with name: \", batch_job_name)\n",
    "\n",
    "# Wait until the job finishes\n",
    "try:\n",
    "    sagemaker.get_waiter(\"transform_job_completed_or_stopped\").wait(TransformJobName=batch_job_name)\n",
    "finally:\n",
    "    response = sagemaker.describe_transform_job(TransformJobName=batch_job_name)\n",
    "    status = response[\"TransformJobStatus\"]\n",
    "    print(\"Transform job ended with status: \" + status)\n",
    "    if status == \"Failed\":\n",
    "        message = response[\"FailureReason\"]\n",
    "        print(\"Transform failed with the following error: {}\".format(message))\n",
    "        raise Exception(\"Transform job failed\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the output of the Batch Transform job in S3. It should show the list probabilities of tumors being malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "def get_csv_output_from_s3(s3uri, batch_file):\n",
    "    file_name = \"{}.out\".format(batch_file)\n",
    "    match = re.match(\"s3://([^/]+)/(.*)\", \"{}/{}\".format(s3uri, file_name))\n",
    "    output_bucket, output_prefix = match.group(1), match.group(2)\n",
    "    s3.download_file(output_bucket, output_prefix, file_name)\n",
    "    return pd.read_csv(file_name, sep=\",\", header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = get_csv_output_from_s3(output_location, batch_file_noID)\n",
    "output_df.head(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Join the input and the prediction results \n",
    "Now, let's use the new feature to associate the prediction results with their corresponding input records. We can also use the __InputFilter__ to exclude the ID column easily and there's no need to have a separate file in S3.\n",
    "\n",
    "* Set __InputFilter__ to \"$[1:]\": indicates that we are excluding column 0 (the 'ID') before processing the inferences and keeping everything from column 1 to the last column (all the features or predictors)  \n",
    "  \n",
    "  \n",
    "* Set __JoinSource__ to \"Input\": indicates our desire to join the input data with the inference results  \n",
    "\n",
    "\n",
    "* Leave __OutputFilter__ to default (\"$\"), indicating that the joined input and inference results be will saved as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "batch_job_name = \"Batch-Transform-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "input_location = \"s3://{}/{}/batch/{}\".format(\n",
    "    bucket, prefix, batch_file\n",
    ")  # use input data with ID column cause InputFilter will filter it out\n",
    "output_location = \"s3://{}/{}/output/{}\".format(bucket, prefix, batch_job_name)\n",
    "\n",
    "request[\"TransformJobName\"] = batch_job_name\n",
    "request[\"TransformInput\"][\"DataSource\"][\"S3DataSource\"][\"S3Uri\"] = input_location\n",
    "request[\"TransformOutput\"][\"S3OutputPath\"] = output_location\n",
    "\n",
    "request[\"DataProcessing\"] = {\n",
    "    \"InputFilter\": \"$[1:]\",  # exclude the ID column (index 0)\n",
    "    \"JoinSource\": \"Input\",  # join the input with the inference results\n",
    "}\n",
    "\n",
    "sagemaker.create_transform_job(**request)\n",
    "print(\"Created Transform job with name: \", batch_job_name)\n",
    "\n",
    "# Wait until the job finishes\n",
    "try:\n",
    "    sagemaker.get_waiter(\"transform_job_completed_or_stopped\").wait(TransformJobName=batch_job_name)\n",
    "finally:\n",
    "    response = sagemaker.describe_transform_job(TransformJobName=batch_job_name)\n",
    "    status = response[\"TransformJobStatus\"]\n",
    "    print(\"Transform job ended with status: \" + status)\n",
    "    if status == \"Failed\":\n",
    "        message = response[\"FailureReason\"]\n",
    "        print(\"Transform failed with the following error: {}\".format(message))\n",
    "        raise Exception(\"Transform job failed\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the output of the Batch Transform job in S3. It should show the list of tumors identified by their original feature columns and their corresponding probabilities of being malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = get_csv_output_from_s3(output_location, batch_file)\n",
    "output_df.head(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Update the output filter to keep only ID and prediction results\n",
    "Let's try to change the output filter  to \"$[0,-1]\", indicating that when presenting the output, we only want to keep column 0 (the 'ID') and the last column (the inference result i.e. the probability of a given tumor to be malignant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "batch_job_name = \"Batch-Transform-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "output_location = \"s3://{}/{}/output/{}\".format(bucket, prefix, batch_job_name)\n",
    "\n",
    "request[\"TransformJobName\"] = batch_job_name\n",
    "request[\"TransformOutput\"][\"S3OutputPath\"] = output_location\n",
    "request[\"DataProcessing\"][\n",
    "    \"OutputFilter\"\n",
    "] = \"$[0, -1]\"  # keep the first and last column of the joined output\n",
    "\n",
    "sagemaker.create_transform_job(**request)\n",
    "print(\"Created Transform job with name: \", batch_job_name)\n",
    "\n",
    "# Wait until the job finishes\n",
    "try:\n",
    "    sagemaker.get_waiter(\"transform_job_completed_or_stopped\").wait(TransformJobName=batch_job_name)\n",
    "finally:\n",
    "    response = sagemaker.describe_transform_job(TransformJobName=batch_job_name)\n",
    "    status = response[\"TransformJobStatus\"]\n",
    "    print(\"Transform job ended with status: \" + status)\n",
    "    if status == \"Failed\":\n",
    "        message = response[\"FailureReason\"]\n",
    "        print(\"Transform failed with the following error: {}\".format(message))\n",
    "        raise Exception(\"Transform job failed\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's inspect the output of the Batch Transform job in S3 again. It should show 2 columns: the ID and their corresponding probabilities of being malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = get_csv_output_from_s3(output_location, batch_file)\n",
    "output_df.head(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, we can use DataProcessing to \n",
    "1. Filter / select useful features from the input dataset. e.g. exclude ID columns.\n",
    "2. Associate the prediction results with their corresponding input records.\n",
    "3. Filter the original or joined results before saving to S3. e.g. keep ID and probability columns only."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with lowel level SDK.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the License). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the license file accompanying this file. This file is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}