 archived/Batch Transform - breast cancer prediction with high level SDK.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "42b5e80b-ad1d-4335-a1f7-10a91127e3dc"
    }
   },
   "source": [
    "# Amazon SageMaker Batch Transform: Associate prediction results with their corresponding input records\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "42b5e80b-ad1d-4335-a1f7-10a91127e3dc"
    }
   },
   "source": [
    "_**Use SageMaker's XGBoost to train a binary classification model and for a list of tumors in batch file, predict if each is malignant**_\n",
    "\n",
    "_**It also shows how to use the input output joining / filter feature in Batch transform in details**_\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "\n",
    "## Background\n",
    "This purpose of this notebook is to train a model using SageMaker's XGBoost and UCI's breast cancer diagnostic data set to illustrate at how to run batch inferences and how to use the Batch Transform I/O join feature. UCI's breast cancer diagnostic data set is available at https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29. The data set is also available on Kaggle at https://www.kaggle.com/uciml/breast-cancer-wisconsin-data. The purpose here is to use this data set to build a predictve model of whether a breast mass image indicates benign or malignant tumor. \n",
    "\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Setup\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "* The SageMaker role arn used to give training and batch transform access to your data. The snippet below will use the same role used by your SageMaker notebook instance. Otherwise, specify the full ARN of a role with the SageMakerFullAccess policy attached.\n",
    "* The S3 bucket that you want to use for training and storing model objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "isConfigCell": true,
    "nbpresent": {
     "id": "6427e831-8f89-45c0-b150-0b134397d79a"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "sess = sagemaker.Session()\n",
    "region = sess.boto_region_name\n",
    "\n",
    "bucket = sess.default_bucket()\n",
    "prefix = \"DEMO-breast-cancer-prediction-xgboost-highlevel\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "142777ae-c072-448e-b941-72bc75735d01"
    }
   },
   "source": [
    "---\n",
    "## Data sources\n",
    "\n",
    "> Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.\n",
    "\n",
    "> Breast Cancer Wisconsin (Diagnostic) Data Set [https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)].\n",
    "\n",
    "> _Also see:_ Breast Cancer Wisconsin (Diagnostic) Data Set [https://www.kaggle.com/uciml/breast-cancer-wisconsin-data].\n",
    "\n",
    "## Data preparation\n",
    "\n",
    "\n",
    "Let's download the data and save it in the local folder with the name data.csv and take a look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "nbpresent": {
     "id": "f8976dad-6897-4c7e-8c95-ae2f53070ef5"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>diagnosis</th>\n",
       "      <th>radius_mean</th>\n",
       "      <th>texture_mean</th>\n",
       "      <th>perimeter_mean</th>\n",
       "      <th>area_mean</th>\n",
       "      <th>smoothness_mean</th>\n",
       "      <th>compactness_mean</th>\n",
       "      <th>concavity_mean</th>\n",
       "      <th>concave points_mean</th>\n",
       "      <th>...</th>\n",
       "      <th>radius_worst</th>\n",
       "      <th>texture_worst</th>\n",
       "      <th>perimeter_worst</th>\n",
       "      <th>area_worst</th>\n",
       "      <th>smoothness_worst</th>\n",
       "      <th>compactness_worst</th>\n",
       "      <th>concavity_worst</th>\n",
       "      <th>concave points_worst</th>\n",
       "      <th>symmetry_worst</th>\n",
       "      <th>fractal_dimension_worst</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>888264</td>\n",
       "      <td>M</td>\n",
       "      <td>17.35</td>\n",
       "      <td>23.06</td>\n",
       "      <td>111.00</td>\n",
       "      <td>933.1</td>\n",
       "      <td>0.08662</td>\n",
       "      <td>0.06290</td>\n",
       "      <td>0.02891</td>\n",
       "      <td>0.02837</td>\n",
       "      <td>...</td>\n",
       "      <td>19.85</td>\n",
       "      <td>31.47</td>\n",
       "      <td>128.20</td>\n",
       "      <td>1218.0</td>\n",
       "      <td>0.12400</td>\n",
       "      <td>0.1486</td>\n",
       "      <td>0.12110</td>\n",
       "      <td>0.08235</td>\n",
       "      <td>0.2452</td>\n",
       "      <td>0.06515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>8610404</td>\n",
       "      <td>M</td>\n",
       "      <td>16.07</td>\n",
       "      <td>19.65</td>\n",
       "      <td>104.10</td>\n",
       "      <td>817.7</td>\n",
       "      <td>0.09168</td>\n",
       "      <td>0.08424</td>\n",
       "      <td>0.09769</td>\n",
       "      <td>0.06638</td>\n",
       "      <td>...</td>\n",
       "      <td>19.77</td>\n",
       "      <td>24.56</td>\n",
       "      <td>128.80</td>\n",
       "      <td>1223.0</td>\n",
       "      <td>0.15000</td>\n",
       "      <td>0.2045</td>\n",
       "      <td>0.28290</td>\n",
       "      <td>0.15200</td>\n",
       "      <td>0.2650</td>\n",
       "      <td>0.06387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>325</th>\n",
       "      <td>89511502</td>\n",
       "      <td>B</td>\n",
       "      <td>12.67</td>\n",
       "      <td>17.30</td>\n",
       "      <td>81.25</td>\n",
       "      <td>489.9</td>\n",
       "      <td>0.10280</td>\n",
       "      <td>0.07664</td>\n",
       "      <td>0.03193</td>\n",
       "      <td>0.02107</td>\n",
       "      <td>...</td>\n",
       "      <td>13.71</td>\n",
       "      <td>21.10</td>\n",
       "      <td>88.70</td>\n",
       "      <td>574.4</td>\n",
       "      <td>0.13840</td>\n",
       "      <td>0.1212</td>\n",
       "      <td>0.10200</td>\n",
       "      <td>0.05602</td>\n",
       "      <td>0.2688</td>\n",
       "      <td>0.06888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>869476</td>\n",
       "      <td>B</td>\n",
       "      <td>11.90</td>\n",
       "      <td>14.65</td>\n",
       "      <td>78.11</td>\n",
       "      <td>432.8</td>\n",
       "      <td>0.11520</td>\n",
       "      <td>0.12960</td>\n",
       "      <td>0.03710</td>\n",
       "      <td>0.03003</td>\n",
       "      <td>...</td>\n",
       "      <td>13.15</td>\n",
       "      <td>16.51</td>\n",
       "      <td>86.26</td>\n",
       "      <td>509.6</td>\n",
       "      <td>0.14240</td>\n",
       "      <td>0.2517</td>\n",
       "      <td>0.09420</td>\n",
       "      <td>0.06042</td>\n",
       "      <td>0.2727</td>\n",
       "      <td>0.10360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>438</th>\n",
       "      <td>909231</td>\n",
       "      <td>B</td>\n",
       "      <td>13.85</td>\n",
       "      <td>19.60</td>\n",
       "      <td>88.68</td>\n",
       "      <td>592.6</td>\n",
       "      <td>0.08684</td>\n",
       "      <td>0.06330</td>\n",
       "      <td>0.01342</td>\n",
       "      <td>0.02293</td>\n",
       "      <td>...</td>\n",
       "      <td>15.63</td>\n",
       "      <td>28.01</td>\n",
       "      <td>100.90</td>\n",
       "      <td>749.1</td>\n",
       "      <td>0.11180</td>\n",
       "      <td>0.1141</td>\n",
       "      <td>0.04753</td>\n",
       "      <td>0.05890</td>\n",
       "      <td>0.2513</td>\n",
       "      <td>0.06911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>390</th>\n",
       "      <td>90317302</td>\n",
       "      <td>B</td>\n",
       "      <td>10.26</td>\n",
       "      <td>12.22</td>\n",
       "      <td>65.75</td>\n",
       "      <td>321.6</td>\n",
       "      <td>0.09996</td>\n",
       "      <td>0.07542</td>\n",
       "      <td>0.01923</td>\n",
       "      <td>0.01968</td>\n",
       "      <td>...</td>\n",
       "      <td>11.38</td>\n",
       "      <td>15.65</td>\n",
       "      <td>73.23</td>\n",
       "      <td>394.5</td>\n",
       "      <td>0.13430</td>\n",
       "      <td>0.1650</td>\n",
       "      <td>0.08615</td>\n",
       "      <td>0.06696</td>\n",
       "      <td>0.2937</td>\n",
       "      <td>0.07722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>426</th>\n",
       "      <td>907409</td>\n",
       "      <td>B</td>\n",
       "      <td>10.48</td>\n",
       "      <td>14.98</td>\n",
       "      <td>67.49</td>\n",
       "      <td>333.6</td>\n",
       "      <td>0.09816</td>\n",
       "      <td>0.10130</td>\n",
       "      <td>0.06335</td>\n",
       "      <td>0.02218</td>\n",
       "      <td>...</td>\n",
       "      <td>12.13</td>\n",
       "      <td>21.57</td>\n",
       "      <td>81.41</td>\n",
       "      <td>440.4</td>\n",
       "      <td>0.13270</td>\n",
       "      <td>0.2996</td>\n",
       "      <td>0.29390</td>\n",
       "      <td>0.09310</td>\n",
       "      <td>0.3020</td>\n",
       "      <td>0.09646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>290</th>\n",
       "      <td>89143602</td>\n",
       "      <td>B</td>\n",
       "      <td>14.41</td>\n",
       "      <td>19.73</td>\n",
       "      <td>96.03</td>\n",
       "      <td>651.0</td>\n",
       "      <td>0.08757</td>\n",
       "      <td>0.16760</td>\n",
       "      <td>0.13620</td>\n",
       "      <td>0.06602</td>\n",
       "      <td>...</td>\n",
       "      <td>15.77</td>\n",
       "      <td>22.13</td>\n",
       "      <td>101.70</td>\n",
       "      <td>767.3</td>\n",
       "      <td>0.09983</td>\n",
       "      <td>0.2472</td>\n",
       "      <td>0.22200</td>\n",
       "      <td>0.10210</td>\n",
       "      <td>0.2272</td>\n",
       "      <td>0.08799</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id diagnosis  radius_mean  texture_mean  perimeter_mean  area_mean  \\\n",
       "261    888264         M        17.35         23.06          111.00      933.1   \n",
       "75    8610404         M        16.07         19.65          104.10      817.7   \n",
       "325  89511502         B        12.67         17.30           81.25      489.9   \n",
       "145    869476         B        11.90         14.65           78.11      432.8   \n",
       "438    909231         B        13.85         19.60           88.68      592.6   \n",
       "390  90317302         B        10.26         12.22           65.75      321.6   \n",
       "426    907409         B        10.48         14.98           67.49      333.6   \n",
       "290  89143602         B        14.41         19.73           96.03      651.0   \n",
       "\n",
       "     smoothness_mean  compactness_mean  concavity_mean  concave points_mean  \\\n",
       "261          0.08662           0.06290         0.02891              0.02837   \n",
       "75           0.09168           0.08424         0.09769              0.06638   \n",
       "325          0.10280           0.07664         0.03193              0.02107   \n",
       "145          0.11520           0.12960         0.03710              0.03003   \n",
       "438          0.08684           0.06330         0.01342              0.02293   \n",
       "390          0.09996           0.07542         0.01923              0.01968   \n",
       "426          0.09816           0.10130         0.06335              0.02218   \n",
       "290          0.08757           0.16760         0.13620              0.06602   \n",
       "\n",
       "     ...  radius_worst  texture_worst  perimeter_worst  area_worst  \\\n",
       "261  ...         19.85          31.47           128.20      1218.0   \n",
       "75   ...         19.77          24.56           128.80      1223.0   \n",
       "325  ...         13.71          21.10            88.70       574.4   \n",
       "145  ...         13.15          16.51            86.26       509.6   \n",
       "438  ...         15.63          28.01           100.90       749.1   \n",
       "390  ...         11.38          15.65            73.23       394.5   \n",
       "426  ...         12.13          21.57            81.41       440.4   \n",
       "290  ...         15.77          22.13           101.70       767.3   \n",
       "\n",
       "     smoothness_worst  compactness_worst  concavity_worst  \\\n",
       "261           0.12400             0.1486          0.12110   \n",
       "75            0.15000             0.2045          0.28290   \n",
       "325           0.13840             0.1212          0.10200   \n",
       "145           0.14240             0.2517          0.09420   \n",
       "438           0.11180             0.1141          0.04753   \n",
       "390           0.13430             0.1650          0.08615   \n",
       "426           0.13270             0.2996          0.29390   \n",
       "290           0.09983             0.2472          0.22200   \n",
       "\n",
       "     concave points_worst  symmetry_worst  fractal_dimension_worst  \n",
       "261               0.08235          0.2452                  0.06515  \n",
       "75                0.15200          0.2650                  0.06387  \n",
       "325               0.05602          0.2688                  0.06888  \n",
       "145               0.06042          0.2727                  0.10360  \n",
       "438               0.05890          0.2513                  0.06911  \n",
       "390               0.06696          0.2937                  0.07722  \n",
       "426               0.09310          0.3020                  0.09646  \n",
       "290               0.10210          0.2272                  0.08799  \n",
       "\n",
       "[8 rows x 32 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "s3 = boto3.client(\"s3\")\n",
    "\n",
    "filename = \"wdbc.csv\"\n",
    "s3.download_file(\n",
    "    f\"sagemaker-example-files-prod-{region}\", \"datasets/tabular/breast_cancer/wdbc.csv\", filename\n",
    ")\n",
    "data = pd.read_csv(filename, header=None)\n",
    "\n",
    "# specify columns extracted from wbdc.names\n",
    "data.columns = [\n",
    "    \"id\",\n",
    "    \"diagnosis\",\n",
    "    \"radius_mean\",\n",
    "    \"texture_mean\",\n",
    "    \"perimeter_mean\",\n",
    "    \"area_mean\",\n",
    "    \"smoothness_mean\",\n",
    "    \"compactness_mean\",\n",
    "    \"concavity_mean\",\n",
    "    \"concave points_mean\",\n",
    "    \"symmetry_mean\",\n",
    "    \"fractal_dimension_mean\",\n",
    "    \"radius_se\",\n",
    "    \"texture_se\",\n",
    "    \"perimeter_se\",\n",
    "    \"area_se\",\n",
    "    \"smoothness_se\",\n",
    "    \"compactness_se\",\n",
    "    \"concavity_se\",\n",
    "    \"concave points_se\",\n",
    "    \"symmetry_se\",\n",
    "    \"fractal_dimension_se\",\n",
    "    \"radius_worst\",\n",
    "    \"texture_worst\",\n",
    "    \"perimeter_worst\",\n",
    "    \"area_worst\",\n",
    "    \"smoothness_worst\",\n",
    "    \"compactness_worst\",\n",
    "    \"concavity_worst\",\n",
    "    \"concave points_worst\",\n",
    "    \"symmetry_worst\",\n",
    "    \"fractal_dimension_worst\",\n",
    "]\n",
    "\n",
    "# save the data\n",
    "data.to_csv(\"data.csv\", sep=\",\", index=False)\n",
    "\n",
    "data.sample(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Key observations:\n",
    "* The data has 569 observations and 32 columns.\n",
    "* The first field is the 'id' attribute that we will want to drop before batch inference and add to the final inference output next to the probability of malignancy.\n",
    "* Second field, 'diagnosis', is an indicator of the actual diagnosis ('M' = Malignant; 'B' = Benign).\n",
    "* There are 30 other numeric features that we will use for training and inferencing."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's replace the M/B diagnosis with a 1/0 boolean value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"diagnosis\"] = data[\"diagnosis\"].apply(lambda x: ((x == \"M\")) + 0)\n",
    "data.sample(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's split the data as follows: 80% for training, 10% for validation and let's set 10% aside for our batch inference job. In addition, let's drop the 'id' field on the training set and validation set as 'id' is not a training feature. For our batch set however, we keep the 'id' feature. We'll want to filter it out prior to running our inferences so that the input data features match the ones of training set and then ultimately, we'll want to join it with inference result. We are however dropping the diagnosis attribute for the batch set since this is what we'll try to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data split in three sets, training, validation and batch inference\n",
    "rand_split = np.random.rand(len(data))\n",
    "train_list = rand_split < 0.8\n",
    "val_list = (rand_split >= 0.8) & (rand_split < 0.9)\n",
    "batch_list = rand_split >= 0.9\n",
    "\n",
    "data_train = data[train_list].drop([\"id\"], axis=1)\n",
    "data_val = data[val_list].drop([\"id\"], axis=1)\n",
    "data_batch = data[batch_list].drop([\"diagnosis\"], axis=1)\n",
    "data_batch_noID = data_batch.drop([\"id\"], axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "ff9d10f9-b611-423b-80da-6dcdafd1c8b9"
    }
   },
   "source": [
    "Let's upload those data sets in S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "cd8e3431-79d9-40b6-91d1-d67cd61894e7"
    }
   },
   "outputs": [],
   "source": [
    "train_file = \"train_data.csv\"\n",
    "data_train.to_csv(train_file, index=False, header=False)\n",
    "sess.upload_data(train_file, key_prefix=\"{}/train\".format(prefix))\n",
    "\n",
    "validation_file = \"validation_data.csv\"\n",
    "data_val.to_csv(validation_file, index=False, header=False)\n",
    "sess.upload_data(validation_file, key_prefix=\"{}/validation\".format(prefix))\n",
    "\n",
    "batch_file = \"batch_data.csv\"\n",
    "data_batch.to_csv(batch_file, index=False, header=False)\n",
    "sess.upload_data(batch_file, key_prefix=\"{}/batch\".format(prefix))\n",
    "\n",
    "batch_file_noID = \"batch_data_noID.csv\"\n",
    "data_batch_noID.to_csv(batch_file_noID, index=False, header=False)\n",
    "sess.upload_data(batch_file_noID, key_prefix=\"{}/batch\".format(prefix))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "71cbcebd-a2a5-419e-8e50-b2bc0909f564"
    }
   },
   "source": [
    "---\n",
    "\n",
    "## Training job and model creation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bd113b8e-adc1-4091-a26f-a426149fe604"
    }
   },
   "source": [
    "The below cell uses the [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk) to kick off the training job using both our training set and validation set. Not that the objective is set to 'binary:logistic' which trains a model to output a probability between 0 and 1 (here the probability of a tumor being malignant)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbpresent": {
     "id": "f3b125ad-a2d5-464c-8cfa-bd203034eee4"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "from time import gmtime, strftime\n",
    "\n",
    "job_name = \"xgb-\" + strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "output_location = \"s3://{}/{}/output/{}\".format(bucket, prefix, job_name)\n",
    "image = sagemaker.image_uris.retrieve(\n",
    "    framework=\"xgboost\", region=boto3.Session().region_name, version=\"1.7-1\"\n",
    ")\n",
    "\n",
    "sm_estimator = sagemaker.estimator.Estimator(\n",
    "    image,\n",
    "    role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.4xlarge\",\n",
    "    volume_size=50,\n",
    "    input_mode=\"File\",\n",
    "    output_path=output_location,\n",
    "    sagemaker_session=sess,\n",
    ")\n",
    "\n",
    "sm_estimator.set_hyperparameters(\n",
    "    objective=\"binary:logistic\",\n",
    "    max_depth=5,\n",
    "    eta=0.2,\n",
    "    gamma=4,\n",
    "    min_child_weight=6,\n",
    "    subsample=0.8,\n",
    "    verbosity=0,\n",
    "    num_round=100,\n",
    ")\n",
    "\n",
    "train_data = sagemaker.inputs.TrainingInput(\n",
    "    \"s3://{}/{}/train\".format(bucket, prefix),\n",
    "    distribution=\"FullyReplicated\",\n",
    "    content_type=\"text/csv\",\n",
    "    s3_data_type=\"S3Prefix\",\n",
    ")\n",
    "validation_data = sagemaker.inputs.TrainingInput(\n",
    "    \"s3://{}/{}/validation\".format(bucket, prefix),\n",
    "    distribution=\"FullyReplicated\",\n",
    "    content_type=\"text/csv\",\n",
    "    s3_data_type=\"S3Prefix\",\n",
    ")\n",
    "data_channels = {\"train\": train_data, \"validation\": validation_data}\n",
    "\n",
    "# Start training by calling the fit method in the estimator\n",
    "sm_estimator.fit(inputs=data_channels, logs=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "397fb60a-c48b-453f-88ea-4d832b70c919"
    }
   },
   "source": [
    "---\n",
    "\n",
    "## Batch Transform\n",
    "\n",
    "In SageMaker Batch Transform, we introduced 3 new attributes - __input_filter__, __join_source__ and __output_filter__. In the below cell, we use the [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk) to kick-off several Batch Transform jobs using different configurations of these 3 new attributes. Please refer to [this page](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform-data-processing.html) to learn more about how to use them.\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Create a transform job with the default configurations\n",
    "Let's first skip these 3 new attributes and inspect the inference results. We'll use it as a baseline to compare to the results with data processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "sm_transformer = sm_estimator.transformer(1, \"ml.m4.xlarge\")\n",
    "\n",
    "# start a transform job\n",
    "input_location = \"s3://{}/{}/batch/{}\".format(\n",
    "    bucket, prefix, batch_file_noID\n",
    ")  # use input data without ID column\n",
    "sm_transformer.transform(input_location, content_type=\"text/csv\", split_type=\"Line\")\n",
    "sm_transformer.wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the output of the Batch Transform job in S3. It should show the list probabilities of tumors being malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "def get_csv_output_from_s3(s3uri, batch_file):\n",
    "    file_name = \"{}.out\".format(batch_file)\n",
    "    match = re.match(\"s3://([^/]+)/(.*)\", \"{}/{}\".format(s3uri, file_name))\n",
    "    output_bucket, output_prefix = match.group(1), match.group(2)\n",
    "    s3.download_file(output_bucket, output_prefix, file_name)\n",
    "    return pd.read_csv(file_name, sep=\",\", header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = get_csv_output_from_s3(sm_transformer.output_path, batch_file_noID)\n",
    "output_df.head(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Join the input and the prediction results \n",
    "Now, let's associate the prediction results with their corresponding input records. We can also use the __input_filter__ to exclude the ID column easily and there's no need to have a separate file in S3.\n",
    "\n",
    "* Set __input_filter__ to \"$[1:]\": indicates that we are excluding column 0 (the 'ID') before processing the inferences and keeping everything from column 1 to the last column (all the features or predictors)  \n",
    "  \n",
    "  \n",
    "* Set __join_source__ to \"Input\": indicates our desire to join the input data with the inference results  \n",
    "\n",
    "* Leave __output_filter__ to default ('$'), indicating that the joined input and inference results be will saved as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# content_type / accept and split_type / assemble_with are required to use IO joining feature\n",
    "sm_transformer.assemble_with = \"Line\"\n",
    "sm_transformer.accept = \"text/csv\"\n",
    "\n",
    "# start a transform job\n",
    "input_location = \"s3://{}/{}/batch/{}\".format(\n",
    "    bucket, prefix, batch_file\n",
    ")  # use input data with ID column cause InputFilter will filter it out\n",
    "sm_transformer.transform(\n",
    "    input_location,\n",
    "    split_type=\"Line\",\n",
    "    content_type=\"text/csv\",\n",
    "    input_filter=\"$[1:]\",\n",
    "    join_source=\"Input\",\n",
    ")\n",
    "sm_transformer.wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's inspect the output of the Batch Transform job in S3. It should show the list of tumors identified by their original feature columns and their corresponding probabilities of being malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = get_csv_output_from_s3(sm_transformer.output_path, batch_file)\n",
    "output_df.head(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Update the output filter to keep only ID and prediction results\n",
    "Let's change __output_filter__ to \"$[0,-1]\", indicating that when presenting the output, we only want to keep column 0 (the 'ID') and the last column (the inference result i.e. the probability of a given tumor to be malignant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start another transform job\n",
    "sm_transformer.transform(\n",
    "    input_location,\n",
    "    split_type=\"Line\",\n",
    "    content_type=\"text/csv\",\n",
    "    input_filter=\"$[1:]\",\n",
    "    join_source=\"Input\",\n",
    "    output_filter=\"$[0,-1]\",\n",
    ")\n",
    "sm_transformer.wait()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's inspect the output of the Batch Transform job in S3 again. It should show 2 columns: the ID and their corresponding probabilities of being malignant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_df = get_csv_output_from_s3(sm_transformer.output_path, batch_file)\n",
    "output_df.head(8)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, we can use newly introduced 3 attributes - __input_filter__, __join_source__, __output_filter__ to \n",
    "1. Filter / select useful features from the input dataset. e.g. exclude ID columns.\n",
    "2. Associate the prediction results with their corresponding input records.\n",
    "3. Filter the original or joined results before saving to S3. e.g. keep ID and probability columns only."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker_batch_transform|batch_transform_associate_predictions_with_input|Batch Transform - breast cancer prediction with high level SDK.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the License). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the license file accompanying this file. This file is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}