 archived/r_xgboost_hpo_batch_transform.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparamter Optimization Using R with Amazon SageMaker\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Read before running this notebook:**\n",
    "\n",
    "- This sample notebook has been updated for SageMaker SDK v2.0. \n",
    "- If you are using SageMaker Notebook instances, select `R` kernel for the notebook. If you are using SageMaker Studio notebooks, you will need to create a custom R kernel for your studio domain. Follow the instructions in this blog post to create and attach a custom R kernel. \n",
    "    - [Bringing your own R environment to Amazon SageMaker Studio](https://aws.amazon.com/blogs/machine-learning/bringing-your-own-r-environment-to-amazon-sagemaker-studio/)\n",
    "\n",
    "**Summary:**\n",
    "\n",
    "This sample Notebook demonstrates how to conduct Hyperparamter tuning and how to generate predictions for abalone age using two methods:\n",
    "\n",
    "- [Batch Transform](https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html) using a Transformer.\n",
    "- [Deploying the model](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html) as an endpoint and making online inferences. \n",
    "\n",
    "The goal is to demonstrate how these methods work in R. \n",
    "\n",
    "Abalone age is measured by the number of rings in the shell. The notebook will use the public [abalone dataset](https://archive.ics.uci.edu/ml/datasets/abalone) hosted by [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php). \n",
    "\n",
    "We will use two different libraries to interact with SageMaker:\n",
    "- [`Reticulate` library](https://rstudio.github.io/reticulate/): that provides an R interface to make API calls [Amazon SageMaker Python SDK](https://sagemaker.readthedocs.io/en/latest/index.html) to make API calls to Amazon SageMaker. The `reticulate` package translates between R and Python objects, and Amazon SageMaker provides a serverless data science environment to train and deploy ML models at scale.\n",
    "- [`paws` library](https://cran.r-project.org/web/packages/paws/index.html): that provides an interface to make API calls to AWS services, similar to how [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) works. `boto3` is the Amazon Web Services (AWS) SDK for Python. It enables Python developers to create, configure, and manage AWS services, such as EC2 and S3. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services. `paws` provides the same capabilities in R.\n",
    "\n",
    "Table of Contents:\n",
    "- [Reticulating the Amazon SageMaker Python SDK](#Reticulating-the-Amazon-SageMaker-Python-SDK)\n",
    "- [Creating and Accessing the Data Storage](#Creating-and-accessing-the-data-storage)\n",
    "- [Downloading and Processing the Dataset](#Downloading-and-processing-the-dataset)\n",
    "- [Preparing the Dataset for Model Training](#Preparing-the-dataset-for-model-training)\n",
    "- [Hyperparameter Tuning for the XGBoost Model](#Hyperparameter-Tuning-for-the-XGBoost-Model)\n",
    "    - [Using the paws Library to Interact with AWS Services and Get the Status of the Tuning Job](#Using-the-paws-Library-to-Interact-with-AWS-Services-and-Get-the-Status-of-the-Tuning-Job)\n",
    "- [Option 1: Batch Transform](#Option-1:-Batch-Transform)\n",
    "    - [Create a Model using the Best Training Job](#Create-a-Model-using-the-Best-Training-Job)\n",
    "    - [Batch Transform using the Tuned Estimator](#Batch-Transform-using-the-Tuned-Estimator)\n",
    "    - [Download the Data](#Download-the-Data)\n",
    "- [Option 2: Make Inference using Endpoint](#Option-2:-Make-Inference-using-Endpoint)\n",
    "    - [Deploying the Tuner](#Deploying-the-Tuner)\n",
    "    - [Generating Predictions with the Deployed Model](#Generating-Predictions-with-the-Deployed-Model)\n",
    "    - [Deleting the Endpoint](#Deleting-the-Endpoint)\n",
    "    \n",
    "    \n",
    "**Note:** The first portion of this notebook focused on data ingestion and preparing the data for model training is similar to the data preparation outlined in the [\"Using R with Amazon SageMaker\"](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/r_kernel/using_r_with_amazon_sagemaker.ipynb) notebook on AWS SageMaker Examples Github repository with some modifications.\n",
    "Also the last portion of this notebook focused on making inference using an endpoint is inspired by the method outlined in the notebook referenced [here](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/r_examples/r_end_2_end/r_sagemaker_abalone.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reticulating the Amazon SageMaker Python SDK\n",
    "\n",
    "First, load the `reticulate` library and import the `sagemaker` Python module. Once the module is loaded, use the `$` notation in R instead of the `.` notation in Python to use available classes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn warnings off globally\n",
    "options(warn=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install reticulate library and import sagemaker\n",
    "library(reticulate)\n",
    "sagemaker <- import('sagemaker')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating and Accessing the Data Storage\n",
    "\n",
    "The `Session` class provides operations for working with the following [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) resources with Amazon SageMaker:\n",
    "\n",
    "* [S3](https://boto3.readthedocs.io/en/latest/reference/services/s3.html)\n",
    "* [SageMaker](https://boto3.readthedocs.io/en/latest/reference/services/sagemaker.html)\n",
    "* [SageMakerRuntime](https://boto3.readthedocs.io/en/latest/reference/services/sagemaker-runtime.html)\n",
    "\n",
    "Let's create an [Amazon Simple Storage Service](https://aws.amazon.com/s3/) bucket for your data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session <- sagemaker$Session()\n",
    "bucket <- session$default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** - The `default_bucket` function creates a unique Amazon S3 bucket with the following name: \n",
    "\n",
    "`sagemaker-<aws-region-name>-<aws account number>`\n",
    "\n",
    "Specify the IAM role's [ARN](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) to allow Amazon SageMaker to access the Amazon S3 bucket. You can use the same IAM role used to create this Notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "role_arn <- sagemaker$get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading and Processing the Dataset\n",
    "\n",
    "The model uses the [abalone dataset](https://archive.ics.uci.edu/ml/datasets/abalone) from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php). First, download the data and start the [exploratory data analysis](https://en.wikipedia.org/wiki/Exploratory_data_analysis). Use tidyverse packages to read, plot, and transform the data into ML format for Amazon SageMaker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(readr)\n",
    "data_file <- 's3://sagemaker-sample-files/datasets/tabular/uci_abalone/abalone.csv'\n",
    "abalone <- read_csv(file = sagemaker$s3$S3Downloader$read_file(data_file, sagemaker_session=session), col_names = FALSE)\n",
    "names(abalone) <- c('sex', 'length', 'diameter', 'height', 'whole_weight', 'shucked_weight', 'viscera_weight', 'shell_weight', 'rings')\n",
    "head(abalone)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above shows that `sex` is a factor data type but is currently a character data type (F is Female, M is male, and I is infant). Change `sex` to a factor and view the statistical summary of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abalone$sex <- as.factor(abalone$sex)\n",
    "summary(abalone)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary above shows that the minimum value for `height` is 0.\n",
    "\n",
    "Visually explore which abalones have height equal to 0 by plotting the relationship between `rings` and `height` for each value of `sex`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(ggplot2)\n",
    "options(repr.plot.width = 5, repr.plot.height = 4) \n",
    "ggplot(abalone, aes(x = height, y = rings, color = sex)) + geom_point() + geom_jitter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot shows multiple outliers: two infant abalones with a height of 0 and a few female and male abalones with greater heights than the rest. Let's filter out the two infant abalones with a height of 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(dplyr)\n",
    "abalone <- abalone %>%\n",
    "  filter(height != 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Dataset for Model Training\n",
    "\n",
    "The model needs three datasets: one for training, testing, and validation. First, convert `sex` into a [dummy variable](https://en.wikipedia.org/wiki/Dummy_variable_(statistics)) and move the target, `rings`, to the first column. Amazon SageMaker algorithm require the target to be in the first column of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abalone <- abalone %>%\n",
    "  mutate(female = as.integer(ifelse(sex == 'F', 1, 0)),\n",
    "         male = as.integer(ifelse(sex == 'M', 1, 0)),\n",
    "         infant = as.integer(ifelse(sex == 'I', 1, 0))) %>%\n",
    "  select(-sex)\n",
    "abalone <- abalone %>%\n",
    "  select(rings:infant, length:shell_weight)\n",
    "head(abalone)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, sample 70% of the data for training the ML algorithm. Split the remaining 30% into two halves, one for testing and one for validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abalone_train <- abalone %>%\n",
    "  sample_frac(size = 0.7)\n",
    "abalone <- anti_join(abalone, abalone_train)\n",
    "abalone_test <- abalone %>%\n",
    "  sample_frac(size = 0.5)\n",
    "abalone_valid <- anti_join(abalone, abalone_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later in the notebook, we are going to use Batch Transform and Endpoint to make inference in two different ways and we will compare the results. The maximum number of rows that we can send to an endpoint for inference in one batch is 500 rows. We are going to reduce the number of rows for the test dataset to 500 and use this for batch and online inference for comparison. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_predict_rows <- 500\n",
    "abalone_test <- abalone_test[1:num_predict_rows, ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the training and validation data to Amazon S3 so that you can train the model. First, write the training and validation datasets to the local filesystem in .csv format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "write_csv(abalone_train, 'abalone_train.csv', col_names = FALSE)\n",
    "write_csv(abalone_valid, 'abalone_valid.csv', col_names = FALSE)\n",
    "\n",
    "# Remove target from test\n",
    "write_csv(abalone_test[-1], 'abalone_test.csv', col_names = FALSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, upload the two datasets to the Amazon S3 bucket into the `data` key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_train <- session$upload_data(path = 'abalone_train.csv', \n",
    "                                bucket = bucket, \n",
    "                                key_prefix = 'data')\n",
    "s3_valid <- session$upload_data(path = 'abalone_valid.csv', \n",
    "                                bucket = bucket, \n",
    "                                key_prefix = 'data')\n",
    "\n",
    "s3_test <- session$upload_data(path = 'abalone_test.csv', \n",
    "                                bucket = bucket, \n",
    "                                key_prefix = 'data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, define the Amazon S3 input types for the Amazon SageMaker algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_train_input <- sagemaker$inputs$TrainingInput(s3_data = s3_train,\n",
    "                                     content_type = 'csv')\n",
    "s3_valid_input <- sagemaker$inputs$TrainingInput(s3_data = s3_valid,\n",
    "                                     content_type = 'csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning for the XGBoost Model\n",
    "\n",
    "Amazon SageMaker algorithms are available via a [Docker](https://www.docker.com/) container. To train an [XGBoost](https://en.wikipedia.org/wiki/Xgboost) model, specify the training containers in [Amazon Elastic Container Registry](https://aws.amazon.com/ecr/) (Amazon ECR) for the AWS Region. We will use the `latest` version of the algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "container <- sagemaker$image_uris$retrieve(framework='xgboost', region= session$boto_region_name, version='latest')\n",
    "cat('XGBoost Container Image URL: ', container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define an Amazon SageMaker [Estimator](http://sagemaker.readthedocs.io/en/latest/estimators.html), which can train any supplied algorithm that has been containerized with Docker. When creating the Estimator, use the following arguments:\n",
    "* **image_uri** - The container image to use for training\n",
    "* **role** - The Amazon SageMaker service role\n",
    "* **train_instance_count** - The number of Amazon EC2 instances to use for training\n",
    "* **train_instance_type** - The type of Amazon EC2 instance to use for training\n",
    "* **train_volume_size** - The size in GB of the [Amazon Elastic Block Store](https://aws.amazon.com/ebs/) (Amazon EBS) volume to use for storing input data during training\n",
    "* **train_max_run** - The timeout in seconds for training\n",
    "* **input_mode** - The input mode that the algorithm supports\n",
    "* **output_path** - The Amazon S3 location for saving the training results (model artifacts and output files)\n",
    "* **output_kms_key** - The [AWS Key Management Service](https://aws.amazon.com/kms/) (AWS KMS) key for encrypting the training output\n",
    "* **base_job_name** - The prefix for the name of the training job\n",
    "* **sagemaker_session** - The Session object that manages interactions with Amazon SageMaker API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_output <- paste0('s3://', bucket, '/output')\n",
    "estimator <- sagemaker$estimator$Estimator(image_uri = container,\n",
    "                                           role = role_arn,\n",
    "                                           train_instance_count = 1L,\n",
    "                                           train_instance_type = 'ml.m5.4xlarge',\n",
    "                                           train_volume_size = 30L,\n",
    "                                           train_max_run = 3600L,\n",
    "                                           input_mode = 'File',\n",
    "                                           output_path = s3_output,\n",
    "                                           output_kms_key = NULL,\n",
    "                                           base_job_name = NULL,\n",
    "                                           sagemaker_session = NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** - The equivalent to `None` in Python is `NULL` in R.\n",
    "\n",
    "Next, we Specify the [XGBoost hyperparameters](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html) for the estimator, and also define the range of hyperparameters that we want to use for [SageMaker Hyperparamter Tuning](https://sagemaker.readthedocs.io/en/stable/tuner.html). You can find the list of [Tunable Hyperparamters for XGBoost algorithm here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html). \n",
    "\n",
    "In addition, you need to specify the tuning evaluation metric. XGboost allows one of these nine objectives to be used (for the description of these objectives visit [\"Tune an XGBoost Model\"](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html) page) :\n",
    "\n",
    "- validation:accuracy\n",
    "- validation:auc\n",
    "- validation:error\n",
    "- validation:f1\n",
    "- validation:logloss\n",
    "- validation:mae\n",
    "- validation:map\n",
    "- validation:merror\n",
    "- validation:mlogloss\n",
    "- validation:mse\n",
    "- validation:ndcg\n",
    "- validation:rmse\t\n",
    "\n",
    "In this case, since this is a regression problem, we select `validation:rmse` as the tuning objective.\n",
    "\n",
    "For tuning the hyperparamters you need to also specify the type and range of hyperparamters to be tuned. You can specify either a `ContinuousParameter` or an `IntegerParameter`, as outlined in the documentation. In addition, the algorithm documentation provides suggestions for the hyperparamter range.\n",
    "\n",
    "\n",
    "Once the Estimator and its hyperparamters and tunable hyperparamter ranges are specified, you can create a `HyperparameterTuner` (tuner). You can train (or fit) that tuner which will conduct the tuning and will select the most optimzied model. You can then generate predictions using this model with Batch Transform, or by deploying the model as an endpoint and using it for online inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set Hyperparameters\n",
    "estimator$set_hyperparameters(eval_metric='rmse',\n",
    "                              objective='reg:linear',\n",
    "                              num_round=100L,\n",
    "                              rate_drop=0.3,\n",
    "                              tweedie_variance_power=1.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set Hyperparameter Ranges\n",
    "hyperparameter_ranges = list('eta' = sagemaker$parameter$ContinuousParameter(0,1),\n",
    "                        'min_child_weight'= sagemaker$parameter$ContinuousParameter(0,10),\n",
    "                        'alpha'= sagemaker$parameter$ContinuousParameter(0,2),\n",
    "                        'max_depth'= sagemaker$parameter$IntegerParameter(0L,10L))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the tuning objective to RMSE\n",
    "objective_metric_name = 'validation:rmse'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `HyperparameterTuner` accepts multiple paramters. A short list of these parameters are described below. For the complete list and more details you can visit [`HyperparameterTuner` Documentation](https://sagemaker.readthedocs.io/en/stable/tuner.html#hyperparametertuner) :\n",
    "\n",
    "- **estimator** (sagemaker.estimator.EstimatorBase) \u2013 An estimator object that has been initialized with the desired configuration. There does not need to be a training job associated with this instance.\n",
    "- **objective_metric_name** (str) \u2013 Name of the metric for evaluating training jobs.\n",
    "- **hyperparameter_ranges** (dict[str, sagemaker.parameter.ParameterRange]) \u2013 Dictionary of parameter ranges. These parameter ranges can be one of three types: Continuous, Integer, or Categorical. \n",
    "- **objective_type** (str) \u2013 The type of the objective metric for evaluating training jobs. This value can be either \u2018Minimize\u2019 or \u2018Maximize\u2019 (default: \u2018Maximize\u2019).\n",
    "- **max_jobs** (int) \u2013 Maximum total number of training jobs to start for the hyperparameter tuning job (default: 1).\n",
    "- **max_parallel_jobs** (int) \u2013 Maximum number of parallel training jobs to start (default: 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a hyperparamter tuner\n",
    "tuner <- sagemaker$tuner$HyperparameterTuner(estimator,\n",
    "                                             objective_metric_name,\n",
    "                                             hyperparameter_ranges,\n",
    "                                             objective_type='Minimize',\n",
    "                                             max_jobs=10L,\n",
    "                                             max_parallel_jobs=4L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tuning job name\n",
    "job_name <- paste('tune-xgboost', format(Sys.time(), '%Y%m%d-%H-%M-%S'), sep = '-')\n",
    "\n",
    "# Define the data channels for train and validation datasets\n",
    "input_data <- list('train' = s3_train_input,\n",
    "                   'validation' = s3_valid_input)\n",
    "\n",
    "# train the tuner\n",
    "tuner$fit(inputs = input_data, job_name = job_name, wait=FALSE)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the `boto3` SDK to Interact with AWS Services and Get the Status of the Tuning Job\n",
    "\n",
    "With [`boto3` Python SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) you can to create, configure, and manage AWS services, such as Amazon Simple Storage Service (Amazon S3), Amazon SageMaker and other AWS services. The SDK provides an object-oriented API as well as low-level access to AWS services. Using `reticulate` library, you can leverage this SDK in R. \n",
    "\n",
    "Since running a tuning job may take a while, we are going to use `boto3` to get the status of the tuning job using `sagemaker$describe_hyper_parameter_tuning_job`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "boto3_r <- import('boto3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a paws SageMaker session\n",
    "sm <- boto3_r$client('sagemaker')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the status of the tuning job\n",
    "status <- sm$describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuner$latest_tuning_job$job_name)\n",
    "\n",
    "message(cat('Hyperparameter Tuning Job Name: ', job_name,'\\n'))\n",
    "while (status$HyperParameterTuningJobStatus != \"Completed\") {\n",
    "    message(cat('Hyperparameter Tuning Job Status: ', status$HyperParameterTuningJobStatus,'\\n'))\n",
    "    message(cat('Succeeded Models:', status$ObjectiveStatusCounters$Succeeded, ' InProgress Models: ', status$ObjectiveStatusCounters$Pending, ' Failed Models: ', status$ObjectiveStatusCounters$Failed, '\\n'))\n",
    "    Sys.sleep(60)\n",
    "    status <- sm$describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuner$latest_tuning_job$job_name)    \n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print best training hyperparamters\n",
    "status$BestTrainingJob$TunedHyperParameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print Evaluation Metric\n",
    "status$BestTrainingJob$FinalHyperParameterTuningJobObjectiveMetric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Name of the best training job model\n",
    "status$BestTrainingJob$TrainingJobName"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Option 1: Batch Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Model using the Best Training Job\n",
    "This section demonstrates how to create a model using the best training job results from the HPO task, and using the model artifacts saved on S3.\n",
    "\n",
    "First, we need to create a model container, which needs the following parameters:\n",
    "- **Image:** URL of the algorithm container \n",
    "- **ModelDataUrl:** Location of the model tar ball (model.tar.gz) on S3 that is saved by the Hyperparamter training job\n",
    "\n",
    "We can extract the **ModelDataUrl** by describing the best training job using `boto3` SDK and `describe_training_job()` method. [More details can be found here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.describe_training_job).\n",
    " \n",
    "Then we will create a model using this model container. We will use `paws` library and `create_model` method. [Documentation of this method can be found here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe best training model from hypertuning to get the location of the model artifact on S3\n",
    "model_artifact <- sm$describe_training_job(\n",
    "    TrainingJobName = status$BestTrainingJob$TrainingJobName\n",
    ")$ModelArtifacts$S3ModelArtifacts\n",
    "\n",
    "model_artifact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a model container\n",
    "model_container <- list(\n",
    "    \"Image\"= container,\n",
    "    \"ModelDataUrl\" = model_artifact\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a model\n",
    "\n",
    "model_name <- paste('model-xgboost', format(Sys.time(), '%Y%m%d-%H-%M-%S'), sep = '-')\n",
    "\n",
    "best_model <- sm$create_model(\n",
    "    ModelName = model_name,\n",
    "    PrimaryContainer = model_container,\n",
    "    ExecutionRoleArn = role_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch Transform using the Tuned Estimator\n",
    "\n",
    "For more details on SageMaker Batch Transform, you can visit this example notebook on [Amazon SageMaker Batch Transform](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/introduction_to_batch_transform/batch_transform_pca_dbscan_movie_clusters.ipynb).\n",
    "\n",
    "In many situations, using a deployed model for making inference is not the best option, especially when the goal is not to make online real-time inference but to generate predictions from a trained model on a large dataset. In these situations, using Batch Transform may be more efficient and appropriate.\n",
    "\n",
    "This section of the notebook explain how to set up the Batch Transform Job, and generate predictions.\n",
    "\n",
    "To do this, first we need to define the batch input data path on S3, and also where to save the generated predictions on S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define S3 path for Test data and output path\n",
    "\n",
    "s3_test_url <- paste('s3:/',bucket,'data','abalone_test.csv', sep = '/')\n",
    "output_path <- paste('s3:/',bucket,'output/batch_transform_output',job_name, sep = '/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create a `Transformer`. [Transformers](https://sagemaker.readthedocs.io/en/stable/transformer.html#transformer) take multiple paramters, including the following. For more details and the complete list visit the [documentation page](https://sagemaker.readthedocs.io/en/stable/transformer.html#transformer).\n",
    "\n",
    "- **model_name** (str) \u2013 Name of the SageMaker model being used for the transform job.\n",
    "- **instance_count** (int) \u2013 Number of EC2 instances to use.\n",
    "- **instance_type** (str) \u2013 Type of EC2 instance to use, for example, \u2018ml.c4.xlarge\u2019.\n",
    "\n",
    "- **output_path** (str) \u2013 S3 location for saving the transform result. If not specified, results are stored to a default bucket.\n",
    "\n",
    "- **base_transform_job_name** (str) \u2013 Prefix for the transform job when the transform() method launches. If not specified, a default prefix will be generated based on the training image name that was used to train the model associated with the transform job.\n",
    "\n",
    "- **sagemaker_session** (sagemaker.session.Session) \u2013 Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed. If not specified, the estimator creates one using the default AWS configuration chain.\n",
    "\n",
    "Once we create a `Transformer` we can transform the batch input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a SageMaker transformer\n",
    "transformer <- sagemaker$transformer$Transformer(\n",
    "    model_name = model_name,\n",
    "    instance_count=1L,\n",
    "    instance_type='ml.m5.4xlarge',\n",
    "    output_path=output_path,\n",
    "    base_transform_job_name='R-Transformer',\n",
    "    sagemaker_session=session)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tranform the test data and wait until the task completes\n",
    "transformer$transform(s3_test_url)\n",
    "transformer$wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the status of Batch Transform\n",
    "sm$describe_transform_job(TransformJobName = transformer$latest_transform_job$job_name)$TransformJobStatus\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker$s3$S3Downloader$download(paste(output_path,\"abalone_test.csv.out\",sep = '/'),\n",
    "                          \"batch_output\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the batch csv from sagemaker local files\n",
    "library(readr)\n",
    "predictions <- read_csv(file = 'batch_output/abalone_test.csv.out', col_names = 'predicted_rings')\n",
    "head(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Column-bind the predicted rings to the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Concatenate predictions and test for comparison\n",
    "abalone_predictions <- cbind(predicted_rings = predictions, \n",
    "                      abalone_test)\n",
    "# Convert predictions to Integer\n",
    "abalone_predictions$predicted_rings = as.integer(abalone_predictions$predicted_rings);\n",
    "head(abalone_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a function to calculate RMSE\n",
    "rmse <- function(m, o){\n",
    "  sqrt(mean((m - o)^2))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calucalte RMSE\n",
    "abalone_rmse <- rmse(abalone_predictions$rings, abalone_predictions$predicted_rings)\n",
    "cat('RMSE for Batch Transform: ', round(abalone_rmse, digits = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Option 2: Make Inference using Endpoint\n",
    "### Deploying the Tuner\n",
    "\n",
    "This section walks you through the deployment process of the tuned/trained model. We will then use the deployed model (as an endpoint) to make predictions using the test data. Deploying the model as as endpoint is suitable for cases where you need to make online inference. For making predictions using batch data, the preferred method is using Batch Transform, which was demonstrated in the previous section.\n",
    "\n",
    "Amazon SageMaker lets you [deploy your model](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html) by providing an endpoint that consumers can invoke by a secure and simple API call using an HTTPS request. Let's deploy our trained model to a `ml.t2.medium` instance. This will take a couple of minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_endpoint <- tuner$deploy(initial_instance_count = 1L,\n",
    "                                   instance_type = 'ml.t2.medium')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating Predictions with the Deployed Model\n",
    "\n",
    "Use the test data to generate predictions. Pass comma-separated text to be serialized into JSON format by specifying `text/csv` and `csv_serializer` for the endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_endpoint$serializer <- sagemaker$serializers$CSVSerializer(content_type='text/csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove the target column and convert the dataframe to a matrix with no column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_sample <- as.matrix(abalone_test[-1])\n",
    "dimnames(test_sample)[[2]] <- NULL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** - 500 observations was chosen because it doesn't exceed the endpoint limitation.\n",
    "\n",
    "Generate predictions from the endpoint and convert the returned comma-separated string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(stringr)\n",
    "predictions_ep <- model_endpoint$predict(test_sample)\n",
    "predictions_ep <- str_split(predictions_ep, pattern = ',', simplify = TRUE)\n",
    "predictions_ep <- as.integer(unlist(predictions_ep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Column-bind the predicted rings to the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Convert predictions to Integer\n",
    "abalone_predictions_ep <- cbind(predicted_rings = predictions_ep, \n",
    "                      abalone_test)\n",
    "# abalone_predictions = as.integer(abalone_predictions)\n",
    "head(abalone_predictions_ep)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calucalte RMSE\n",
    "abalone_rmse_ep <- rmse(abalone_predictions_ep$rings, abalone_predictions_ep$predicted_rings)\n",
    "cat('RMSE for Endpoint 500-Row Prediction: ', round(abalone_rmse_ep, digits = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting the Endpoint\n",
    "\n",
    "When you're done with the model, delete the endpoint to avoid incurring deployment costs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_endpoint$delete_model()\n",
    "session$delete_endpoint(model_endpoint$endpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer$delete_model()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/r_examples|r_xgboost_hpo_batch_transform|r_xgboost_hpo_batch_transform.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.0.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}