 archived/R_binary_classification_algorithms_comparison.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf4dc0b3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e4fd44cc",
   "metadata": {},
   "source": [
    "## Compare built-in Sagemaker classification algorithms for a binary classification problem using Iris dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69f342df",
   "metadata": {},
   "source": [
    "In the notebook tutorial, we build 3 classification models using HPO and then compare the AUC on test dataset on 3 deployed models\n",
    "\n",
    "IRIS is perhaps the best known database to be found in the pattern recognition literature. Fisher's paper is a classic in the field and is referenced frequently to this day. (See Duda & Hart, for example.) The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. The dataset is built-in by default into R or can also be downloaded from https://archive.ics.uci.edu/ml/datasets/iris\n",
    "\n",
    "The iris dataset, besides its historical importance, is also a fun dataset to play with since it can educate us about various ML techniques such as clustering, classification and regression, all in one dataset.\n",
    "\n",
    "The dataset is built into any base R installation, so no download is required.\n",
    "\n",
    "Attribute Information:\n",
    "\n",
    "1. sepal length in cm\n",
    "2. sepal width in cm\n",
    "3. petal length in cm\n",
    "4. petal width in cm\n",
    "5. Species of flowers: Iris setosa, Iris versicolor, Iris virginica\n",
    "\n",
    "The prediction we will perform is `Species ~ f(sepal.length,sepal.width,petal.width,petal.length)`\n",
    "\n",
    "Predicted attribute: Species of iris plant.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b7bc401",
   "metadata": {},
   "source": [
    "### Load required libraries and initialize variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae96595e",
   "metadata": {},
   "outputs": [],
   "source": [
    "rm(list=ls())\n",
    "library(reticulate) # be careful not to install reticulate again. since it can cause problems.\n",
    "library(tidyverse)\n",
    "library(pROC)\n",
    "set.seed(1324)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ef8e97",
   "metadata": {},
   "source": [
    "SageMaker needs to be imported using the reticulate library. If this was performed in a local computer, we would have to make sure that Python and appropriate SageMaker libraries are installed, but inside a SageMaker notebook R kernels, these are all pre-loaded and the R user does not have to worry about installing reticulate or Python. \n",
    "\n",
    "Session is the unique session ID associated with each SageMaker call. It remains the same throughout the execution of the program and can be recalled later to close a session or open a new session.\n",
    "\n",
    "The bucket is the Amazon S3 bucket where we will be storing our data output. The Amazon S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "\n",
    "The role is the role of the SageMaker notebook as when it was initially deployed. The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the boto regexp with appropriate full IAM role arn string(s).\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "107a91cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker <- import('sagemaker')\n",
    "session <- sagemaker$Session()\n",
    "bucket <- session$default_bucket() # you may replace with name of your personal S3 bucket\n",
    "role_arn <- sagemaker$get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78ac95c7",
   "metadata": {},
   "source": [
    "### Input the data and basic pre-processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5be74fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "head(iris)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "190824ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(iris)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0391e7e",
   "metadata": {},
   "source": [
    "In above, we see that there are 50 flowers of the setosa species, 50 flowers of the versicolor species, and 50 flowers of the virginica species."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a67ad2",
   "metadata": {},
   "source": [
    "In this case, the target variable is the Species prediction. We are trying to predict the species of the flower given its numerical measurements of Sepal length, sepal width, petal length, and petal width. Since we are trying to do binary classification, we will only take the flower species setosa and versicolor for simplicity. Also we will perform one-hot encoding on the categorical variable Species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c699c5b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris1 <- iris %>% \n",
    "    dplyr::select(Species,Sepal.Length,Sepal.Width,Petal.Length,Petal.Width) %>% # change order of columns such that the label column is the first column.\n",
    "    dplyr::filter(Species %in% c(\"setosa\",\"versicolor\")) %>%                     #only select two flower for binary classification.\n",
    "    dplyr::mutate(Species = as.numeric(Species) -1)                              # one-hot encoding,starting with 0 as setosa and 1 as versicolor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e338575f",
   "metadata": {},
   "outputs": [],
   "source": [
    "head(iris1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82954ca2",
   "metadata": {},
   "source": [
    "We now obtain some basic descriptive statistics of the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c47022b",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris1 %>% group_by(Species) %>% summarize(mean_sepal_length = mean(Sepal.Length),\n",
    "                                         mean_petal_length = mean(Petal.Length),\n",
    "                                         mean_sepal_width = mean(Sepal.Width),\n",
    "                                         mean_petal_width = mean(Petal.Width),\n",
    "                                         )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e492f0",
   "metadata": {},
   "source": [
    "In the summary statistics, we observe that mean sepal length is longer than mean petal length for both flowers. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7833b26",
   "metadata": {},
   "source": [
    "### Prepare for modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd7b7a46",
   "metadata": {},
   "source": [
    "##### We split the train and test and validate into 70%, 15%, and 15%, using random sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4f6d609",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_train <- iris1 %>%\n",
    "                    sample_frac(size = 0.7)\n",
    "iris_test <- anti_join(iris1, iris_train) %>%  \n",
    "                  sample_frac(size = 0.5)\n",
    "iris_validate <- anti_join(iris1, iris_train) %>%\n",
    "                        anti_join(., iris_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55124158",
   "metadata": {},
   "source": [
    "##### We do a check of the summary statistics to make sure train, test, validate datasets are appropriately split and have proper class balance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e4a2cc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "table(iris_train$Species)\n",
    "nrow(iris_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b970cf0",
   "metadata": {},
   "source": [
    "We see that the class balance between 0 and 1 is almost 50% each for the binary classification. We also see that there are 70 rows in the train dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "207d6b36",
   "metadata": {},
   "outputs": [],
   "source": [
    "table(iris_validate$Species)\n",
    "nrow(iris_validate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba950db",
   "metadata": {},
   "source": [
    "We see that the class balance in validation dataset between 0 and 1 is almost 50% each for the binary classification. We also see that there are 15 rows in the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d76514ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "table(iris_test$Species)\n",
    "nrow(iris_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2271689",
   "metadata": {},
   "source": [
    "We see that the class balance in test dataset between 0 and 1 is almost 50% each for the binary classification. We also see that there are 15 rows in the test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2307a959",
   "metadata": {},
   "source": [
    "### Write the data to Amazon S3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55026467",
   "metadata": {},
   "source": [
    "Different algorithms in SageMaker will have different data formats required for training and for testing. These formats are created to make model production easier. csv is the most well known of these formats and has been used here as input in all algorithms to make it consistent.\n",
    "\n",
    "SageMaker algorithms take in data from an Amazon S3 object and output data to an Amazon S3 object, so data has to be stored in Amazon S3 as csv,json, proto-buf or any format that is supported by the algorithm that you are going to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f25eabdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "write_csv(iris_train, 'iris_train.csv', col_names = FALSE)\n",
    "write_csv(iris_validate, 'iris_valid.csv', col_names = FALSE)\n",
    "write_csv(iris_test, 'iris_test.csv', col_names = FALSE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d63db1d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_train <- session$upload_data(path = 'iris_train.csv', \n",
    "                                bucket = bucket, \n",
    "                                key_prefix = 'data')\n",
    "s3_valid <- session$upload_data(path = 'iris_valid.csv', \n",
    "                                bucket = bucket, \n",
    "                                key_prefix = 'data')\n",
    "\n",
    "s3_test <- session$upload_data(path = 'iris_test.csv', \n",
    "                                bucket = bucket, \n",
    "                                key_prefix = 'data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54df2b27",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_train_input <- sagemaker$inputs$TrainingInput(s3_data = s3_train,\n",
    "                                     content_type = 'text/csv')\n",
    "s3_valid_input <- sagemaker$inputs$TrainingInput(s3_data = s3_valid,\n",
    "                                     content_type = 'text/csv')\n",
    "s3_test_input <- sagemaker$inputs$TrainingInput(s3_data = s3_test,\n",
    "                                     content_type = 'text/csv')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a2542c",
   "metadata": {},
   "source": [
    "To perform Binary classification on Tabular\tdata, SageMaker contains following algorithms:\n",
    "\n",
    "- XGBoost Algorithm\n",
    "- Linear Learner Algorithm, \n",
    "- K-Nearest Neighbors (k-NN) Algorithm, \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c01f88a2",
   "metadata": {},
   "source": [
    "## Create model 1: XGBoost model in SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d99ff2bc",
   "metadata": {},
   "source": [
    "Use the XGBoost built-in algorithm to build an XGBoost training container as shown in the following code example. You can automatically spot the XGBoost built-in algorithm image URI using the SageMaker image_uris.retrieve API (or the get_image_uri API if using Amazon SageMaker Python SDK version 1). If you want to ensure if the image_uris.retrieve API finds the correct URI, see Common parameters for built-in algorithms and look up XGBoost from the full list of built-in algorithm image URIs and available regions.\n",
    "\n",
    "After specifying the XGBoost image URI, you can use the XGBoost container to construct an estimator using the SageMaker Estimator API and initiate a training job. This XGBoost built-in algorithm mode does not incorporate your own XGBoost training script and runs directly on the input datasets.\n",
    "\n",
    "See https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29145cfe",
   "metadata": {},
   "outputs": [],
   "source": [
    "container <- sagemaker$image_uris$retrieve(framework='xgboost', region= session$boto_region_name, version='latest')\n",
    "cat('XGBoost Container Image URL: ', container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07325371",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_output <- paste0('s3://', bucket, '/output_xgboost')\n",
    "estimator1 <- sagemaker$estimator$Estimator(image_uri = container,\n",
    "                                           role = role_arn,\n",
    "                                           train_instance_count = 1L,\n",
    "                                           train_instance_type = 'ml.m5.4xlarge',\n",
    "                                           input_mode = 'File',\n",
    "                                           output_path = s3_output,\n",
    "                                           output_kms_key = NULL,\n",
    "                                           base_job_name = NULL,\n",
    "                                           sagemaker_session = NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a7b6207",
   "metadata": {},
   "source": [
    "How would an untuned model perform compared to a tuned model? Is it worth the effort? Before going deeper into XGBoost model tuning, let\u2019s highlight the reasons why you have to tune your model. The main reason to perform hyper-parameter tuning is to increase predictability of our models by choosing our hyperparameters in a well thought manner. There are 3 ways to perform hyperparameter tuning: grid search, random search, bayesian search. Popular packages like scikit-learn use grid search and random search techniques. SageMaker uses Bayesian search techniques.\n",
    "\n",
    "We need to choose \n",
    "\n",
    "- a learning objective function to optimize during model training\n",
    "- an eval_metric to use to evaluate model performance during validation\n",
    "- a set of hyperparameters and a range of values for each to use when tuning the model automatically\n",
    "\n",
    "SageMaker XGBoost model can be tuned with many hyperparameters. The hyperparameters that have the greatest effect on optimizing the XGBoost evaluation metrics are: \n",
    "\n",
    "- alpha, \n",
    "- min_child_weight, \n",
    "- subsample, \n",
    "- eta, \n",
    "- num_round.\n",
    "\n",
    "\n",
    "The hyperparameters that are required are num_class (the number of classes if it is a multi-class classification problem) and num_round ( the number of rounds to run the training on). All other hyperparameters are optional and will be set to default values if it is not specified by the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c33f178",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check to make sure which are required and which are optional\n",
    "estimator1$set_hyperparameters(eval_metric='auc',\n",
    "                              objective='binary:logistic',\n",
    "                              num_round = 6L\n",
    "                              )\n",
    "\n",
    "# Set Hyperparameter Ranges, check to make sure which are integer and which are continuos parameters. \n",
    "hyperparameter_ranges = list('eta' = sagemaker$parameter$ContinuousParameter(0,1),\n",
    "                        'min_child_weight'= sagemaker$parameter$ContinuousParameter(0,10),\n",
    "                        'alpha'= sagemaker$parameter$ContinuousParameter(0,2),\n",
    "                        'max_depth'= sagemaker$parameter$IntegerParameter(0L,10L))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42771dbc",
   "metadata": {},
   "source": [
    "The evaluation metric that we will use for our binary classification purpose is validation:auc, but you could use any other metric that is right for your problem. You do have to be careful to change your objective_type to point to the right direction of Maximize or Minimize according to the objective metric you have chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "056f2add",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a hyperparamter tuner\n",
    "objective_metric_name = 'validation:auc'\n",
    "tuner1 <- sagemaker$tuner$HyperparameterTuner(estimator1,\n",
    "                                             objective_metric_name,\n",
    "                                             hyperparameter_ranges,\n",
    "                                             objective_type='Maximize',\n",
    "                                             max_jobs=4L,\n",
    "                                             max_parallel_jobs=2L)\n",
    "\n",
    "# Define the data channels for train and validation datasets\n",
    "input_data <- list('train' = s3_train_input,\n",
    "                   'validation' = s3_valid_input)\n",
    "\n",
    "# train the tuner\n",
    "tuner1$fit(inputs = input_data, \n",
    "           job_name = paste('tune-xgb', format(Sys.time(), '%Y%m%d-%H-%M-%S'), sep = '-'), \n",
    "           wait=TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79451cba",
   "metadata": {},
   "source": [
    "The output of the tuning job can be checked in SageMaker if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "278ea863",
   "metadata": {},
   "source": [
    "### Calculate AUC for the test data on model 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d1951de",
   "metadata": {},
   "source": [
    "SageMaker will automatically recognize the training job with the best evaluation metric and load the hyperparameters associated with that training job when we deploy the model. One of the benefits of SageMaker is that we can easily deploy models in a different instance than the instance in which the notebook is running. So we can deploy into a more powerful instance or a less powerful instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8596325",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_endpoint1 <- tuner1$deploy(initial_instance_count = 1L,\n",
    "                                   instance_type = 'ml.t2.medium')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "181de542",
   "metadata": {},
   "source": [
    "The serializer tells SageMaker what format the model expects data to be input in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05d97e07",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_endpoint1$serializer <- sagemaker$serializers$CSVSerializer(content_type='text/csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec87668",
   "metadata": {},
   "source": [
    "We input the `iris_test` dataset without the labels into the model using the `predict` function and check its AUC value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc3ce0c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the test sample for input into the model\n",
    "test_sample <- as.matrix(iris_test[-1])\n",
    "dimnames(test_sample)[[2]] <- NULL\n",
    "\n",
    "# Predict using the deployed model\n",
    "predictions_ep <- model_endpoint1$predict(test_sample)\n",
    "predictions_ep <- stringr::str_split(predictions_ep, pattern = ',', simplify = TRUE)\n",
    "predictions_ep <- as.numeric(predictions_ep > 0.5)\n",
    "\n",
    "# Add the predictions to the test dataset.\n",
    "iris_predictions_ep1 <- dplyr::bind_cols(predicted_flower = predictions_ep, \n",
    "                      iris_test)\n",
    "iris_predictions_ep1\n",
    "\n",
    "# Get the AUC\n",
    "auc(roc(iris_predictions_ep1$predicted_flower,iris_test$Species))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d29c25b7",
   "metadata": {},
   "source": [
    "## Create model 2: Linear Learner in SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42aa8a6a",
   "metadata": {},
   "source": [
    "Linear models are supervised learning algorithms used for solving either classification or regression problems. For input, you give the model labeled examples (x, y). x is a high-dimensional vector and y is a numeric label. For binary classification problems, the label must be either 0 or 1.\n",
    "\n",
    "The linear learner algorithm requires a data matrix, with rows representing the observations, and columns representing the dimensions of the features. It also requires an additional column that contains the labels that match the data points. At a minimum, Amazon SageMaker linear learner requires you to specify input and output data locations, and objective type (classification or regression) as arguments. The feature dimension is also required. You can specify additional parameters in the HyperParameters string map of the request body. These parameters control the optimization procedure, or specifics of the objective function that you train on. For example, the number of epochs, regularization, and loss type.\n",
    "\n",
    "See https://docs.aws.amazon.com/sagemaker/latest/dg/linear-learner.html for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c34b9b3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "container <- sagemaker$image_uris$retrieve(framework='linear-learner', region= session$boto_region_name, version='latest')\n",
    "cat('Linear Learner Container Image URL: ', container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ed66e62",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_output <- paste0('s3://', bucket, '/output_glm')\n",
    "estimator2 <- sagemaker$estimator$Estimator(image_uri = container,\n",
    "                                           role = role_arn,\n",
    "                                           train_instance_count = 1L,\n",
    "                                           train_instance_type = 'ml.m5.4xlarge',\n",
    "                                           input_mode = 'File',\n",
    "                                           output_path = s3_output,\n",
    "                                           output_kms_key = NULL,\n",
    "                                           base_job_name = NULL,\n",
    "                                           sagemaker_session = NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f92dbbdd",
   "metadata": {},
   "source": [
    "For the text/csv input type, the first column is assumed to be the label, which is the target variable for prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b2391b3",
   "metadata": {},
   "source": [
    "predictor_type is the only hyperparameter that is required to be pre-defined for tuning. The rest are optional."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f1a07f5",
   "metadata": {},
   "source": [
    "Normalization, or feature scaling, is an important preprocessing step for certain loss functions that ensures the model being trained on a dataset does not become dominated by the weight of a single feature. Decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require normalization for the inputs either.\n",
    "\n",
    "However, Generalized Linear Models require a normalization of their input. The Amazon SageMaker Linear Learner algorithm has a normalization option to assist with this preprocessing step. If normalization is turned on, the algorithm first goes over a small sample of the data to learn the mean value and standard deviation for each feature and for the label. Each of the features in the full dataset is then shifted to have mean of zero and scaled to have a unit standard deviation.\n",
    "\n",
    "To make our job easier, we do not have to go back to our previous steps to do normalization. Normalization is built in as a hyper-parameter in SageMaker Linear learner algorithm. So no need to worry about normalization for the training portions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23e49f71",
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator2$set_hyperparameters(predictor_type=\"binary_classifier\",\n",
    "                               normalize_data = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eb28cfd",
   "metadata": {},
   "source": [
    "The tunable hyperparameters for linear learner are:\n",
    "\n",
    "- wd\n",
    "- l1\n",
    "- learning_rate\n",
    "- mini_batch_size\n",
    "- use_bias\n",
    "- positive_example_weight_mult\n",
    "\n",
    "Be careful to check which parameters are integers and which parameters are continuous because that is one of the common sources of errors. Also be careful to give a proper range for hyperparameters that makes sense for your problem. Training jobs can sometimes fail if the mini-batch size is too big compared to the training data available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b79b90bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set Hyperparameter Ranges\n",
    "hyperparameter_ranges = list('wd'  = sagemaker$parameter$ContinuousParameter(0.00001,1),\n",
    "                             'l1'  = sagemaker$parameter$ContinuousParameter(0.00001,1),\n",
    "                             'learning_rate'  = sagemaker$parameter$ContinuousParameter(0.00001,1),\n",
    "                             'mini_batch_size'  = sagemaker$parameter$IntegerParameter(10L, 50L) \n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f777e12",
   "metadata": {},
   "source": [
    "The evaluation metric we will be using in our case to compare the models will be the objective loss and is based on the validation dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab03023",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a hyperparamter tuner\n",
    "objective_metric_name = 'validation:objective_loss'\n",
    "tuner2 <- sagemaker$tuner$HyperparameterTuner(estimator2,\n",
    "                                             objective_metric_name,\n",
    "                                             hyperparameter_ranges,\n",
    "                                             objective_type='Minimize',\n",
    "                                             max_jobs=4L,\n",
    "                                             max_parallel_jobs=2L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5803b03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tuning job name\n",
    "job_name <- paste('tune-linear', format(Sys.time(), '%Y%m%d-%H-%M-%S'), sep = '-')\n",
    "\n",
    "# Define the data channels for train and validation datasets\n",
    "input_data <- list('train' = s3_train_input,\n",
    "                   'validation' = s3_valid_input)\n",
    "\n",
    "# Train the tuner\n",
    "tuner2$fit(inputs = input_data, job_name = job_name, wait=TRUE, content_type='csv') # since we are using csv files as input into the model, we need to specify content type as csv."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b7bf6a1",
   "metadata": {},
   "source": [
    "### Calculate AUC for the test data on model 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "672ca373",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deploy the model into an instance of your choosing.\n",
    "model_endpoint2 <- tuner2$deploy(initial_instance_count = 1L,\n",
    "                                   instance_type = 'ml.t2.medium')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "949973bd",
   "metadata": {},
   "source": [
    "For inference, the linear learner algorithm supports the application/json, application/x-recordio-protobuf, and text/csv formats. For more information, https://docs.aws.amazon.com/sagemaker/latest/dg/LL-in-formats.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da8d7f55",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify what data formats you want the input and output of your model to look like.\n",
    "model_endpoint2$serializer <- sagemaker$serializers$CSVSerializer(content_type='text/csv')\n",
    "model_endpoint2$deserializer <- sagemaker$deserializers$JSONDeserializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aed6097",
   "metadata": {},
   "source": [
    "In Linear Learner the output inference files are in JSON or RecordIO formats. https://docs.aws.amazon.com/sagemaker/latest/dg/LL-in-formats.html  \n",
    "\n",
    "When you make predictions on new data, the contents of the response data depends on the type of model you choose within Linear Learner. For regression (predictor_type='regressor'), the score is the prediction produced by the model. For classification (predictor_type='binary_classifier' or predictor_type='multiclass_classifier'), the model returns a score and also a predicted_label. The predicted_label is the class predicted by the model and the score measures the strength of that prediction. So, for binary classification, predicted_label is 0 or 1, and score is a single floating point number that indicates how strongly the algorithm believes that the label should be 1.\n",
    "\n",
    "To interpret the score in classification problems, you have to consider the loss function used. If the loss hyperparameter value is logistic for binary classification or softmax_loss for multiclass classification, then the score can be interpreted as the probability of the corresponding class. These are the loss values used by the linear learner when the `loss` hyperparameter is set to auto as default value. But if the `loss` is set to `hinge_loss`, then the score cannot be interpreted as a probability. This is because hinge loss corresponds to a Support Vector Classifier, which does not produce probability estimates. In the current example, since our loss hyperparameter is logistic for binary classification, we can interpret it as probability of the corresponding class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd578e7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the test data for input into the model\n",
    "test_sample <- as.matrix(iris_test[-1])\n",
    "dimnames(test_sample)[[2]] <- NULL\n",
    "\n",
    "# Predict using the test data on the deployed model\n",
    "predictions_ep <- model_endpoint2$predict(test_sample)\n",
    "\n",
    "# Add the predictions to the test dataset.\n",
    "df <- data.frame(matrix(unlist(predictions_ep$predictions), nrow=length(predictions_ep$predictions), byrow=TRUE))\n",
    "df <- df %>% dplyr::rename(score = X1, predicted_label = X2)\n",
    "iris_predictions_ep2 <- dplyr::bind_cols(predicted_flower = df$predicted_label, \n",
    "                      iris_test)\n",
    "iris_predictions_ep2\n",
    "\n",
    "# Get the AUC\n",
    "auc(roc(iris_predictions_ep2$predicted_flower,iris_test$Species))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8aff600",
   "metadata": {},
   "source": [
    "## Create model 3: KNN in SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48ef375f",
   "metadata": {},
   "source": [
    "Amazon SageMaker k-nearest neighbors (k-NN) algorithm is an index-based algorithm. It uses a non-parametric method for classification or regression. For classification problems, the algorithm queries the k points that are closest to the sample point and returns the most frequently used label of their class as the predicted label. For regression problems, the algorithm queries the k closest points to the sample point and returns the average of their feature values as the predicted value.\n",
    "\n",
    "Training with the k-NN algorithm has three steps: sampling, dimension reduction, and index building. Sampling reduces the size of the initial dataset so that it fits into memory. For dimension reduction, the algorithm decreases the feature dimension of the data to reduce the footprint of the k-NN model in memory and inference latency. We provide two methods of dimension reduction methods: random projection and the fast Johnson-Lindenstrauss transform. Typically, you use dimension reduction for high-dimensional (d >1000) datasets to avoid the \u201ccurse of dimensionality\u201d that troubles the statistical analysis of data that becomes sparse as dimensionality increases. The main objective of k-NN's training is to construct the index. The index enables efficient lookups of distances between points whose values or class labels have not yet been determined and the k nearest points to use for inference.\n",
    "\n",
    "See https://docs.aws.amazon.com/sagemaker/latest/dg/k-nearest-neighbors.html for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc3befe5",
   "metadata": {},
   "outputs": [],
   "source": [
    "container <- sagemaker$image_uris$retrieve(framework='knn', region= session$boto_region_name, version='latest')\n",
    "cat('KNN Container Image URL: ', container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8952078c",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_output <- paste0('s3://', bucket, '/output_knn')\n",
    "estimator3 <- sagemaker$estimator$Estimator(image_uri = container,\n",
    "                                           role = role_arn,\n",
    "                                           train_instance_count = 1L,\n",
    "                                           train_instance_type = 'ml.m5.4xlarge',\n",
    "                                           input_mode = 'File',\n",
    "                                           output_path = s3_output,\n",
    "                                           output_kms_key = NULL,\n",
    "                                           base_job_name = NULL,\n",
    "                                           sagemaker_session = NULL)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "952cee1b",
   "metadata": {},
   "source": [
    "Hyperparameter `dimension_reduction_target` should not be set when `dimension_reduction_type` is set to its default value, which is `None`. If 'dimension_reduction_target' is set to a certain number without setting `dimension_reduction_type`, then SageMaker will ask us to remove 'dimension_reduction_target' from the specified hyperparameters and try again. In this tutorial, we are not performing dimensionality reduction, since we only have 4 features; so `dimension_reduction_type` is set to its default value of `None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37ef7aa7",
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator3$set_hyperparameters( \n",
    "                              feature_dim = 4L, \n",
    "                              sample_size = 10L, \n",
    "                              predictor_type = \"classifier\"\n",
    "                                )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "484f987c",
   "metadata": {},
   "source": [
    "Amazon SageMaker k-nearest neighbor model can be tuned  with the following hyperparameters:\n",
    "- k \n",
    "- sample_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a91bd395",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set Hyperparameter Ranges\n",
    "hyperparameter_ranges = list('k' = sagemaker$parameter$IntegerParameter(1L,10L)\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59ad0005",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a hyperparamter tuner\n",
    "objective_metric_name = 'test:accuracy'\n",
    "tuner3 <- sagemaker$tuner$HyperparameterTuner(estimator3,\n",
    "                                             objective_metric_name,\n",
    "                                             hyperparameter_ranges,\n",
    "                                             objective_type='Maximize',\n",
    "                                             max_jobs=2L,\n",
    "                                             max_parallel_jobs=2L)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6d49b48",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tuning job name\n",
    "job_name <- paste('tune-knn', format(Sys.time(), '%Y%m%d-%H-%M-%S'), sep = '-')\n",
    "\n",
    "# Define the data channels for train and validation datasets\n",
    "input_data <- list('train' = s3_train_input,\n",
    "                   'test' = s3_valid_input # KNN needs a test data, does not work without it.\n",
    "                    ) \n",
    "\n",
    "# train the tuner\n",
    "tuner3$fit(inputs = input_data, job_name = job_name, wait=TRUE, content_type='text/csv;label_size=0')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0887c62d",
   "metadata": {},
   "source": [
    "### Calculate AUC for the test data on model 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c204889d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deploy the model into an instance of your choosing.\n",
    "model_endpoint3 <- tuner3$deploy(initial_instance_count = 1L,\n",
    "                                   instance_type = 'ml.t2.medium')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4243dba",
   "metadata": {},
   "source": [
    "For inference, the linear learner algorithm supports the application/json, application/x-recordio-protobuf, and text/csv formats. For more information, https://docs.aws.amazon.com/sagemaker/latest/dg/LL-in-formats.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "167ced33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify what data formats you want the input and output of your model to look like.\n",
    "model_endpoint3$serializer <- sagemaker$serializers$CSVSerializer(content_type='text/csv')\n",
    "model_endpoint3$deserializer <- sagemaker$deserializers$JSONDeserializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ce50fd",
   "metadata": {},
   "source": [
    "In KNN, the input formats for inference are:\n",
    "- CSV\n",
    "- JSON\n",
    "- JSONLINES\n",
    "- RECORDIO\n",
    "\n",
    "The output formats for inference are:\n",
    "- JSON\n",
    "- JSONLINES\n",
    "- Verbose JSON\n",
    "- Verbose RecordIO-ProtoBuf\n",
    "\n",
    "Notice that there is no CSV output format for inference.  \n",
    "\n",
    "See https://docs.aws.amazon.com/sagemaker/latest/dg/kNN-inference-formats.html for more details.\n",
    "\n",
    "When you make predictions on new data, the contents of the response data depends on the type of model you choose within Linear Learner. For regression (predictor_type='regressor'), the score is the prediction produced by the model. For classification (predictor_type='binary_classifier' or predictor_type='multiclass_classifier'), the model returns a score and also a predicted_label. The predicted_label is the class predicted by the model and the score measures the strength of that prediction. So, for binary classification, predicted_label is 0 or 1, and score is a single floating point number that indicates how strongly the algorithm believes that the label should be 1.\n",
    "\n",
    "To interpret the score in classification problems, you have to consider the loss function used. If the loss hyperparameter value is logistic for binary classification or softmax_loss for multiclass classification, then the score can be interpreted as the probability of the corresponding class. These are the loss values used by the linear learner when the loss hyperparameter is set to auto as default value. But if the loss is set to hinge_loss, then the score cannot be interpreted as a probability. This is because hinge loss corresponds to a Support Vector Classifier, which does not produce probability estimates. In the current example, since our loss hyperparameter is logistic for binary classification, we can interpret it as probability of the corresponding class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "017e7253",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the test data for input into the model\n",
    "test_sample <- as.matrix(iris_test[-1])\n",
    "dimnames(test_sample)[[2]] <- NULL\n",
    "\n",
    "# Predict using the test data on the deployed model\n",
    "predictions_ep <- model_endpoint3$predict(test_sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66a9203b",
   "metadata": {},
   "source": [
    "We see that the output is of a deserialized JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d2380e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_ep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9c12fab",
   "metadata": {},
   "outputs": [],
   "source": [
    "typeof(predictions_ep)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a755d464",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add the predictions to the test dataset.\n",
    "df = data.frame(predicted_flower = unlist(predictions_ep$predictions))\n",
    "iris_predictions_ep2 <- dplyr::bind_cols(predicted_flower = df$predicted_flower, \n",
    "                      iris_test)\n",
    "iris_predictions_ep2\n",
    "\n",
    "# Get the AUC\n",
    "auc(roc(iris_predictions_ep2$predicted_flower,iris_test$Species))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d36155c6",
   "metadata": {},
   "source": [
    "## Compare the AUC of 3 models for the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1cca8d",
   "metadata": {},
   "source": [
    "\n",
    "- AUC of Sagemaker XGBoost = 1 \n",
    "\n",
    "- AUC of Sagemaker Linear Learner = 0.83\n",
    "\n",
    "- AUC of Sagemaker KNN = 1\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e81c277",
   "metadata": {},
   "source": [
    "Based on the AUC metric (the higher the better), both XGBoost and KNN perform equally well and are better than the Linear Learner. We can also explore the 3 models with other binary classification metrics such as accuracy, F1 score, and misclassification error. Comparing only the AUC, in this example, we could chose either the XGBoost model or the KNN model to move onto production and close the other two. The deployed model of our choosing can be passed onto production to generate predictions of flower species given that the user only has its sepal and petal measurements. The performance of the deployed model can also be tracked in Amazon CloudWatch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a9dd325",
   "metadata": {},
   "source": [
    "## Clean up "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4b7ef60",
   "metadata": {},
   "source": [
    "##### We close the endpoints which we created to free up resources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f27da27",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_endpoint1$delete_model()\n",
    "model_endpoint2$delete_model()\n",
    "model_endpoint3$delete_model()\n",
    "\n",
    "session$delete_endpoint(model_endpoint1$endpoint)\n",
    "session$delete_endpoint(model_endpoint2$endpoint)\n",
    "session$delete_endpoint(model_endpoint3$endpoint)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/r_examples|r_sagemaker_binary_classification_algorithms|R_binary_classification_algorithms_comparison.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}