modeling-with-amazon-sagemaker-experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Streamline Modeling With Amazon SageMaker Studio and Amazon Experiments SDK\n",
    "\n",
    "Modeling phase is a highly iterative process in Machine Learning projects, where Data Scientists experiment with various data pre-processing and feature engineering strategies, intertwined with different model architectures, which are then trained with disparate sets of hyperparameter values. This highly iterative process, with many moving parts, can, over time, manifest into a tremendous headache in terms of keeping track of all design decisions applied in each iteration and how the training and evaluation metrics of each iteration compare to the previous versions of the model.\n",
    "\n",
    "This notebook walks you through an end-to-end example of how [Amazon SageMaker Studio](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-studio.html) can effectively leverage [Amazon SageMaker Experiments](https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html) to organize, track, visualize and compare our iterative work during development of a Keras model, trained to predict the age of an abalone, a sea snail, based on a set of features that describe it. While this example is specific to Keras, the same approach can be extended to other Machine Learning frameworks and algorithms. \n",
    "\n",
    "Amazon SageMaker Studio and Amazon SageMaker Experiments were unveiled at the [AWS Re:invent](https://aws.amazon.com/new/reinvent/), at the end of 2019:\n",
    "* [SageMaker Studio Announcement](https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-amazon-sagemaker-studio-the-first-integrated-development-environment-ide-for-machine-learning/?trk=ls_card)\n",
    "* [SageMaker Experiments Announcement](https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-amazon-sagemaker-experiments-organize-track-and-compare-your-machine-learning-training-experiments-on-amazon-sagemaker/?trk=ls_card)\n",
    "\n",
    "In this walkthrough, we will explore how Amazon SageMaker Studio, and the [Experiments SDK](https://sagemaker-experiments.readthedocs.io/en/latest/), which has been [open-sourced](https://github.com/aws/sagemaker-experiments), can be utilized to experiment with a Keras model and track data preprocessing required to prepare data for the model's consumption.\n",
    "\n",
    "Now, before we dive into hands-on exercise, let's first take a step back and discuss the building blocks of each Experiment and their referential relationships.\n",
    "* **Experiment** - a Machine Learning problem that we want to solve. Each experiment consists of a collection of Trials. Note that the name of an experiment must be unique in a given region of a particular AWS account.\n",
    "* **Trial** - an execution of a data-science workflow related to an experiment. Each Trial consists of several Trial Components. Note that the name of a trial must be unique in a given region of a particular AWS account.\n",
    "* **Trial Component** - a stage in a given trial. For instance, as we will see in our example, we will create one Trial Component for data pre-preprocessing stage and one Trial Component for model training. Similarly, in other use cases, we can also have a Trial Component for data post-processing. Unlike Experiments and Trials, Trial Components do not have to be uniquely named as they tend to represent the typical and very common stages in an ML pipeline.\n",
    "* **Tracker** - a mechanism that records various metadata about a particular Trial Component, including any Parameters, Inputs, Outputs, Artifacts and Metrics. When creating a Tracker, each Tracker is linked to a particular Training Component.\n",
    "\n",
    "![](img/experiment_structure_t.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "Now that we've set a rock-solid foundation on the key building blocks of Experiments SDK, let's dive into the fun hands-on component of this exercise. Before we jump into data pre-processing and feature engineering, let's first set up our environment and make sure that we have all the packages imported and variables defined.\n",
    "\n",
    "First, select `Python 3 (TensorFlow 2 CPU Optimized)` kernel for your notebook in your Studio environment. The smallest instance with 2 virtual CPUs and 4 GiB of memory will be more than sufficient for our exercise.\n",
    "\n",
    "Let's begin by upgrading the `sagemaker-experiments` package, which will enable us to work with Experiments SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade sagemaker-experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sagemaker as sm\n",
    "import boto3\n",
    "from urllib.request import urlretrieve\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "from sklearn import preprocessing as pp\n",
    "import pickle\n",
    "from sagemaker.tensorflow import TensorFlow\n",
    "from smexperiments.experiment import Experiment\n",
    "from smexperiments.trial import Trial\n",
    "from smexperiments.trial_component import TrialComponent\n",
    "from smexperiments.tracker import Tracker\n",
    "\n",
    "role = sm.get_execution_role()\n",
    "sm_bucket = sm.session.Session().default_bucket()\n",
    "project_path = 'experiments-keras-abalone'\n",
    "source_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/'\n",
    "processed_data_path = os.path.join('s3://',sm_bucket,project_path,'data/processed/')\n",
    "artifacts_path = os.path.join(project_path,'artifacts')\n",
    "output_path = os.path.join('s3://',sm_bucket,project_path,'output/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Pre-processing and Feature Engineering\n",
    "\n",
    "Excellent! Now, let's dive into data pre-processing and feature engineering. First, let's pull our dataset, and its metadata, from [the archive of UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/) and familiarize with the features by reviewing the portion of the metadata file that describes them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urlretrieve(source_url+'abalone.data', 'abalone.data')\n",
    "urlretrieve(source_url+'abalone.names', 'abalone.names')\n",
    "! cat abalone.names | grep 'Attribute information:' -A 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we understand the features, let's see how many examples we have and print a few examples to see the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col = ['Sex', 'Length', 'Diameter', 'Height', 'Whole weight', 'Shucked weight', 'Viscera weight', 'Shell weight','Rings']\n",
    "df = pd.read_csv('abalone.data',header=None, names=col)\n",
    "print('The dataset contains {} examples with {} features and the label in the last column.\\n'.format(df.shape[0],df.shape[1]))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's move on to data transformations and feature engineering, starting with nominal features. We will apply one-hot encoding with a dropout to the nominal feature and drop the original one. We will then rename the encoded columns to ensure that their names clearly illustrate the data they contain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = [df.columns[0]]\n",
    "encoders = {}\n",
    "\n",
    "for feature in cat_features:\n",
    "    oe = pp.OneHotEncoder(drop='first')\n",
    "    oe.fit(df[[feature]])\n",
    "    encoded_feature = pd.DataFrame(oe.transform(df[[feature]]).toarray())\n",
    "    df = encoded_feature.join(df)\n",
    "    encoders[feature] = oe\n",
    "\n",
    "df.drop(columns=cat_features,inplace=True)\n",
    "df.rename(columns={0:'gender_oh1', 1:'gender_oh2'},inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will tackle continuous numerical features and apply MinMaxScaler on them to confine all values between 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_features = df.columns[2:-1]\n",
    "scalers = {}\n",
    "\n",
    "for feature in num_features:\n",
    "    mms = pp.MinMaxScaler()\n",
    "    mms.fit(df[[feature]])\n",
    "    scaled_feature = pd.DataFrame(mms.transform(df[[feature]]))\n",
    "    df[feature] = scaled_feature\n",
    "    scalers[feature] = mms\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now proceed with splitting the pre-processed dataset for training and testing, where we will retain 80% of the dataset for training and use the remaining examples as a hold-out validation set. We will also further split the labels from the features since our Keras model, which we will review shortly, expects features and labels to be fed separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the dataset into train, test and validation\n",
    "train_perc = 80\n",
    "train_cnt = int(np.floor(df.shape[0]*train_perc/100))\n",
    "\n",
    "train_df = df[:train_cnt]\n",
    "test_df = df[train_cnt:]\n",
    "\n",
    "train_features = train_df.drop(columns=['Rings'])\n",
    "train_labels = train_df['Rings']\n",
    "test_features = test_df.drop(columns=['Rings'])\n",
    "test_labels = test_df['Rings']\n",
    "\n",
    "print('Train record count: {}\\nTest record count: {}'.format(train_df.shape[0], test_df.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's proceed with pushing the split datasets to the processed path in S3, from where the model can import them during the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_features.to_csv('abalone_train_features.csv')\n",
    "train_labels.to_csv('abalone_train_labels.csv')\n",
    "test_features.to_csv('abalone_test_features.csv')\n",
    "test_labels.to_csv('abalone_test_labels.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp . $processed_data_path --exclude \"*\" --include \"abalone_*.csv\" --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also combine our fitted encoders and scalers into a dictionary and serialize it as a pickle file. As we will see shortly, we will attach these fitted processors to our Trial Component that tracks data pre-processing stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('preprocessors.pickle','wb') as obj:\n",
    "    pickle.dump({'encoders' : encoders, 'scalers': scalers}, obj)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's utilize the Experiments SDK to create a Pre-processing Trial Component and keep track of what've accomplished so far. We start by creating an experiment. Remember that each experiment name must be unique within a given region of a particular AWS account.\n",
    "\n",
    "Once we have our experiment instantiated, we will proceed with creating a Tracker to capture various details about the data pre-processing we completed so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm = boto3.client('sagemaker')\n",
    "ts = datetime.now().strftime('%Y-%m-%d-%H-%M-%S-%f')\n",
    "\n",
    "abalone_experiment = Experiment.create(\n",
    "    experiment_name = 'predict-abalone-age-' + ts,\n",
    "    description = 'Predicting the age of an abalone based on a set of features describing it',\n",
    "    sagemaker_boto_client=sm)\n",
    "\n",
    "\n",
    "with Tracker.create(display_name='Pre-processing', sagemaker_boto_client=sm, artifact_bucket=sm_bucket, artifact_prefix=artifacts_path) as tracker:\n",
    "    tracker.log_parameters({\n",
    "        'train_test_split': 0.8\n",
    "    })\n",
    "    tracker.log_input(name='raw data', media_type='s3/uri', value=source_url)\n",
    "    tracker.log_output(name='preprocessed data', media_type='s3/uri', value=processed_data_path)\n",
    "    tracker.log_artifact(name='preprocessors', media_type='s3/uri', file_path='preprocessors.pickle')\n",
    "    \n",
    "processing_component = tracker.trial_component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic! We now have our experiment ready and we've already done our due diligence to capture our data pre-processing approach as a Trial Component. Next, let's dive into the modeling phase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling with SageMaker Experiments\n",
    "\n",
    "So far, we've done solid amount of work to preprocess data for our model. Before we proceed with training the model, let's take a moment to familiarize with it. We observe that our model has two fully connected hidden layers with a variable number of neurons and variable activation functions. This flexibility will enable us to easily pass these values as arguments to a training job and quickly parallelize our experimentation with several model architectures.\n",
    "\n",
    "We have Mean Squared Logarithmic Error defined as the loss function and the model will be using Adam optimization algorithm. Finally, the model will track Mean Squared Logarithmic Error as our metric, which will automatically propagate into our Training Trial Component in our Experiment, as we will see shortly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Review the architecture of the model\n",
    "!pygmentize 'entrypoint.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to define a few sets of hyperparameters that we want to experiment with and kick off our training jobs to run in parallel. Note that we can also easily capture custom metrics during training by configuring a regex pattern to match the desired values in the logs generated by the training job. In our case, for demonstration purposes, we will capture the number of samples (`num_train_samples`) that the Keras framework reports as being available for training the model.\n",
    "\n",
    "Note that, in our example, we do not make any changes to our pre-processing strategy between different trials, so we will attach the same Pre-processing Trial Component that we create earlier, to each of the Trials we create in the next step.\n",
    "\n",
    "**Note:** If you elect to use your own execution role instead of the one defined in this notebook, please make sure that the access policy attached to your custom execution role, that you pass to your training job, allows `sagemaker:BatchGetMetrics` and `sagemaker:BatchPutMetrics` actions, in order for us to be able to capture non-custom metrics during training and visualize them in Studio. The four training jobs that we will kick off next will take only about 5 minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define sets of hyperparameters, create a trial for each and kick of training jobs\n",
    "hyperparameters_groups=[{\n",
    "                         'learning_rate': 0.001,\n",
    "                         'epochs': 12,\n",
    "                         'batch_size': 128,\n",
    "                         'l1_size': 20,\n",
    "                         'l1_activation': 'tanh',\n",
    "                         'l2_size': 20,\n",
    "                         'l2_activation': 'relu'\n",
    "                        },\n",
    "                        {\n",
    "                         'learning_rate': 0.001,\n",
    "                         'epochs': 12,\n",
    "                         'batch_size': 128,\n",
    "                         'l1_size': 10,\n",
    "                         'l1_activation': 'tanh',\n",
    "                         'l2_size': 10,\n",
    "                         'l2_activation': 'relu'\n",
    "                        },\n",
    "                        {\n",
    "                         'learning_rate': 0.001,\n",
    "                         'epochs': 12,\n",
    "                         'batch_size': 128,\n",
    "                         'l1_size': 20,\n",
    "                         'l1_activation': 'relu',\n",
    "                         'l2_size': 10,\n",
    "                         'l2_activation': 'relu'\n",
    "                        },\n",
    "                        {\n",
    "                         'learning_rate': 0.001,\n",
    "                         'epochs': 12,\n",
    "                         'batch_size': 128,\n",
    "                         'l1_size': 10,\n",
    "                         'l1_activation': 'relu',\n",
    "                         'l1_size': 5,\n",
    "                         'l1_activation': 'relu'\n",
    "                        }]\n",
    "\n",
    "\n",
    "for i,hp_set in enumerate(hyperparameters_groups):\n",
    "\n",
    "    ts = datetime.now().strftime('%Y-%m-%d-%H-%M-%S-%f')\n",
    "    abalone_trial = abalone_experiment.create_trial(trial_name='abalone-trial-' + str(i) + '-'+ ts)\n",
    "    abalone_trial.add_trial_component(processing_component)\n",
    "    \n",
    "    abalone_estimator = TensorFlow(entry_point='entrypoint.py',\n",
    "                                   role=role,\n",
    "                                   framework_version='2.1.0',\n",
    "                                   py_version='py3',\n",
    "                                   hyperparameters=hp_set,\n",
    "                                   train_instance_count=1,\n",
    "                                   train_instance_type='ml.m5.large',\n",
    "                                   output_path = output_path,\n",
    "                                   metric_definitions=[\n",
    "                                       {'Name': 'num_train_samples', 'Regex': 'Train on (\\d+) samples'}\n",
    "                                   ]\n",
    "                                   )\n",
    "\n",
    "    job_name = 'abalone-train-' + str(i) + '-'+ ts\n",
    "    \n",
    "    abalone_estimator.fit(processed_data_path,\n",
    "                          job_name=job_name,\n",
    "                          wait=False,\n",
    "                          experiment_config={\n",
    "                                            'ExperimentName': abalone_experiment.experiment_name,\n",
    "                                            'TrialName': abalone_trial.trial_name,\n",
    "                                            'TrialComponentDisplayName': 'Training',\n",
    "                                            })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploration\n",
    "\n",
    "Upon completion of the training jobs, we can, in just a few seconds, start visualizing how different variations of the model compare in terms of the metrics collected during model training. For instance, in just a few clicks, we can visualize how the loss has been decreasing by epoch for each variation of the model and very quickly observe the model architecture that is most effective in decreasing the loss.\n",
    "\n",
    "![](img/visualize_loss.png)\n",
    "\n",
    "You can create the same plot by following the five simple steps listed below:\n",
    "1. Select the *SageMaker Experiments List* icon on the left sidebar.\n",
    "2. Double-click on your experiment to open it and use Shift key on your keyboard to select all four trials.\n",
    "3. Right-click on any of the highlighted trials and select *Open in trial component list*.\n",
    "4. Use Shift key on your keyboard to select the four Trial Components representing the Training jobs and click on *Add chart* button.\n",
    "5. Click on *New chart* and customize it to plot the collected metrics that you would like to analyze.\n",
    "\n",
    "![](img/visualization_steps_line.gif)\n",
    "\n",
    "Similarly, we can generate a scatter plot that helps us determine whether there is a relationship between the size of the first hidden layer in the network and the Mean Squared Logarithmic Error.\n",
    "\n",
    "![](img/scatter_msle.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! How quick and effortless was that?! I encourage you to further explore plotting various other metrics on your own.\n",
    "\n",
    "Next, let's select our best performing trial (`abalone-trial-0`) and double-click on it. As expected, we see two Trial Components. One captures our data Pre-processing work, and the other reflects our Model Training.\n",
    "\n",
    "When we open the Training Trial Component, we can see that it captures all the hyper parameters, the input to the model, where the trained model was placed in S3, etc. Similarly, when we open the *Preprocessing* component, we see that it captures where the source data came from, where the processed data was stored in S3, as well as, where we can easily find our trained encoder and scalers, which we've packaged into the `preprocessors.pickle` artifact.\n",
    "\n",
    "![](img/review_artifacts.gif)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup\n",
    "\n",
    "What a fun exploration this has been! Let's now clean up after ourselves by hierarchically deleting all elements of the experiment that we created in this notebook. The function call in the last cell has been commented out to ensure that you do not accidentally delete your experiment before getting a chance to explore it, in case if you execute the entire notebook at once (*Run All Cells* option). To delete your experiment, uncomment the last cell and execute it.\n",
    "\n",
    "I hope that you enjoyed diving into the intricacies of Experiments SDK and exploring how SageMaker Studio smoothly integrates with it, enabling you to lose yourself in experimentation with your Machine Learning model without losing any track of the hard work you've done!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# abalone_experiment.delete_all('--force')"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (TensorFlow 2 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/tensorflow-2.1-cpu-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}