lab/00_Warmup/01_Basic model with a Builtin Algorithm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training/Optimizing a basic model with a Built Algorithm\n",
    "\n",
    "This exercise is about manually execute all the steps of the Machine Learning development pipeline. We'll use here a public dataset called iris. The dataset and the model aren't the focus of this exercise. The idea here is to see how Sagemaker can accelerate your work and void wasting your time with tasks that aren't related to your business. So, we'll do the following:\n",
    "\n",
    " - Train/deploy/test a multiclass model using XGBoost\n",
    " - Optimize the model\n",
    " - Run batch predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART 1 - Train deploy and test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's  start by importing the dataset and visualize it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import datasets\n",
    "sns.set(color_codes=True)\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "X=iris.data\n",
    "y=iris.target\n",
    "\n",
    "dataset = np.insert(iris.data, 0, iris.target,axis=1)\n",
    "\n",
    "df = pd.DataFrame(data=dataset, columns=['iris_id'] + iris.feature_names)\n",
    "df['species'] = df['iris_id'].map(lambda x: 'setosa' if x == 0 else 'versicolor' if x == 1 else 'virginica')\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking the class distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = df.groupby(df['species'])['species'].count().plot(kind='bar')\n",
    "x_offset = -0.05\n",
    "y_offset = 0\n",
    "for p in ax.patches:\n",
    "    b = p.get_bbox()\n",
    "    val = \"{}\".format(int(b.y1 + b.y0))\n",
    "    ax.annotate(val, ((b.x0 + b.x1)/2 + x_offset, b.y1 + y_offset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = df.corr()\n",
    "\n",
    "f, ax = plt.subplots(figsize=(15, 8))\n",
    "sns.heatmap(corr, annot=True, fmt=\"f\",\n",
    "            xticklabels=corr.columns.values,\n",
    "            yticklabels=corr.columns.values,\n",
    "            ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pairplots & histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df.drop(['iris_id'], axis=1), hue='species', size=2.5,diag_kind=\"kde\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now with linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(df.drop(['iris_id'], axis=1), kind=\"reg\", hue='species', size=2.5,diag_kind=\"kde\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fit and plot a univariate or bivariate kernel density estimate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_df = df[(df.iris_id==0.0)]\n",
    "sns.kdeplot(tmp_df['petal width (cm)'], tmp_df['petal length (cm)'], bw='silverman', cmap=\"Reds\", shade=False, shade_lowest=False)\n",
    "\n",
    "tmp_df = df[(df.iris_id==2.0)]\n",
    "sns.kdeplot(tmp_df['petal width (cm)'], tmp_df['petal length (cm)'], bw='silverman', cmap=\"Blues\", shade=False, shade_lowest=False)\n",
    "\n",
    "tmp_df = df[(df.iris_id==1.0)]\n",
    "sns.kdeplot(tmp_df['petal width (cm)'], tmp_df['petal length (cm)'], bw='silverman', cmap=\"Greens\", shade=False, shade_lowest=False)\n",
    "\n",
    "plt.xlabel('species')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Petal length and petal width have the highest linear correlation with our label. Also, sepal width seems to be useless, considering the linear correlation with our label.\n",
    "\n",
    "Since versicolor and virginica cannot be split linearly, we need a more versatile algorithm to create a better classifier. In this case, we'll use XGBoost, a tree ensable that can give us a good model for predicting the flower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ok, now let's split the dataset into training and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42, stratify=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('iris_train.csv', 'w') as csv:\n",
    "    for x_,y_ in zip(X_train, y_train):\n",
    "        line = \"%s,%s\" % (y_, \",\".join( list(map(str, x_)) ) )\n",
    "        csv.write( line + \"\\n\" )\n",
    "    csv.flush()\n",
    "    csv.close()\n",
    "\n",
    "with open('iris_test.csv', 'w') as csv:\n",
    "    for x_,y_ in zip(X_test, y_test):\n",
    "        line = \"%s,%s\" % (y_, \",\".join( list(map(str, x_)) ) )\n",
    "        csv.write( line + \"\\n\" )\n",
    "    csv.flush()\n",
    "    csv.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now it's time to train our model with the builtin algorithm XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "\n",
    "from sagemaker import get_execution_role\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "prefix='mlops/iris'\n",
    "# Retrieve the default bucket\n",
    "sagemaker_session = sagemaker.Session()\n",
    "bucket = sagemaker_session.default_bucket()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the dataset to an S3 bucket\n",
    "input_train = sagemaker_session.upload_data(path='iris_train.csv', key_prefix='%s/data' % prefix)\n",
    "input_test = sagemaker_session.upload_data(path='iris_test.csv', key_prefix='%s/data' % prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = sagemaker.session.s3_input(s3_data=input_train,content_type=\"csv\")\n",
    "test_data = sagemaker.session.s3_input(s3_data=input_test,content_type=\"csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "containers = {'us-west-2': '433757028032.dkr.ecr.us-west-2.amazonaws.com/xgboost:latest',\n",
    "              'us-east-1': '811284229777.dkr.ecr.us-east-1.amazonaws.com/xgboost:latest',\n",
    "              'us-east-2': '825641698319.dkr.ecr.us-east-2.amazonaws.com/xgboost:latest',\n",
    "              'eu-west-1': '685385470294.dkr.ecr.eu-west-1.amazonaws.com/xgboost:latest'}\n",
    "\n",
    "# Create the estimator\n",
    "xgb = sagemaker.estimator.Estimator(containers[boto3.Session().region_name],\n",
    "                                    role, \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.m4.xlarge',\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                    sagemaker_session=sagemaker_session)\n",
    "# Set the hyperparameters\n",
    "xgb.set_hyperparameters(eta=0.1,\n",
    "                        max_depth=10,\n",
    "                        gamma=4,\n",
    "                        reg_lambda=10,\n",
    "                        num_class=len(np.unique(y)),\n",
    "                        alpha=10,\n",
    "                        min_child_weight=6,\n",
    "                        silent=0,\n",
    "                        objective='multi:softmax',\n",
    "                        num_round=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# takes around 3min 11s\n",
    "xgb.fit({'train': train_data, 'validation': test_data, })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the model and create an endpoint for it\n",
    "The following action will:\n",
    " * get the assets from the job we just ran and then create an input in the Models Catalog\n",
    " * create a endpoint configuration (a metadata for our final endpoint)\n",
    " * create an enpoint, which is our model wrapped in a format of a WebService\n",
    " \n",
    "After that we'll be able to call our deployed endpoint for doing predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "xgb_predictor = xgb.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_name = xgb_predictor.endpoint\n",
    "model_name = boto3.client('sagemaker').describe_endpoint_config(\n",
    "    EndpointConfigName=endpoint_name\n",
    ")['ProductionVariants'][0]['ModelName']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now, let's do a basic test with the deployed endpoint\n",
    "In this test, we'll use a helper object called predictor. This object is always returned from a **Deploy** call. The predictor is just for testing purposes and we'll not use it inside our real application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "xgb_predictor.content_type = 'text/csv'\n",
    "xgb_predictor.serializer = csv_serializer\n",
    "xgb_predictor.deserializer = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_test = [ float(xgb_predictor.predict(x).decode('utf-8')) for x in X_test] \n",
    "score = f1_score(y_test,predictions_test,labels=[0.0,1.0,2.0],average='micro')\n",
    "\n",
    "print('F1 Score(micro): %.1f' % (score * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Then, let's  test the API for our trained model\n",
    "This is how your application will call the endpoint. Using boto3 for getting a sagemaker runtime client and then we'll call invoke_endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sm = boto3.client('sagemaker-runtime')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "\n",
    "resp = sm.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    ContentType='text/csv',\n",
    "    Body=csv_serializer(X_test[0])\n",
    ")\n",
    "prediction = float(resp['Body'].read().decode('utf-8'))\n",
    "print('Predicted class: %.1f for [%s]' % (prediction, csv_serializer(X_test[0])) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART 2 - Model optimization with Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning Jobs\n",
    "#### A.K.A. Hyperparameter Optimization\n",
    "\n",
    "## Let's tune our model before using it for our batch prediction\n",
    "We know that the iris dataset is an easy challenge. We can achieve a better score with XGBoost. However, we don't want to wast time testing all the possible variations of the hyperparameters in order to optimize the training process.\n",
    "\n",
    "Instead, we'll use the Sagemaker's tuning feature. For that, we'll use the same estimator, but let's create a Tuner and ask it for optimize the model for us. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner\n",
    "\n",
    "hyperparameter_ranges = {'eta': ContinuousParameter(0, 1),\n",
    "                        'min_child_weight': ContinuousParameter(1, 10),\n",
    "                        'alpha': ContinuousParameter(0, 2),\n",
    "                         'gamma': ContinuousParameter(0, 10),\n",
    "                        'max_depth': IntegerParameter(1, 10)}\n",
    "\n",
    "objective_metric_name = 'validation:merror'\n",
    "\n",
    "tuner = HyperparameterTuner(xgb,\n",
    "                            objective_metric_name,\n",
    "                            hyperparameter_ranges,\n",
    "                            max_jobs=20,\n",
    "                            max_parallel_jobs=4,\n",
    "                            objective_type='Minimize')\n",
    "\n",
    "tuner.fit({'train': train_data, 'validation': test_data, })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_name = tuner.latest_tuning_job.name\n",
    "attached_tuner = HyperparameterTuner.attach(job_name)\n",
    "xgb_predictor2 = attached_tuner.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_endpoint_name = endpoint_name\n",
    "endpoint_name = xgb_predictor2.endpoint\n",
    "model_name = boto3.client('sagemaker').describe_endpoint_config(\n",
    "    EndpointConfigName=endpoint_name\n",
    ")['ProductionVariants'][0]['ModelName']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A simple test before we move on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "xgb_predictor2.content_type = 'text/csv'\n",
    "xgb_predictor2.serializer = csv_serializer\n",
    "xgb_predictor2.deserializer = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_test = [ float(xgb_predictor2.predict(x).decode('utf-8')) for x in X_test] \n",
    "score = f1_score(y_test,predictions_test,labels=[0.0,1.0,2.0],average='micro')\n",
    "\n",
    "print('F1 Score(micro): %.1f' % (score * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PART 3 - Batch Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch transform job\n",
    "If you have a file with the samples you want to predict, just upload that file to an S3 bucket and start a Batch Transform job. For this task, you don't need to deploy an endpoint. Sagemaker will create all the resources needed to do this batch prediction, save the results into an S3 bucket and then it will destroy the resources automatically for you"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_dataset_filename='batch_dataset.csv'\n",
    "with open(batch_dataset_filename, 'w') as csv:\n",
    "    for x_ in X:\n",
    "        line = \",\".join( list(map(str, x_)) )\n",
    "        csv.write( line + \"\\n\" )\n",
    "    csv.flush()\n",
    "    csv.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_batch = sagemaker_session.upload_data(path=batch_dataset_filename, key_prefix='%s/data' % prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "# Initialize the transformer object\n",
    "transformer=sagemaker.transformer.Transformer(\n",
    "    base_transform_job_name='mlops-iris',\n",
    "    model_name=model_name,\n",
    "    instance_count=1,\n",
    "    instance_type='ml.c4.xlarge',\n",
    "    output_path='s3://{}/{}/batch_output'.format(bucket, prefix),\n",
    ")\n",
    "# To start a transform job:\n",
    "transformer.transform(input_batch, content_type='text/csv', split_type='Line')\n",
    "# Then wait until transform job is completed\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "predictions_filename='iris_predictions.csv'\n",
    "s3 = boto3.client('s3')\n",
    "s3.download_file(bucket, '{}/batch_output/{}.out'.format(prefix, batch_dataset_filename), predictions_filename)\n",
    "\n",
    "df2 = pd.read_csv(predictions_filename, sep=',', encoding='utf-8',header=None, names=[ 'predicted_iris_id'])\n",
    "df3 = df.copy()\n",
    "df3['predicted_iris_id'] = df2['predicted_iris_id']\n",
    "df3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "score = f1_score(df3['iris_id'], df3['predicted_iris_id'],labels=[0.0,1.0,2.0],average='micro')\n",
    "\n",
    "print('F1 Score(micro): %.1f' % (score * 100.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "cnf_matrix = confusion_matrix(df3['iris_id'], df3['predicted_iris_id'])\n",
    "\n",
    "f, ax = plt.subplots(figsize=(15, 8))\n",
    "sns.heatmap(cnf_matrix, annot=True, fmt=\"f\", mask=np.zeros_like(cnf_matrix, dtype=np.bool), \n",
    "            cmap=sns.diverging_palette(220, 10, as_cmap=True),\n",
    "            square=True, ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor.delete_endpoint()\n",
    "xgb_predictor2.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The end"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}