sagemaker_processing/scikit_learn_data_processing_and_model_evaluation/scikit_learn_data_processing_and_model_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon SageMaker Processing jobs\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "With Amazon SageMaker Processing jobs, you can leverage a simplified, managed experience to run data pre- or post-processing and model evaluation workloads on the Amazon SageMaker platform.\n",
    "\n",
    "A processing job downloads input from Amazon Simple Storage Service (Amazon S3), then uploads outputs to Amazon S3 during or after the processing job.\n",
    "\n",
    "<img src=\"Processing-1.jpg\">\n",
    "\n",
    "This notebook shows how you can:\n",
    "\n",
    "1. Run a processing job to run a scikit-learn script that cleans, pre-processes, performs feature engineering, and splits the input data into train and test sets.\n",
    "2. Run a training job on the pre-processed training data to train a model\n",
    "3. Run a processing job on the pre-processed test data to evaluate the trained model's performance\n",
    "4. Use your own custom container to run processing jobs with your own Python libraries and dependencies.\n",
    "\n",
    "The dataset used here is the [Census-Income KDD Dataset](https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29). You select features from this dataset, clean the data, and turn the data into features that the training algorithm can use to train a binary classification model, and split the data into train and test sets. The task is to predict whether rows representing census responders have an income greater than `$50,000`, or less than `$50,000`. The dataset is heavily class imbalanced, with most records being labeled as earning less than `$50,000`. After training a logistic regression model, you evaluate the model against a hold-out test dataset, and save the classification evaluation metrics, including precision, recall, and F1 score for each label, and accuracy and ROC AUC for the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data pre-processing and feature engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the scikit-learn preprocessing script as a processing job, create a `SKLearnProcessor`, which lets you run scripts inside of processing jobs using the scikit-learn image provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --quiet --upgrade s3fs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "region = boto3.session.Session().region_name\n",
    "\n",
    "role = get_execution_role()\n",
    "sklearn_processor = SKLearnProcessor(\n",
    "    framework_version=\"0.20.0\", role=role, instance_type=\"ml.m5.xlarge\", instance_count=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before introducing the script you use for data cleaning, pre-processing, and feature engineering, inspect the first 20 rows of the dataset. The target is predicting the `income` category. The features from the dataset you select are `age`, `education`, `major industry code`, `class of worker`, `num persons worked for employer`, `capital gains`, `capital losses`, and `dividends from stocks`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "input_data = \"s3://sagemaker-sample-data-{}/processing/census/census-income.csv\".format(region)\n",
    "df = pd.read_csv(input_data, nrows=10)\n",
    "df.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook cell writes a file `preprocessing.py`, which contains the pre-processing script. You can update the script, and rerun this cell to overwrite `preprocessing.py`. You run this as a processing job in the next cell. In this script, you\n",
    "\n",
    "* Remove duplicates and rows with conflicting data\n",
    "* transform the target `income` column into a column containing two labels.\n",
    "* transform the `age` and `num persons worked for employer` numerical columns into categorical features by binning them\n",
    "* scale the continuous `capital gains`, `capital losses`, and `dividends from stocks` so they're suitable for training\n",
    "* encode the `education`, `major industry code`, `class of worker` so they're suitable for training\n",
    "* split the data into training and test datasets, and saves the training features and labels and test features and labels.\n",
    "\n",
    "Our training script will use the pre-processed training features and labels to train a model, and our model evaluation script will use the trained model and pre-processed test features and labels to evaluate the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile preprocessing.py\n",
    "\n",
    "import argparse\n",
    "import os\n",
    "import warnings\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelBinarizer, KBinsDiscretizer\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "from sklearn.exceptions import DataConversionWarning\n",
    "\n",
    "warnings.filterwarnings(action=\"ignore\", category=DataConversionWarning)\n",
    "\n",
    "\n",
    "columns = [\n",
    "    \"age\",\n",
    "    \"education\",\n",
    "    \"major industry code\",\n",
    "    \"class of worker\",\n",
    "    \"num persons worked for employer\",\n",
    "    \"capital gains\",\n",
    "    \"capital losses\",\n",
    "    \"dividends from stocks\",\n",
    "    \"income\",\n",
    "]\n",
    "class_labels = [\" - 50000.\", \" 50000+.\"]\n",
    "\n",
    "\n",
    "def print_shape(df):\n",
    "    negative_examples, positive_examples = np.bincount(df[\"income\"])\n",
    "    print(\n",
    "        \"Data shape: {}, {} positive examples, {} negative examples\".format(\n",
    "            df.shape, positive_examples, negative_examples\n",
    "        )\n",
    "    )\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    parser = argparse.ArgumentParser()\n",
    "    parser.add_argument(\"--train-test-split-ratio\", type=float, default=0.3)\n",
    "    args, _ = parser.parse_known_args()\n",
    "\n",
    "    print(\"Received arguments {}\".format(args))\n",
    "\n",
    "    input_data_path = os.path.join(\"/opt/ml/processing/input\", \"census-income.csv\")\n",
    "\n",
    "    print(\"Reading input data from {}\".format(input_data_path))\n",
    "    df = pd.read_csv(input_data_path)\n",
    "    df = pd.DataFrame(data=df, columns=columns)\n",
    "    df.dropna(inplace=True)\n",
    "    df.drop_duplicates(inplace=True)\n",
    "    df.replace(class_labels, [0, 1], inplace=True)\n",
    "\n",
    "    negative_examples, positive_examples = np.bincount(df[\"income\"])\n",
    "    print(\n",
    "        \"Data after cleaning: {}, {} positive examples, {} negative examples\".format(\n",
    "            df.shape, positive_examples, negative_examples\n",
    "        )\n",
    "    )\n",
    "\n",
    "    split_ratio = args.train_test_split_ratio\n",
    "    print(\"Splitting data into train and test sets with ratio {}\".format(split_ratio))\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        df.drop(\"income\", axis=1), df[\"income\"], test_size=split_ratio, random_state=0\n",
    "    )\n",
    "\n",
    "    preprocess = make_column_transformer(\n",
    "        (\n",
    "            [\"age\", \"num persons worked for employer\"],\n",
    "            KBinsDiscretizer(encode=\"onehot-dense\", n_bins=10),\n",
    "        ),\n",
    "        ([\"capital gains\", \"capital losses\", \"dividends from stocks\"], StandardScaler()),\n",
    "        ([\"education\", \"major industry code\", \"class of worker\"], OneHotEncoder(sparse=False)),\n",
    "    )\n",
    "    print(\"Running preprocessing and feature engineering transformations\")\n",
    "    train_features = preprocess.fit_transform(X_train)\n",
    "    test_features = preprocess.transform(X_test)\n",
    "\n",
    "    print(\"Train data shape after preprocessing: {}\".format(train_features.shape))\n",
    "    print(\"Test data shape after preprocessing: {}\".format(test_features.shape))\n",
    "\n",
    "    train_features_output_path = os.path.join(\"/opt/ml/processing/train\", \"train_features.csv\")\n",
    "    train_labels_output_path = os.path.join(\"/opt/ml/processing/train\", \"train_labels.csv\")\n",
    "\n",
    "    test_features_output_path = os.path.join(\"/opt/ml/processing/test\", \"test_features.csv\")\n",
    "    test_labels_output_path = os.path.join(\"/opt/ml/processing/test\", \"test_labels.csv\")\n",
    "\n",
    "    print(\"Saving training features to {}\".format(train_features_output_path))\n",
    "    pd.DataFrame(train_features).to_csv(train_features_output_path, header=False, index=False)\n",
    "\n",
    "    print(\"Saving test features to {}\".format(test_features_output_path))\n",
    "    pd.DataFrame(test_features).to_csv(test_features_output_path, header=False, index=False)\n",
    "\n",
    "    print(\"Saving training labels to {}\".format(train_labels_output_path))\n",
    "    y_train.to_csv(train_labels_output_path, header=False, index=False)\n",
    "\n",
    "    print(\"Saving test labels to {}\".format(test_labels_output_path))\n",
    "    y_test.to_csv(test_labels_output_path, header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run this script as a processing job. Use the `SKLearnProcessor.run()` method. You give the `run()` method one `ProcessingInput` where the `source` is the census dataset in Amazon S3, and the `destination` is where the script reads this data from, in this case `/opt/ml/processing/input`. These local paths inside the processing container must begin with `/opt/ml/processing/`.\n",
    "\n",
    "Also give the `run()` method a `ProcessingOutput`, where the `source` is the path the script writes output data to. For outputs, the `destination` defaults to an S3 bucket that the Amazon SageMaker Python SDK creates for you, following the format `s3://sagemaker-<region>-<account_id>/<processing_job_name>/output/<output_name/`. You also give the ProcessingOutputs values for `output_name`, to make it easier to retrieve these output artifacts after the job is run.\n",
    "\n",
    "The `arguments` parameter in the `run()` method are command-line arguments in our `preprocessing.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "\n",
    "sklearn_processor.run(\n",
    "    code=\"preprocessing.py\",\n",
    "    inputs=[ProcessingInput(source=input_data, destination=\"/opt/ml/processing/input\")],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train_data\", source=\"/opt/ml/processing/train\"),\n",
    "        ProcessingOutput(output_name=\"test_data\", source=\"/opt/ml/processing/test\"),\n",
    "    ],\n",
    "    arguments=[\"--train-test-split-ratio\", \"0.2\"],\n",
    ")\n",
    "\n",
    "preprocessing_job_description = sklearn_processor.jobs[-1].describe()\n",
    "\n",
    "output_config = preprocessing_job_description[\"ProcessingOutputConfig\"]\n",
    "for output in output_config[\"Outputs\"]:\n",
    "    if output[\"OutputName\"] == \"train_data\":\n",
    "        preprocessed_training_data = output[\"S3Output\"][\"S3Uri\"]\n",
    "    if output[\"OutputName\"] == \"test_data\":\n",
    "        preprocessed_test_data = output[\"S3Output\"][\"S3Uri\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now inspect the output of the pre-processing job, which consists of the processed features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_features = pd.read_csv(preprocessed_training_data + \"/train_features.csv\", nrows=10)\n",
    "print(\"Training features shape: {}\".format(training_features.shape))\n",
    "training_features.head(n=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training using the pre-processed data\n",
    "\n",
    "We create a `SKLearn` instance, which we will use to run a training job using the training script `train.py`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.sklearn.estimator import SKLearn\n",
    "\n",
    "sklearn = SKLearn(\n",
    "    entry_point=\"train.py\", framework_version=\"0.20.0\", instance_type=\"ml.m5.xlarge\", role=role\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training script `train.py` trains a logistic regression model on the training data, and saves the model to the `/opt/ml/model` directory, which Amazon SageMaker tars and uploads into a `model.tar.gz` file into S3 at the end of the training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile train.py\n",
    "\n",
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.externals import joblib\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    training_data_directory = \"/opt/ml/input/data/train\"\n",
    "    train_features_data = os.path.join(training_data_directory, \"train_features.csv\")\n",
    "    train_labels_data = os.path.join(training_data_directory, \"train_labels.csv\")\n",
    "    print(\"Reading input data\")\n",
    "    X_train = pd.read_csv(train_features_data, header=None)\n",
    "    y_train = pd.read_csv(train_labels_data, header=None)\n",
    "\n",
    "    model = LogisticRegression(class_weight=\"balanced\", solver=\"lbfgs\")\n",
    "    print(\"Training LR model\")\n",
    "    model.fit(X_train, y_train)\n",
    "    model_output_directory = os.path.join(\"/opt/ml/model\", \"model.joblib\")\n",
    "    print(\"Saving model to {}\".format(model_output_directory))\n",
    "    joblib.dump(model, model_output_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the training job using `train.py` on the preprocessed training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sklearn.fit({\"train\": preprocessed_training_data})\n",
    "training_job_description = sklearn.jobs[-1].describe()\n",
    "model_data_s3_uri = \"{}{}/{}\".format(\n",
    "    training_job_description[\"OutputDataConfig\"][\"S3OutputPath\"],\n",
    "    training_job_description[\"TrainingJobName\"],\n",
    "    \"output/model.tar.gz\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "\n",
    "`evaluation.py` is the model evaluation script. Since the script also runs using scikit-learn as a dependency,  run this using the `SKLearnProcessor` you created previously. This script takes the trained model and the test dataset as input, and produces a JSON file containing classification evaluation metrics, including precision, recall, and F1 score for each label, and accuracy and ROC AUC for the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile evaluation.py\n",
    "\n",
    "import json\n",
    "import os\n",
    "import tarfile\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.metrics import classification_report, roc_auc_score, accuracy_score\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    model_path = os.path.join(\"/opt/ml/processing/model\", \"model.tar.gz\")\n",
    "    print(\"Extracting model from path: {}\".format(model_path))\n",
    "    with tarfile.open(model_path) as tar:\n",
    "        tar.extractall(path=\".\")\n",
    "    print(\"Loading model\")\n",
    "    model = joblib.load(\"model.joblib\")\n",
    "\n",
    "    print(\"Loading test input data\")\n",
    "    test_features_data = os.path.join(\"/opt/ml/processing/test\", \"test_features.csv\")\n",
    "    test_labels_data = os.path.join(\"/opt/ml/processing/test\", \"test_labels.csv\")\n",
    "\n",
    "    X_test = pd.read_csv(test_features_data, header=None)\n",
    "    y_test = pd.read_csv(test_labels_data, header=None)\n",
    "    predictions = model.predict(X_test)\n",
    "\n",
    "    print(\"Creating classification evaluation report\")\n",
    "    report_dict = classification_report(y_test, predictions, output_dict=True)\n",
    "    report_dict[\"accuracy\"] = accuracy_score(y_test, predictions)\n",
    "    report_dict[\"roc_auc\"] = roc_auc_score(y_test, predictions)\n",
    "\n",
    "    print(\"Classification report:\\n{}\".format(report_dict))\n",
    "\n",
    "    evaluation_output_path = os.path.join(\"/opt/ml/processing/evaluation\", \"evaluation.json\")\n",
    "    print(\"Saving classification report to {}\".format(evaluation_output_path))\n",
    "\n",
    "    with open(evaluation_output_path, \"w\") as f:\n",
    "        f.write(json.dumps(report_dict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from sagemaker.s3 import S3Downloader\n",
    "\n",
    "sklearn_processor.run(\n",
    "    code=\"evaluation.py\",\n",
    "    inputs=[\n",
    "        ProcessingInput(source=model_data_s3_uri, destination=\"/opt/ml/processing/model\"),\n",
    "        ProcessingInput(source=preprocessed_test_data, destination=\"/opt/ml/processing/test\"),\n",
    "    ],\n",
    "    outputs=[ProcessingOutput(output_name=\"evaluation\", source=\"/opt/ml/processing/evaluation\")],\n",
    ")\n",
    "evaluation_job_description = sklearn_processor.jobs[-1].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now retrieve the file `evaluation.json` from Amazon S3, which contains the evaluation report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_output_config = evaluation_job_description[\"ProcessingOutputConfig\"]\n",
    "for output in evaluation_output_config[\"Outputs\"]:\n",
    "    if output[\"OutputName\"] == \"evaluation\":\n",
    "        evaluation_s3_uri = output[\"S3Output\"][\"S3Uri\"] + \"/evaluation.json\"\n",
    "        break\n",
    "\n",
    "evaluation_output = S3Downloader.read_file(evaluation_s3_uri)\n",
    "evaluation_output_dict = json.loads(evaluation_output)\n",
    "print(json.dumps(evaluation_output_dict, sort_keys=True, indent=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running processing jobs with FrameworkProcessor to include custom dependencies\n",
    "\n",
    "Above, you used a processing container that has scikit-learn installed, but there was no way to add dependencies to the processing container.\n",
    "\n",
    "A new sub-class of existing `ScriptProcessor` called `FrameworkProcessor` has been added to SageMaker SDK which allows to specify `source_dir` using which `requirements.txt` file can be included.  The processor object will first install listed dependencies inside the target container prior to triggering processing job.\n",
    "\n",
    "Below is a simple example of how to create a processing container, and how to use a `FrameworkProcessor` to run your own code within a container. Create a scikit-learn container and run a processing job using the same `preprocessing.py` script you used above. You can provide your own dependencies inside this container to run your processing script with `source_dir` and `requirements.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import FrameworkProcessor\n",
    "\n",
    "est_cls = sagemaker.sklearn.estimator.SKLearn\n",
    "framework_version_str = \"0.20.0\"\n",
    "\n",
    "script_processor = FrameworkProcessor(\n",
    "    role=role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    "    estimator_cls=est_cls,\n",
    "    framework_version=framework_version_str,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the same `preprocessing.py` script you ran above, but now, this code is running inside the container that now has additional dependencies installed.  You can update the processing script to use these dependencies and perform custom processing.  This approach can be applied to your own pre-processing, feature-engineering, and model evaluation scripts inside of this container."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's copy the previously created `preprocessing.py` script to `code` folder for next `run` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp preprocessing.py ./code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "\n",
    "script_processor.run(\n",
    "    code=\"preprocessing.py\",\n",
    "    source_dir=\"code\",\n",
    "    inputs=[ProcessingInput(source=input_data, destination=\"/opt/ml/processing/input\")],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train_data\", source=\"/opt/ml/processing/train\"),\n",
    "        ProcessingOutput(output_name=\"test_data\", source=\"/opt/ml/processing/test\"),\n",
    "    ],\n",
    "    arguments=[\"--train-test-split-ratio\", \"0.2\"],\n",
    ")\n",
    "script_processor_job_description = script_processor.jobs[-1].describe()\n",
    "print(script_processor_job_description)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "As you can see all the dependencies from `requirements.txt` file have been installed on the target container prior to starting the processing script.  \n",
    "`FrameworkProcessor` makes it easy to use existing built-in framework processors like SciKit Learn, MXNet, PyTorch etc along with specifying custom dependencies for processing script.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional)Next Steps\n",
    "\n",
    "As an optional step, this notebook shows an example below to build custom container from scratch using `docker`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default optional steps run automatically, set `run_optional_steps` to False if you don't want to execute optional steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_optional_steps = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will stop the below cells from executing if \"Run All Cells\" was used on the notebook.\n",
    "if not run_optional_steps:\n",
    "    raise SystemExit(\"Stop here. Do not automatically execute optional steps.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Running processing jobs with your own dependencies\n",
    "\n",
    "Above, you used a processing container that has scikit-learn installed, but you can run your own processing container in your processing job as well, and still provide a script to run within your processing container.\n",
    "\n",
    "Below, you walk through how to create a processing container, and how to use a `ScriptProcessor` to run your own code within a container. Create a scikit-learn container and run a processing job using the same `preprocessing.py` script you used above. You can provide your own dependencies inside this container to run your processing script with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the Dockerfile to create the processing container. Install `pandas` and `scikit-learn` into it. You can install your own dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile docker/Dockerfile\n",
    "\n",
    "FROM python:3.7-slim-buster\n",
    "\n",
    "RUN pip3 install pandas==0.25.3 scikit-learn==0.21.3\n",
    "ENV PYTHONUNBUFFERED=TRUE\n",
    "\n",
    "ENTRYPOINT [\"python3\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This block of code builds the container using the `docker` command, creates an Amazon Elastic Container Registry (Amazon ECR) repository, and pushes the image to Amazon ECR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "account_id = boto3.client(\"sts\").get_caller_identity().get(\"Account\")\n",
    "ecr_repository = \"sagemaker-processing-container\"\n",
    "tag = \":latest\"\n",
    "\n",
    "uri_suffix = \"amazonaws.com\"\n",
    "if region in [\"cn-north-1\", \"cn-northwest-1\"]:\n",
    "    uri_suffix = \"amazonaws.com.cn\"\n",
    "processing_repository_uri = \"{}.dkr.ecr.{}.{}/{}\".format(\n",
    "    account_id, region, uri_suffix, ecr_repository + tag\n",
    ")\n",
    "\n",
    "# Create ECR repository and push docker image\n",
    "!docker build -t $ecr_repository docker\n",
    "!$(aws ecr get-login --region $region --registry-ids $account_id --no-include-email)\n",
    "!aws ecr create-repository --repository-name $ecr_repository\n",
    "!docker tag {ecr_repository + tag} $processing_repository_uri\n",
    "!docker push $processing_repository_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ScriptProcessor` class lets you run a command inside this container, which you can use to run your own script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ScriptProcessor\n",
    "\n",
    "script_processor = ScriptProcessor(\n",
    "    command=[\"python3\"],\n",
    "    image_uri=processing_repository_uri,\n",
    "    role=role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.m5.xlarge\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the same `preprocessing.py` script you ran above, but now, this code is running inside of the Docker container you built in this notebook, not the scikit-learn image maintained by Amazon SageMaker. You can add the dependencies to the Docker image, and run your own pre-processing, feature-engineering, and model evaluation scripts inside of this container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_processor.run(\n",
    "    code=\"preprocessing.py\",\n",
    "    inputs=[ProcessingInput(source=input_data, destination=\"/opt/ml/processing/input\")],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"train_data\", source=\"/opt/ml/processing/train\"),\n",
    "        ProcessingOutput(output_name=\"test_data\", source=\"/opt/ml/processing/test\"),\n",
    "    ],\n",
    "    arguments=[\"--train-test-split-ratio\", \"0.2\"],\n",
    ")\n",
    "script_processor_job_description = script_processor.jobs[-1].describe()\n",
    "print(script_processor_job_description)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker_processing|scikit_learn_data_processing_and_model_evaluation|scikit_learn_data_processing_and_model_evaluation.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}