       prepare_data/sm-processing_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a8f18b23",
   "metadata": {
    "papermill": {
     "duration": 0.006395,
     "end_time": "2022-04-18T00:08:55.010149",
     "exception": false,
     "start_time": "2022-04-18T00:08:55.003754",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "# Get started with SageMaker Processing\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8f18b23",
   "metadata": {
    "papermill": {
     "duration": 0.006395,
     "end_time": "2022-04-18T00:08:55.010149",
     "exception": false,
     "start_time": "2022-04-18T00:08:55.003754",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "\n",
    "This notebook corresponds to the section \"Preprocessing Data With The Built-In Scikit-Learn Container\" in the blog post [Amazon SageMaker Processing – Fully Managed Data Processing and Model Evaluation](https://aws.amazon.com/blogs/aws/amazon-sagemaker-processing-fully-managed-data-processing-and-model-evaluation/). \n",
    "It shows a lightweight example of using SageMaker Processing to create train, test, and validation datasets. SageMaker Processing is used to create these datasets, which then are written back to S3.\n",
    "\n",
    "## Runtime\n",
    "\n",
    "This notebook takes approximately 5 minutes to run.\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Prepare resources](#Prepare-resources)\n",
    "1. [Download data](#Download-data)\n",
    "1. [Prepare Processing script](#Prepare-Processing-script)\n",
    "1. [Run Processing job](#Run-Processing-job)\n",
    "1. [Conclusion](#Conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf7028a",
   "metadata": {
    "papermill": {
     "duration": 0.006333,
     "end_time": "2022-04-18T00:08:55.022942",
     "exception": false,
     "start_time": "2022-04-18T00:08:55.016609",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Prepare resources\n",
    "\n",
    "First, let’s create an SKLearnProcessor object, passing the scikit-learn version we want to use, as well as our managed infrastructure requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "862f8d1f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-18T00:08:55.039310Z",
     "iopub.status.busy": "2022-04-18T00:08:55.038857Z",
     "iopub.status.idle": "2022-04-18T00:08:56.057474Z",
     "shell.execute_reply": "2022-04-18T00:08:56.057892Z"
    },
    "papermill": {
     "duration": 1.028712,
     "end_time": "2022-04-18T00:08:56.058050",
     "exception": false,
     "start_time": "2022-04-18T00:08:55.029338",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.sklearn.processing import SKLearnProcessor\n",
    "\n",
    "region = sagemaker.Session().boto_region_name\n",
    "role = get_execution_role()\n",
    "sklearn_processor = SKLearnProcessor(\n",
    "    framework_version=\"1.2-1\", role=role, instance_type=\"ml.m5.xlarge\", instance_count=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b35ea4ea",
   "metadata": {
    "papermill": {
     "duration": 0.006588,
     "end_time": "2022-04-18T00:08:56.071404",
     "exception": false,
     "start_time": "2022-04-18T00:08:56.064816",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Download data\n",
    "\n",
    "Read in the raw data from a public S3 bucket. This example uses the [Census-Income (KDD) Dataset](https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29) from the UCI Machine Learning Repository.\n",
    "\n",
    "> Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6eaf6050",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-18T00:08:56.096003Z",
     "iopub.status.busy": "2022-04-18T00:08:56.095500Z",
     "iopub.status.idle": "2022-04-18T00:09:00.816015Z",
     "shell.execute_reply": "2022-04-18T00:09:00.815586Z"
    },
    "papermill": {
     "duration": 4.738175,
     "end_time": "2022-04-18T00:09:00.816126",
     "exception": false,
     "start_time": "2022-04-18T00:08:56.077951",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>class of worker</th>\n",
       "      <th>detailed industry recode</th>\n",
       "      <th>detailed occupation recode</th>\n",
       "      <th>education</th>\n",
       "      <th>wage per hour</th>\n",
       "      <th>enroll in edu inst last wk</th>\n",
       "      <th>marital stat</th>\n",
       "      <th>major industry code</th>\n",
       "      <th>major occupation code</th>\n",
       "      <th>...</th>\n",
       "      <th>country of birth father</th>\n",
       "      <th>country of birth mother</th>\n",
       "      <th>country of birth self</th>\n",
       "      <th>citizenship</th>\n",
       "      <th>own business or self employed</th>\n",
       "      <th>fill inc questionnaire for veteran's admin</th>\n",
       "      <th>veterans benefits</th>\n",
       "      <th>weeks worked in year</th>\n",
       "      <th>year</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>73</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>High school graduate</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>Widowed</td>\n",
       "      <td>Not in universe or children</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>Native- Born in the United States</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>95</td>\n",
       "      <td>- 50000.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>58</td>\n",
       "      <td>Self-employed-not incorporated</td>\n",
       "      <td>4</td>\n",
       "      <td>34</td>\n",
       "      <td>Some college but no degree</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Construction</td>\n",
       "      <td>Precision production craft &amp; repair</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>Native- Born in the United States</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>2</td>\n",
       "      <td>52</td>\n",
       "      <td>94</td>\n",
       "      <td>- 50000.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>18</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10th grade</td>\n",
       "      <td>0</td>\n",
       "      <td>High school</td>\n",
       "      <td>Never married</td>\n",
       "      <td>Not in universe or children</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>...</td>\n",
       "      <td>Vietnam</td>\n",
       "      <td>Vietnam</td>\n",
       "      <td>Vietnam</td>\n",
       "      <td>Foreign born- Not a citizen of U S</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>95</td>\n",
       "      <td>- 50000.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Children</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>Never married</td>\n",
       "      <td>Not in universe or children</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>Native- Born in the United States</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>94</td>\n",
       "      <td>- 50000.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Children</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>Never married</td>\n",
       "      <td>Not in universe or children</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>United-States</td>\n",
       "      <td>Native- Born in the United States</td>\n",
       "      <td>0</td>\n",
       "      <td>Not in universe</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>94</td>\n",
       "      <td>- 50000.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 42 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   age                  class of worker  detailed industry recode  \\\n",
       "0   73                  Not in universe                         0   \n",
       "1   58   Self-employed-not incorporated                         4   \n",
       "2   18                  Not in universe                         0   \n",
       "3    9                  Not in universe                         0   \n",
       "4   10                  Not in universe                         0   \n",
       "\n",
       "   detailed occupation recode                    education  wage per hour  \\\n",
       "0                           0         High school graduate              0   \n",
       "1                          34   Some college but no degree              0   \n",
       "2                           0                   10th grade              0   \n",
       "3                           0                     Children              0   \n",
       "4                           0                     Children              0   \n",
       "\n",
       "  enroll in edu inst last wk    marital stat           major industry code  \\\n",
       "0            Not in universe         Widowed   Not in universe or children   \n",
       "1            Not in universe        Divorced                  Construction   \n",
       "2                High school   Never married   Not in universe or children   \n",
       "3            Not in universe   Never married   Not in universe or children   \n",
       "4            Not in universe   Never married   Not in universe or children   \n",
       "\n",
       "                  major occupation code  ... country of birth father  \\\n",
       "0                       Not in universe  ...           United-States   \n",
       "1   Precision production craft & repair  ...           United-States   \n",
       "2                       Not in universe  ...                 Vietnam   \n",
       "3                       Not in universe  ...           United-States   \n",
       "4                       Not in universe  ...           United-States   \n",
       "\n",
       "  country of birth mother country of birth self  \\\n",
       "0           United-States         United-States   \n",
       "1           United-States         United-States   \n",
       "2                 Vietnam               Vietnam   \n",
       "3           United-States         United-States   \n",
       "4           United-States         United-States   \n",
       "\n",
       "                            citizenship own business or self employed  \\\n",
       "0     Native- Born in the United States                             0   \n",
       "1     Native- Born in the United States                             0   \n",
       "2   Foreign born- Not a citizen of U S                              0   \n",
       "3     Native- Born in the United States                             0   \n",
       "4     Native- Born in the United States                             0   \n",
       "\n",
       "  fill inc questionnaire for veteran's admin  veterans benefits  \\\n",
       "0                            Not in universe                  2   \n",
       "1                            Not in universe                  2   \n",
       "2                            Not in universe                  2   \n",
       "3                            Not in universe                  0   \n",
       "4                            Not in universe                  0   \n",
       "\n",
       "   weeks worked in year  year     income  \n",
       "0                     0    95   - 50000.  \n",
       "1                    52    94   - 50000.  \n",
       "2                     0    95   - 50000.  \n",
       "3                     0    94   - 50000.  \n",
       "4                     0    94   - 50000.  \n",
       "\n",
       "[5 rows x 42 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "s3 = boto3.client(\"s3\")\n",
    "s3.download_file(\n",
    "    \"sagemaker-sample-data-{}\".format(region),\n",
    "    \"processing/census/census-income.csv\",\n",
    "    \"census-income.csv\",\n",
    ")\n",
    "df = pd.read_csv(\"census-income.csv\")\n",
    "df.to_csv(\"dataset.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "240ac1a3",
   "metadata": {
    "papermill": {
     "duration": 0.007158,
     "end_time": "2022-04-18T00:09:00.830803",
     "exception": false,
     "start_time": "2022-04-18T00:09:00.823645",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Prepare Processing script\n",
    "\n",
    "Write the Python script that will be run by SageMaker Processing. This script reads the single data file from S3; splits the rows into train, test, and validation sets; and then writes the three output files to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "64b8e90c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-18T00:09:00.849237Z",
     "iopub.status.busy": "2022-04-18T00:09:00.848512Z",
     "iopub.status.idle": "2022-04-18T00:09:00.851175Z",
     "shell.execute_reply": "2022-04-18T00:09:00.851554Z"
    },
    "papermill": {
     "duration": 0.013691,
     "end_time": "2022-04-18T00:09:00.851669",
     "exception": false,
     "start_time": "2022-04-18T00:09:00.837978",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing preprocessing.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile preprocessing.py\n",
    "import pandas as pd\n",
    "import os\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "input_data_path = os.path.join(\"/opt/ml/processing/input\", \"dataset.csv\")\n",
    "df = pd.read_csv(input_data_path)\n",
    "print(\"Shape of data is:\", df.shape)\n",
    "train, test = train_test_split(df, test_size=0.2)\n",
    "train, validation = train_test_split(train, test_size=0.2)\n",
    "\n",
    "try:\n",
    "    os.makedirs(\"/opt/ml/processing/output/train\")\n",
    "    os.makedirs(\"/opt/ml/processing/output/validation\")\n",
    "    os.makedirs(\"/opt/ml/processing/output/test\")\n",
    "    print(\"Successfully created directories\")\n",
    "except Exception as e:\n",
    "    # if the Processing call already creates these directories (or directory otherwise cannot be created)\n",
    "    print(e)\n",
    "    print(\"Could not make directories\")\n",
    "    pass\n",
    "\n",
    "try:\n",
    "    train.to_csv(\"/opt/ml/processing/output/train/train.csv\")\n",
    "    validation.to_csv(\"/opt/ml/processing/output/validation/validation.csv\")\n",
    "    test.to_csv(\"/opt/ml/processing/output/test/test.csv\")\n",
    "    print(\"Wrote files successfully\")\n",
    "except Exception as e:\n",
    "    print(\"Failed to write the files\")\n",
    "    print(e)\n",
    "    pass\n",
    "\n",
    "print(\"Completed running the processing job\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bab3ff2",
   "metadata": {
    "papermill": {
     "duration": 0.007373,
     "end_time": "2022-04-18T00:09:00.866414",
     "exception": false,
     "start_time": "2022-04-18T00:09:00.859041",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Run Processing job"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68190117",
   "metadata": {
    "papermill": {
     "duration": 0.007318,
     "end_time": "2022-04-18T00:09:00.881109",
     "exception": false,
     "start_time": "2022-04-18T00:09:00.873791",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "Run the Processing job, specifying the script name, input file, and output files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "450368db",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-18T00:09:00.901375Z",
     "iopub.status.busy": "2022-04-18T00:09:00.900644Z",
     "iopub.status.idle": "2022-04-18T00:13:44.601792Z",
     "shell.execute_reply": "2022-04-18T00:13:44.602212Z"
    },
    "papermill": {
     "duration": 283.713812,
     "end_time": "2022-04-18T00:13:44.602351",
     "exception": false,
     "start_time": "2022-04-18T00:09:00.888539",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%capture output\n",
    "\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "\n",
    "sklearn_processor.run(\n",
    "    code=\"preprocessing.py\",\n",
    "    # arguments = [\"arg1\", \"arg2\"], # Arguments can optionally be specified here\n",
    "    inputs=[ProcessingInput(source=\"dataset.csv\", destination=\"/opt/ml/processing/input\")],\n",
    "    outputs=[\n",
    "        ProcessingOutput(source=\"/opt/ml/processing/output/train\"),\n",
    "        ProcessingOutput(source=\"/opt/ml/processing/output/validation\"),\n",
    "        ProcessingOutput(source=\"/opt/ml/processing/output/test\"),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135c2776",
   "metadata": {
    "papermill": {
     "duration": 0.007543,
     "end_time": "2022-04-18T00:13:44.617780",
     "exception": false,
     "start_time": "2022-04-18T00:13:44.610237",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "Get the Processing job logs and retrieve the job name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8f3e9edf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-18T00:13:44.636660Z",
     "iopub.status.busy": "2022-04-18T00:13:44.636183Z",
     "iopub.status.idle": "2022-04-18T00:13:44.638287Z",
     "shell.execute_reply": "2022-04-18T00:13:44.638643Z"
    },
    "papermill": {
     "duration": 0.013467,
     "end_time": "2022-04-18T00:13:44.638753",
     "exception": false,
     "start_time": "2022-04-18T00:13:44.625286",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Job Name:  sagemaker-scikit-learn-2022-04-18-00-09-00-899\n",
      "Inputs:  [{'InputName': 'input-1', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/input/input-1/dataset.csv', 'LocalPath': '/opt/ml/processing/input', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}, {'InputName': 'code', 'AppManaged': False, 'S3Input': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/input/code/preprocessing.py', 'LocalPath': '/opt/ml/processing/input/code', 'S3DataType': 'S3Prefix', 'S3InputMode': 'File', 'S3DataDistributionType': 'FullyReplicated', 'S3CompressionType': 'None'}}]\n",
      "Outputs:  [{'OutputName': 'output-1', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/output/output-1', 'LocalPath': '/opt/ml/processing/output/train', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'output-2', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/output/output-2', 'LocalPath': '/opt/ml/processing/output/validation', 'S3UploadMode': 'EndOfJob'}}, {'OutputName': 'output-3', 'AppManaged': False, 'S3Output': {'S3Uri': 's3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/output/output-3', 'LocalPath': '/opt/ml/processing/output/test', 'S3UploadMode': 'EndOfJob'}}]\n",
      "...........................\n",
      "\u001b[34mShape of data is: (199523, 43)\u001b[0m\n",
      "\u001b[34m[Errno 17] File exists: '/opt/ml/processing/output/train'\u001b[0m\n",
      "\u001b[34mCould not make directories\u001b[0m\n",
      "\u001b[34mWrote files successfully\u001b[0m\n",
      "\u001b[34mCompleted running the processing job\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(output)\n",
    "job_name = str(output).split(\"\\n\")[1].split(\" \")[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "386f656a",
   "metadata": {
    "papermill": {
     "duration": 0.007802,
     "end_time": "2022-04-18T00:13:44.654395",
     "exception": false,
     "start_time": "2022-04-18T00:13:44.646593",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "Confirm that the output dataset files were written to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9b885d73",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-18T00:13:44.677694Z",
     "iopub.status.busy": "2022-04-18T00:13:44.677253Z",
     "iopub.status.idle": "2022-04-18T00:13:45.043939Z",
     "shell.execute_reply": "2022-04-18T00:13:45.044361Z"
    },
    "papermill": {
     "duration": 0.382286,
     "end_time": "2022-04-18T00:13:45.044503",
     "exception": false,
     "start_time": "2022-04-18T00:13:44.662217",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%%\n"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/output/output-1/train.csv\n",
      "s3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/output/output-2/validation.csv\n",
      "s3://sagemaker-us-west-2-000000000000/sagemaker-scikit-learn-2022-04-18-00-09-00-899/output/output-3/test.csv\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "default_bucket = sagemaker.Session().default_bucket()\n",
    "default_bucket_prefix = sagemaker.Session().default_bucket_prefix\n",
    "default_bucket_prefix_path = \"\"\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    default_bucket_prefix_path = f\"{default_bucket_prefix}/\"\n",
    "\n",
    "for i in range(1, 4):\n",
    "    prefix = s3_client.list_objects(\n",
    "        Bucket=default_bucket, Prefix=f\"{default_bucket_prefix_path}sagemaker-scikit-learn\"\n",
    "    )[\"Contents\"][-i][\"Key\"]\n",
    "    print(\"s3://\" + default_bucket + \"/\" + prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd191e62",
   "metadata": {
    "papermill": {
     "duration": 0.008184,
     "end_time": "2022-04-18T00:13:45.060991",
     "exception": false,
     "start_time": "2022-04-18T00:13:45.052807",
     "status": "completed"
    },
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, we read a dataset from S3 and processed it into train, test, and validation sets using a SageMaker Processing job. You can extend this example for preprocessing your own datasets in preparation for machine learning or other applications."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/prepare_data|sm-processing_introduction.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/prepare_data|sm-processing_introduction.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 291.085709,
   "end_time": "2022-04-18T00:13:45.485219",
   "environment_variables": {},
   "exception": null,
   "input_path": "basic_sagemaker_processing.ipynb",
   "output_path": "/opt/ml/processing/output/basic_sagemaker_processing-2022-04-18-00-04-13.ipynb",
   "parameters": {
    "kms_key": "arn:aws:kms:us-west-2:000000000000:1234abcd-12ab-34cd-56ef-1234567890ab"
   },
   "start_time": "2022-04-18T00:08:54.399510",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}