       prepare_data/sm-processing_r.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using R in SageMaker Processing\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Amazon SageMaker Processing is a capability of Amazon SageMaker that lets you easily run your preprocessing, postprocessing and model evaluation workloads on fully managed infrastructure.  In this example, we'll see how to use SageMaker Processing with the R programming language.\n",
    "\n",
    "The workflow for using R with SageMaker Processing involves the following steps:\n",
    "\n",
    "- Writing a R script.\n",
    "- Building a Docker container.\n",
    "- Creating a SageMaker Processing job.\n",
    "- Retrieving and viewing job results.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The R script\n",
    "\n",
    "To use R with SageMaker Processing, first prepare a R script similar to one you would use outside SageMaker.  Below is the R script we'll be using.  It performs operations on data and also saves a .png of a plot for retrieval and display later after the Processing job is complete.  This enables you to perform any kind of analysis and feature engineering at scale with R, and also create visualizations for display anywhere.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile preprocessing.R\n",
    "\n",
    "library(readr)\n",
    "library(dplyr)\n",
    "library(ggplot2)\n",
    "library(forcats)\n",
    "\n",
    "input_dir <- \"/opt/ml/processing/input/\"\n",
    "filename <- Sys.glob(paste(input_dir, \"*.csv\", sep=\"\"))\n",
    "df <- read_csv(filename)\n",
    "\n",
    "plot_data <- df %>%\n",
    "  group_by(state) %>%\n",
    "  count()\n",
    "\n",
    "write_csv(plot_data, \"/opt/ml/processing/csv/plot_data.csv\")\n",
    "\n",
    "plot <- plot_data %>% \n",
    "  ggplot()+\n",
    "  geom_col(aes(fct_reorder(state, n), \n",
    "               n, \n",
    "               fill = n))+\n",
    "  coord_flip()+\n",
    "  labs(\n",
    "    title = \"Number of people by state\",\n",
    "    subtitle = \"From US-500 dataset\",\n",
    "    x = \"State\",\n",
    "    y = \"Number of people\"\n",
    "  )+ \n",
    "  theme_bw()\n",
    "\n",
    "ggsave(\"/opt/ml/processing/images/census_plot.png\", width = 10, height = 8, dpi = 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Docker container\n",
    "\n",
    "Next, there is a one-time step to create a R container.  For subsequent SageMaker Processing jobs, you can just reuse this container (unless you need to add further dependencies, in which case you can just add them to the Dockerfile and rebuild).  To start, set up a local directory for Docker-related files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple Dockerfile can be used to build a Docker container for SageMaker Processing.  For this example, we'll use a parent Docker image from the Rocker Project, which provides a set of convenient R Docker images.  There is no need to include your R script in the container itself because SageMaker Processing will ingest it for you.  This gives you the flexibility to modify the script as needed without having to rebuild the Docker image every time you modify it.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile docker/Dockerfile\n",
    "\n",
    "FROM rocker/tidyverse:latest\n",
    "\n",
    "# tidyverse has all the packages we need, otherwise we could install more as follows\n",
    "# RUN install2.r --error \\\n",
    "#    jsonlite \\\n",
    "#    tseries\n",
    "\n",
    "ENTRYPOINT [\"Rscript\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Dockerfile is now used to build the Docker image.  We'll also create an Amazon Elastic Container Registry (ECR) repository, and push the image to ECR so it can be accessed by SageMaker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "\n",
    "account_id = boto3.client(\"sts\").get_caller_identity().get(\"Account\")\n",
    "region = boto3.session.Session().region_name\n",
    "\n",
    "ecr_repository = \"r-in-sagemaker-processing\"\n",
    "tag = \":latest\"\n",
    "\n",
    "uri_suffix = \"amazonaws.com\"\n",
    "processing_repository_uri = \"{}.dkr.ecr.{}.{}/{}\".format(\n",
    "    account_id, region, uri_suffix, ecr_repository + tag\n",
    ")\n",
    "\n",
    "# Create ECR repository and push Docker image\n",
    "!docker build -t $ecr_repository docker\n",
    "!$(aws ecr get-login --region $region --registry-ids $account_id --no-include-email)\n",
    "!aws ecr create-repository --repository-name $ecr_repository\n",
    "!docker tag {ecr_repository + tag} $processing_repository_uri\n",
    "!docker push $processing_repository_uri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a SageMaker Processing job\n",
    "\n",
    "With our Docker image in ECR, we now prepare for the SageMaker Processing job by specifying Amazon S3 buckets for output and input, and downloading the raw dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "session = sagemaker.Session()\n",
    "s3_output = session.default_bucket()\n",
    "s3_prefix = \"R-in-Processing\"\n",
    "default_bucket_prefix = session.default_bucket_prefix\n",
    "\n",
    "# If a default bucket prefix is specified, append it to the s3 path\n",
    "if default_bucket_prefix:\n",
    "    s3_prefix = f\"{default_bucket_prefix}/{s3_prefix}\"\n",
    "\n",
    "s3_source = \"sagemaker-workshop-pdx\"\n",
    "session.download_data(path=\"./data\", bucket=s3_source, key_prefix=\"R-in-Processing/us-500.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before setting up the SageMaker Processing job, the raw dataset is uploaded to S3 so it is accessible to SageMaker Processing.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rawdata_s3_prefix = \"{}/data/raw\".format(s3_prefix)\n",
    "raw_s3 = session.upload_data(path=\"./data\", key_prefix=rawdata_s3_prefix)\n",
    "print(raw_s3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ScriptProcessor` class of the SageMaker SDK lets you run a command inside a Docker container.  We'll use this to run our own script using the `Rscript` command.  In the `ScriptProcessor` you also can specify the type and number of instances to be used in the SageMaker Processing job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ScriptProcessor\n",
    "\n",
    "script_processor = ScriptProcessor(\n",
    "    command=[\"Rscript\"],\n",
    "    image_uri=processing_repository_uri,\n",
    "    role=role,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.c5.xlarge\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now start the SageMaker Processing job.  The main aspects of the code below are specifying the input and output locations, and the name of our R preprocessing script.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from time import gmtime, strftime\n",
    "\n",
    "processing_job_name = \"R-in-Processing-{}\".format(strftime(\"%d-%H-%M-%S\", gmtime()))\n",
    "output_destination = \"s3://{}/{}/data\".format(s3_output, s3_prefix)\n",
    "\n",
    "script_processor.run(\n",
    "    code=\"preprocessing.R\",\n",
    "    job_name=processing_job_name,\n",
    "    inputs=[ProcessingInput(source=raw_s3, destination=\"/opt/ml/processing/input\")],\n",
    "    outputs=[\n",
    "        ProcessingOutput(\n",
    "            output_name=\"csv\",\n",
    "            destination=\"{}/csv\".format(output_destination),\n",
    "            source=\"/opt/ml/processing/csv\",\n",
    "        ),\n",
    "        ProcessingOutput(\n",
    "            output_name=\"images\",\n",
    "            destination=\"{}/images\".format(output_destination),\n",
    "            source=\"/opt/ml/processing/images\",\n",
    "        ),\n",
    "    ],\n",
    ")\n",
    "\n",
    "preprocessing_job_description = script_processor.jobs[-1].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving and viewing job results\n",
    "\n",
    "From the SageMaker Processing job description, we can look up the S3 URIs of the output, including the output plot .png file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_config = preprocessing_job_description[\"ProcessingOutputConfig\"]\n",
    "for output in output_config[\"Outputs\"]:\n",
    "    if output[\"OutputName\"] == \"csv\":\n",
    "        preprocessed_csv_data = output[\"S3Output\"][\"S3Uri\"]\n",
    "    if output[\"OutputName\"] == \"images\":\n",
    "        preprocessed_images = output[\"S3Output\"][\"S3Uri\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can display the plot produced by the SageMaker Processing job.  A similar workflow applies to retrieving and working with any other output from a job, such as the transformed data itself.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "from IPython.display import display\n",
    "\n",
    "plot_key = \"census_plot.png\"\n",
    "plot_in_s3 = \"{}/{}\".format(preprocessed_images, plot_key)\n",
    "!aws s3 cp {plot_in_s3} .\n",
    "im = Image.open(plot_key)\n",
    "display(im)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/prepare_data|sm-processing_r.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/prepare_data|sm-processing_r.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}