SageMaker Integration Work (#5)

* Created json template for IAM policy and updated script to reflect changes. * Created json template for IAM policy and updated script to reflect changes. * temp place between setting up the new notebook and finishing the markdown docs * renamed notebook * Completed SageMaker integration work. * fixed a few gotchas * Updated based on PR feedback, removed name variables and collapsed notebooks. * missed a name change * updated for API changes
aws-samples · Jan 8, 2019 · ca46a30 · ca46a30
1 parent 1e1fec9
commit ca46a30
Show file tree

Hide file tree

Showing 5 changed files with 178 additions and 20 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,3 @@
 .ipynb_checkpoints
 .DS_Store
+.idea/*
diff --git a/IAM_Policy.json b/IAM_Policy.json
@@ -0,0 +1,28 @@
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "VisualEditor0",
+            "Effect": "Allow",
+            "Action": [
+                "forecast:*",
+                "s3:*",
+                "iam:GetRole",
+                "iam:GetUser",
+                "iam:ListRoles",
+                "iam:ListUsers",
+                "iam:ListGroups",
+                "iam:CreateRole",
+                "iam:PassRole",
+                "iam:PutRolePolicy",
+                "iam:PutUserPolicy",
+                "iam:UpdateAssumeRolePolicy",
+                "kms:List*",
+                "kms:Get*",
+                "kms:Describe*",
+                "kms:Create*"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
diff --git a/SageMaker_Tutorial.md b/SageMaker_Tutorial.md
@@ -0,0 +1,66 @@
+# Getting Started with Amazon Forecast with Amazon SageMaker.
+
+This guide will walk through the creation of a new SageMaker Notebook Instance, the configuration of IAM policies, an S3 bucket
+and your first project with Amazon Forecast. The Notebook Instance can then be used again for additional exploratory work with 
+Amazon Forecast.
+
+## Creating Your Notebook Instance
+
+First you will need to create a new Notebook Instance, to do that begin by logging into the AWS Console.
+
+Next, ensure you are in the US-West-2 Region, do do that look in the top left corner, if it says `Oregon` next to support
+that is correct, otherwise select `Oregon` from the drop-down.
+
+Under `Find services` in the main body of the page, enter `SageMaker`, then select it from the drop-down.
+
+To the left, will see a category titled `Notebook` inside that, click `Notebook instances`.
+
+Click the orange `Create notebook instance` button.
+
+Give the instance a name unique in the account you are using. If a shared account, place your name first like `FirstNameLastNameForecastDemo`. The default Instance
+type is fine. 
+
+The Next component to change is the IAM role. Under the drop-down click `Create a new role`. Then for S3, select `Any S3 Bucket`, finally click `Create role`.
+Note that the role itself has become a link. Open that link in a new tab.
+
+Here you will update the policies of your instance to allow it to work with Forecast. Click the `Attach policies` button.
+
+Next click the `Create policy` button at the top. In the new page, click the `JSON` tab.
+
+Erase all of the content that is in the editor and paste the content in [IAM_Policy.json](IAM_Policy.json).
+
+After pasting, click the `Review policy` button. Give the policy again a personalize name like `FirstNameLastNameForecastIAMPolicy`.
+
+For the description, enter in something about it being used to demo Forecast. Finally click `Create policy`. Close this tab or window.
+
+Once closed you should see the tab for adding permissions to your SageMaker role. Click the `Filter Policies` link, then select
+`Customer managed`. After that, you should see the policy you just created, if the list is long, just paste the name in the search bar to reduce the number
+of items. If you do not see it still, click the refresh icon in the top right of the page.
+
+After clicking the checkbox next to the policy, click `Attach policy` at the bottom of the page. Then close this window.
+
+Back at the SageMaker Notebook Instance creation page, now click `Create notebook instance` at the bottom of the page. This process will take 5-10 minutes to complete. Once the status says `InService` you are ready to continue to the 
+next session.
+
+## Getting Started with Amazon Forecast.
+
+To begin, click `Open Jupyter`, this will take you to the default interface for the Notebook Instance.
+
+Click `New` then click `Terminal`, this will open a BASH shell for you on this instance. 
+
+Enter the following commands to clone this repository onto this instance:
+
+```
+cd SageMaker
+git clone https://github.com/aws-samples/amazon-forecast-samples.git
+```
+
+After that close your Terminal tab and go back to the main Notebook interface.
+
+A new folder titled `amazon-forecast-samples` should be visible, click it, click notebooks, then click `Getting_started_with_Forecast-SageMaker.ipynb` this will open the notebook.
+
+If prompted for a kernel, select `conda_python3`.
+
+From here you will follow the instructions outlined in the notebook. 
+
+**Read Every Cell FULLY Before Executing It**
diff --git a/notebooks/Getting_started_with_Forecast.ipynb b/notebooks/Getting_started_with_Forecast.ipynb
@@ -16,7 +16,9 @@
     "## Table Of Contents\n",
     "* [Setting up](#setup)\n",
     "* [Test Setup - Running first API](#hello)\n",
-    "* [Forecasting Example with Amazon Forecast](#forecastingExample)"
+    "* [Forecasting Example with Amazon Forecast](#forecastingExample)\n",
+    "\n",
+    "**Read Every Cell FULLY before executing it**\n"
    ]
   },
   {
@@ -43,14 +45,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Prerequisites : 1 time install only\n",
-    "# !pip install boto3\n",
-    "# !pip install pandas"
+    "# Prerequisites : 1 time install only, remove the comments to execute the lines.\n",
+    "#!pip install boto3\n",
+    "#!pip install pandas"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 35,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -148,7 +150,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now upload the data to S3."
+    "Now upload the data to S3. But before doing that, go into your AWS Console, select S3 for the service and create a new bucket inside the `Oregon` or `us-west-2` region. Use that bucket name convention of `amazon-forecast-unique-value-data`. The name must be unique, if you get an error, just adjust until your name works, then update the `bucketName` cell below."
    ]
   },
   {
@@ -175,7 +177,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "bucketName = 'amazon-forecast-%s-data'%accountId # use your own bucket\n",
+    "bucketName = 'amazon-forecast-chrisking-data'# Update the unique-value bit here.\n",
     "key=\"elec_data/item-demand-time.csv\""
    ]
   },
@@ -199,7 +201,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 36,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -230,7 +232,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "More details about `Domain` and dataset type can be found on the [documentation](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html) . For this example, we are using [CUSTOM](https://docs.aws.amazon.com/forecast/latest/dg/custom-domain.html) domain with 3 required attributes `timestamp`, `target_value` and `item_id`. "
+    "More details about `Domain` and dataset type can be found on the [documentation](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html) . For this example, we are using [CUSTOM](https://docs.aws.amazon.com/forecast/latest/dg/custom-domain.html) domain with 3 required attributes `timestamp`, `target_value` and `item_id`. Also for your project name, update it to reflect your name in a lowercase format."
    ]
   },
   {
@@ -249,7 +251,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "project = 'workshop'\n",
+    "project = 'workshop_forecastdemo' # Replace this with a unique name here, make sure the entire name is < 30 characters.\n",
     "datasetName= project+'_ds'\n",
     "datasetGroupName= project +'_gp'\n",
     "s3DataPath = \"s3://\"+bucketName+\"/\"+key"
@@ -364,7 +366,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Check the status of dataset, when the status change from **CREATING** to **ACTIVE**, we can continue to next steps. Depending on the data size. It can take 10 mins to be **ACTIVE**."
+    "Check the status of dataset, when the status change from **CREATING** to **ACTIVE**, we can continue to next steps. Depending on the data size. It can take 10 mins to be **ACTIVE**. This process will take 5 to 10 minutes."
    ]
   },
   {
@@ -519,7 +521,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "forecastquery.get_accuracy_metrics(PredictorName= predictorName)"
+    "forecastquery.get_accuracy_metrics(PredictorName=predictorName)"
    ]
   },
   {
@@ -544,7 +546,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "deployedPredictorsResponse=forecast.list_deployed_predictors()"
+    "deployedPredictorsResponse=forecast.list_deployed_predictors()\n",
+    "print(deployedPredictorsResponse)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Please note that the following cell can also take 10 minutes or more to be fully operational. There's no output here, but that is fine as long as the * is there."
    ]
   },
   {
@@ -555,10 +565,12 @@
    "source": [
     "while True:\n",
     "    deployedPredictorStatus = forecast.describe_deployed_predictor(PredictorName=predictorName)['Status']\n",
+    "    print(deployedPredictorStatus)\n",
     "    if deployedPredictorStatus != 'ACTIVE' and deployedPredictorStatus != 'FAILED':\n",
     "        sleep(30)\n",
     "    else:\n",
-    "        break"
+    "        break\n",
+    "print(deployedPredictorStatus)"
    ]
   },
   {
@@ -585,7 +597,8 @@
     "    PredictorName=predictorName,\n",
     "    Interval=\"hour\",\n",
     "    Filters={\"item_id\":\"client_12\"}\n",
-    ")"
+    ")\n",
+    "print(forecastResponse)"
    ]
   },
   {
@@ -599,7 +612,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can batch export forecast to s3 bucket. To do so an role with s3 put access is needed. "
+    "You can batch export forecast to s3 bucket. To do so an role with s3 put access is needed, but this has already been created."
    ]
   },
   {
@@ -669,13 +682,62 @@
    "source": [
     "s3.list_objects(Bucket=bucketName,Prefix=\"output\")"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Cleanup\n",
+    "\n",
+    "While Forecast is in preview there are no charges for using it, but to future proof this work below are the instructions to cleanup your work space."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Delete Deployed Predictor \n",
+    "forecast.delete_deployed_predictor(PredictorName=predictorName)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Delete the Predictor: \n",
+    "forecast.delete_predictor(PredictorName=predictorName)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Delete Import\n",
+    "forecast.delete_dataset_import(DatasetName=datasetName)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Delete Dataset Group\n",
+    "forecast.delete_dataset_group(DatasetGroupName=datasetGroupName)"
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "conda_python3",
    "language": "python",
-   "name": "python3"
+   "name": "conda_python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -687,7 +749,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.0"
+   "version": "3.6.5"
   }
  },
  "nbformat": 4,

diff --git a/setup_forecast_permissions.py b/setup_forecast_permissions.py
@@ -12,14 +12,15 @@
             "Sid": "VisualEditor0",
             "Effect": "Allow",
             "Action": [
-                "forecast:*"
+                "forecast:*",
                 "s3:*",
                 "iam:GetRole",
                 "iam:GetUser",
                 "iam:ListRoles",
                 "iam:ListUsers",
                 "iam:ListGroups",
                 "iam:CreateRole",
+                "iam:PassRole",
                 "iam:PutRolePolicy",
                 "iam:PutUserPolicy",
                 "iam:UpdateAssumeRolePolicy",