notebooks/basic/Tutorial/1.Importing_Your_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Data Ready\n",
    "\n",
    "Forecasting is used in a variety of applications and business use cases: For example, retailers need to forecast the sales of their products to decide how much stock they need by location, Manufacturers need to estimate the number of parts required at their factories to optimize their supply chain, Businesses need to estimate their flexible workforce needs, Utilities need to forecast electricity consumption needs in order to attain an efficient energy network, and enterprises need to estimate their cloud infrastructure needs.\n",
    "\n",
    "<img src=\"https://amazon-forecast-samples.s3-us-west-2.amazonaws.com/common/images/forecast_overview_steps.png\" width=\"98%\">\n",
    "\n",
    "In this notebook we will be walking through the first steps outlined in left-box above.\n",
    "\n",
    "\n",
    "## Table Of Contents\n",
    "* Step 1: [Setup Amazon Forecast](#setup)\n",
    "* Step 2: [Prepare the Datasets](#DataPrep)\n",
    "* Step 3: [Create the Dataset Group and Dataset](#DataSet)\n",
    "* Step 4: [Create the Target Time Series Data Import Job](#DataImport)\n",
    "* [Next Steps](#nextSteps)\n",
    "\n",
    "For more informations about APIs, please check the [documentation](https://docs.aws.amazon.com/forecast/latest/dg/what-is-forecast.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Setup Amazon Forecast<a class=\"anchor\" id=\"setup\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section sets up the permissions and relevant endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install boto3 --upgrade",
    "!pip install pandas s3fs matplotlib ipywidgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "# importing forecast notebook utility from notebooks/common directory\n",
    "sys.path.insert( 0, os.path.abspath(\"../../common\") )\n",
    "import util\n",
    "\n",
    "%reload_ext autoreload\n",
    "import boto3\n",
    "import s3fs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure the S3 bucket name and region name for this lesson.\n",
    "\n",
    "- If you don't have an S3 bucket, create it first on S3. \n",
    "- Although we have set the region to us-west-2 as a default value below, you can choose any of the regions that the service is available in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region = 'us-west-2'\n",
    "bucket_name = 'forecast-demo-uci-electricity'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect API session\n",
    "session = boto3.Session(region_name=region) \n",
    "forecast = session.client(service_name='forecast') \n",
    "forecastquery = session.client(service_name='forecastquery')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Create IAM Role for Forecast</b> <br>\n",
    "Like many AWS services, Forecast will need to assume an IAM role in order to interact with your S3 resources securely. In the sample notebooks, we use the get_or_create_iam_role() utility function to create an IAM role. Please refer to \"notebooks/common/util/fcst_utils.py\" for implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the role to provide to Amazon Forecast.\n",
    "role_name = \"ForecastNotebookRole-Basic\"\n",
    "print(f\"Creating Role {role_name} ...\")\n",
    "role_arn = util.get_or_create_iam_role( role_name = role_name )\n",
    "\n",
    "# echo user inputs without account\n",
    "print(f\"Success! Created role arn = {role_arn.split('/')[1]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last part of the setup process is to validate that your account can communicate with Amazon Forecast, the cell below does just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check you can communicate with Forecast API session\n",
    "forecast.list_predictors()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Prepare the Datasets<a class=\"anchor\" id=\"DataPrep\"></a>\n",
    "\n",
    "For this exercise, we use the individual household electric power consumption dataset. (Dua, D. and Karra Taniskidou, E. (2017). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.) We aggregate the usage data hourly. \n",
    "\n",
    "To begin, use Pandas to read the CSV and to show a sample of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"../../common/data/item-demand-time.csv\", dtype = object, names=['timestamp','value','item'])\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice in the output above there are 3 columns of data:\n",
    "\n",
    "1. The Timestamp\n",
    "1. A Value\n",
    "1. An Item ID\n",
    "\n",
    "These are the 3 key required pieces of information to generate a forecast with Amazon Forecast. More can be added but these 3 must always remain present.\n",
    "\n",
    "The dataset happens to span January 01, 2014 to Deceber 31, 2014. We are only going to use January to October to train Amazon Forecast.\n",
    "\n",
    "You may notice a variable named `df` this is a popular convention when using Pandas if you are using the library's dataframe object, it is similar to a table in a database. You can learn more here: https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select January to October for one dataframe.\n",
    "jan_to_oct = df[(df['timestamp'] >= '2014-01-01') & (df['timestamp'] < '2014-11-01')]\n",
    "\n",
    "print(f\"min timestamp = {jan_to_oct.timestamp.min()}\")\n",
    "print(f\"max timestamp = {jan_to_oct.timestamp.max()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save an item_id for querying later\n",
    "item_id = \"client_12\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now export them to CSV files and place them into your `data` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jan_to_oct.to_csv(\"data/item-demand-time-train.csv\", header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now export a second dataset to CSV this time including November 1st. This extra day will be used to validate our forecast. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "validation = df[(df['timestamp'] >= '2014-01-01') & (df['timestamp'] < '2014-11-02')]\n",
    "validation.to_csv(\"data/item-demand-time-validation.csv\", header=False, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this time the data is ready to be sent to S3 where Forecast will use it later. The following cells will upload the data to S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "key=\"elec_data/item-demand-time-train.csv\"\n",
    "\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(key).upload_file(\"data/item-demand-time-train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Create the Dataset Group and Dataset <a class=\"anchor\" id=\"DataSet\"></a>\n",
    "\n",
    "In Amazon Forecast , a dataset is a collection of file(s) which contain data that is relevant for a forecasting task. A dataset must conform to a schema provided by Amazon Forecast. Since data files are imported headerless, it is important to define a schema for your data.\n",
    "\n",
    "More details about `Domain` and dataset type can be found on the [documentation](https://docs.aws.amazon.com/forecast/latest/dg/howitworks-domains-ds-types.html) . For this example, we are using [CUSTOM](https://docs.aws.amazon.com/forecast/latest/dg/custom-domain.html) domain with 3 required attributes `timestamp`, `target_value` and `item_id`.\n",
    "\n",
    "\n",
    "Next, you need to make some choices. \n",
    "<ol>\n",
    "    <li><b>How many time units do you want to forecast?</b>. For example, if your time unit is Hour, then if you want to forecast out 1 week, that would be 24*7 = 168 hours, so answer = 168. </li>\n",
    "    <li><b>What is the time granularity for your data?</b>. For example, if your time unit is Hour, answer = \"H\". </li>\n",
    "    <li><b>Think of a name you want to give this project (Dataset Group name)</b>, so all files will have the same names.  You should also use this same name for your Forecast DatasetGroup name, to set yourself up for reproducibility. </li>\n",
    "    </ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# what is your forecast horizon in number time units you've selected?\n",
    "# e.g. if you're forecasting in months, how many months out do you want a forecast?\n",
    "FORECAST_LENGTH = 24\n",
    "\n",
    "# What is your forecast time unit granularity?\n",
    "# Choices are: ^Y|M|W|D|H|30min|15min|10min|5min|1min$ \n",
    "DATASET_FREQUENCY = \"H\"\n",
    "TIMESTAMP_FORMAT = \"yyyy-MM-dd hh:mm:ss\"\n",
    "\n",
    "# What name do you want to give this project?  \n",
    "# We will use this same name for your Forecast Dataset Group name.\n",
    "PROJECT = 'util_power_demo'\n",
    "DATA_VERSION = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Dataset Group\n",
    "\n",
    "In this task, we define a container name or Dataset Group name, which will be used to keep track of Dataset import files, schema, and all Forecast results which go together.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dataset_group = f\"{PROJECT}_{DATA_VERSION}\"\n",
    "print(f\"Dataset Group Name = {dataset_group}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_arns = []\n",
    "create_dataset_group_response = \\\n",
    "    forecast.create_dataset_group(Domain=\"CUSTOM\",\n",
    "                                  DatasetGroupName=dataset_group,\n",
    "                                  DatasetArns=dataset_arns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_group_arn = create_dataset_group_response['DatasetGroupArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_group(DatasetGroupArn=dataset_group_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify the schema of your dataset here. Make sure the order of columns matches the raw data files.\n",
    "ts_schema ={\n",
    "   \"Attributes\":[\n",
    "      {\n",
    "         \"AttributeName\":\"timestamp\",\n",
    "         \"AttributeType\":\"timestamp\"\n",
    "      },\n",
    "      {\n",
    "         \"AttributeName\":\"target_value\",\n",
    "         \"AttributeType\":\"float\"\n",
    "      },\n",
    "      {\n",
    "         \"AttributeName\":\"item_id\",\n",
    "         \"AttributeType\":\"string\"\n",
    "      }\n",
    "   ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_name = f\"{PROJECT}_{DATA_VERSION}\"\n",
    "print(ts_dataset_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = \\\n",
    "forecast.create_dataset(Domain=\"CUSTOM\",\n",
    "                        DatasetType='TARGET_TIME_SERIES',\n",
    "                        DatasetName=ts_dataset_name,\n",
    "                        DataFrequency=DATASET_FREQUENCY,\n",
    "                        Schema=ts_schema\n",
    "                       )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_arn = response['DatasetArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset(DatasetArn=ts_dataset_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Update the dataset group with the datasets we created\n",
    "You can have multiple datasets under the same dataset group. Update it with the datasets we created before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_arns = []\n",
    "dataset_arns.append(ts_dataset_arn)\n",
    "forecast.update_dataset_group(DatasetGroupArn=dataset_group_arn, DatasetArns=dataset_arns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Create a Target Time Series Dataset Import Job <a class=\"anchor\" id=\"DataImport\"></a>\n",
    "\n",
    "\n",
    "Now that Forecast knows how to understand the CSV we are providing, the next step is to import the data from S3 into Amazon Forecaast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recall path to your data\n",
    "ts_s3_data_path = \"s3://\"+bucket_name+\"/\"+key\n",
    "print(f\"S3 URI for your data file = {ts_s3_data_path}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_import_job_response = \\\n",
    "    forecast.create_dataset_import_job(DatasetImportJobName=dataset_group,\n",
    "                                       DatasetArn=ts_dataset_arn,\n",
    "                                       DataSource= {\n",
    "                                         \"S3Config\" : {\n",
    "                                             \"Path\": ts_s3_data_path,\n",
    "                                             \"RoleArn\": role_arn\n",
    "                                         } \n",
    "                                       },\n",
    "                                       TimestampFormat=TIMESTAMP_FORMAT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_import_job_arn=ts_dataset_import_job_response['DatasetImportJobArn']\n",
    "ts_dataset_import_job_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop the data import\n",
    "\n",
    "Possibly during fine-tuning development, you'll accidentally upload data before you're ready.  If you don't want to wait for the data upload and processing, there is a handy \"Stop API\" call.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# StopResource\n",
    "stop_ts_dataset_import_job_arn = forecast.stop_resource(ResourceArn=ts_dataset_import_job_arn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the target time series dataset import job\n",
    "util.wait_till_delete(lambda: forecast.delete_dataset_import_job(DatasetImportJobArn=ts_dataset_import_job_arn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Do the data import again\n",
    "\n",
    "Maybe you fixed something you forgot before, and now you're ready to really upload the data for Forecast ingestion and processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ts_dataset_import_job_response = \\\n",
    "    forecast.create_dataset_import_job(DatasetImportJobName=dataset_group,\n",
    "                                       DatasetArn=ts_dataset_arn,\n",
    "                                       DataSource= {\n",
    "                                         \"S3Config\" : {\n",
    "                                             \"Path\": ts_s3_data_path,\n",
    "                                             \"RoleArn\": role_arn\n",
    "                                         } \n",
    "                                       },\n",
    "                                       TimestampFormat=TIMESTAMP_FORMAT)\n",
    "\n",
    "ts_dataset_import_job_arn=ts_dataset_import_job_response['DatasetImportJobArn']\n",
    "ts_dataset_import_job_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the status of dataset, when the status change from **CREATE_IN_PROGRESS** to **ACTIVE**, we can continue to next steps. Depending on the data size. It can take 10 mins to be **ACTIVE**. This process will take 5 to 10 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status = util.wait(lambda: forecast.describe_dataset_import_job(DatasetImportJobArn=ts_dataset_import_job_arn))\n",
    "assert status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast.describe_dataset_import_job(DatasetImportJobArn=ts_dataset_import_job_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps<a class=\"anchor\" id=\"nextSteps\"></a>\n",
    "\n",
    "At this point you have successfully imported your data into Amazon Forecast and now it is time to get started in the next notebook to build your first model. To Continue, execute the cell below to store important variables where they can be used in the next notebook, then open `2.Building_Your_Predictor.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now save your choices for the next notebook \n",
    "%store item_id\n",
    "%store PROJECT\n",
    "%store DATA_VERSION\n",
    "%store FORECAST_LENGTH\n",
    "%store DATASET_FREQUENCY\n",
    "%store TIMESTAMP_FORMAT\n",
    "%store ts_dataset_import_job_arn\n",
    "%store ts_dataset_arn\n",
    "%store dataset_group_arn\n",
    "%store role_arn\n",
    "%store bucket_name\n",
    "%store region\n",
    "%store key"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}