sagemaker-pipelines/time_series_forecasting/amazon_forecast_pipeline/sm_pipeline_with_amazon_forecast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1815f18f",
   "metadata": {},
   "source": [
    "# Creating an Amazon Forecast Predictor with SageMaker Pipelines\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76ef2077",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf3049e4",
   "metadata": {},
   "source": [
    "\n",
    "This example notebook showcases how you can create a dataset, dataset group and predictor with Amazon Forecast and SageMaker Pipelines. This demo is designed to run on SageMaker Notebook Instances. As of February 2022, this code will not properly execute in SageMaker Studio, due to a docker limitation on SageMaker Studio Notebooks.\n",
    "\n",
    "\n",
    "Integrating SageMaker Pipelines with Amazon Forecast is useful for the following three reasons:\n",
    "1. Iteratively improve your model by tracking the performance of each execution using SageMaker Experiments.\n",
    "2. Reproducibility of Forecast experiments. \n",
    "3. Decouple different processes in your Amazon Forecast machine learning project and visualize these in a Directed Acyclic Graph using SageMaker Pipelines.\n",
    "\n",
    "This notebook can be used as a template to start training your own Forecast predictors with SageMaker Pipelines. Before you start, make sure that your SageMaker Execution Role has the following policies:\n",
    "\n",
    "- `AmazonForecastFullAccess`\n",
    "- `AmazonSageMakerFullAccess`\n",
    "\n",
    "Your SageMaker Execution Role should have access to S3 already. If not you can add an S3 policy.\n",
    "You will also need to the inline policy described below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fac8375",
   "metadata": {},
   "outputs": [],
   "source": [
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"VisualEditor0\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"iam:GetRole\",\n",
    "                \"s3:*\",\n",
    "                \"iam:CreateRole\",\n",
    "                \"iam:AttachRolePolicy\",\n",
    "                \"forecast:*\",\n",
    "            ],\n",
    "            \"Resource\": \"*\",\n",
    "        }\n",
    "    ],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d70b3e0b",
   "metadata": {},
   "source": [
    "Finally, you will need the following trust policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1489fcdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"Service\": [\"s3.amazonaws.com\", \"forecast.amazonaws.com\", \"sagemaker.amazonaws.com\"]\n",
    "            },\n",
    "            \"Action\": \"sts:AssumeRole\",\n",
    "        }\n",
    "    ],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7e7b206",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "First, we are going to import the SageMaker SDK and set some default variables such as the `role` for permissioned execution and the `default_bucket` to store model artifacts.\n",
    "\n",
    "Then, we have to update the base Scikit-learn SageMaker image to update boto3 and botocore. As of February 2022, the Scikit-learn image has an older version of botocore (*1.19.4*) which does not yet contain code for API calls you need to make to Amazon Forecast. \n",
    "The script below creates an ECR repository with the given `repo_name` within your AWS account in the region you are running this notebook from. It then pulls as base image the Prebuilt Amazon SageMaker Docker Image for Scikit-learn. This notebook automatically selects the correct `image_acc_id` for the region you're in using the `region_to_account_id` dictionary, according to https://docs.aws.amazon.com/sagemaker/latest/dg/pre-built-docker-containers-scikit-learn-spark.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eca88d44",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install sagemaker==2.93.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b76a5a4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import tarfile\n",
    "import time\n",
    "\n",
    "import boto3\n",
    "import botocore\n",
    "import sagemaker\n",
    "import os\n",
    "import zipfile\n",
    "import pandas as pd\n",
    "from sagemaker.processing import ProcessingInput, ProcessingOutput\n",
    "from sagemaker.sklearn.estimator import SKLearn\n",
    "from sagemaker.sklearn.processing import ScriptProcessor\n",
    "from sagemaker.workflow.execution_variables import ExecutionVariables\n",
    "from sagemaker.workflow.functions import Join\n",
    "from sagemaker.workflow.parameters import (\n",
    "    ParameterInteger,\n",
    "    ParameterString,\n",
    ")\n",
    "from sagemaker.workflow.pipeline_experiment_config import PipelineExperimentConfig\n",
    "from sagemaker.workflow.steps import ProcessingStep, TrainingStep\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "sagemaker_session = sagemaker.session.Session()\n",
    "role_arn = sagemaker.get_execution_role()\n",
    "image_uri = sagemaker.image_uris.retrieve(\n",
    "    framework=\"sklearn\", region=region, version=\"1.0-1\", image_scope=\"training\"\n",
    ")\n",
    "image_acc_id = image_uri.split(\".\")[0]\n",
    "default_bucket = sagemaker_session.default_bucket()\n",
    "default_bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a713cf0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%sh -s \"$image_acc_id\" \"$image_uri\"\n",
    "\n",
    "# The name of our algorithm\n",
    "repo_name=sagemaker-sklearn-botocore-updated\n",
    "\n",
    "account=$(aws sts get-caller-identity --query Account --output text)\n",
    "\n",
    "# Get the region defined in the current configuration (default to us-west-2 if none defined)\n",
    "region=$(aws configure get region)\n",
    "\n",
    "# Write the Dockerfile\n",
    "mkdir docker\n",
    "cd docker\n",
    "\n",
    "printf \"FROM $2 \\n\n",
    "RUN python3 -m pip install --upgrade pip \\n\n",
    "RUN python3 -m pip install boto3==1.20.25 \\n\n",
    "RUN python3 -m pip install botocore==1.23.25 \" > Dockerfile \n",
    "\n",
    "\n",
    "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${repo_name}:latest\"\n",
    "aws_base_image_acc=\"$1.dkr.ecr.${region}.amazonaws.com\"\n",
    "\n",
    "# If the repository doesn't exist in ECR, create it.\n",
    "aws ecr describe-repositories --repository-names \"${repo_name}\" > /dev/null 2>&1\n",
    "\n",
    "if [ $? -ne 0 ]\n",
    "then\n",
    "    aws ecr create-repository --repository-name \"${repo_name}\" > /dev/null\n",
    "fi\n",
    "\n",
    "# Get the login command from ECR and execute it directly\n",
    "aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin ${aws_base_image_acc}\n",
    "\n",
    "# Build the docker image locally with the image name and then push it to ECR\n",
    "# with the full name.\n",
    "docker build -t ${repo_name} .\n",
    "docker tag ${repo_name} ${fullname}\n",
    "\n",
    "aws ecr get-login-password --region ${region}|docker login --username AWS --password-stdin \"${account}\".dkr.ecr.\"${region}\".amazonaws.com\n",
    "docker push ${fullname}\n",
    "\n",
    "# Clean up unencrypted credentials and Dockerfile\n",
    "cd ..\n",
    "rm -rf docker\n",
    "> /home/ec2-user/.docker/config.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29352a0c",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "Let's inspect the train dataset we will be using in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cf21bea",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_HOST = f\"sagemaker-example-files-prod-{boto3.Session().region_name}\"\n",
    "DATA_PATH = \"datasets/timeseries/uci_electricity/\"\n",
    "ARCHIVE_NAME = \"LD2011_2014.txt.zip\"\n",
    "FILE_NAME = ARCHIVE_NAME[:-4]\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "\n",
    "if not os.path.isfile(FILE_NAME):\n",
    "    print(\"downloading dataset (258MB), can take a few minutes depending on your connection\")\n",
    "    s3_client.download_file(DATA_HOST, DATA_PATH + ARCHIVE_NAME, ARCHIVE_NAME)\n",
    "\n",
    "    print(\"\\nextracting data archive\")\n",
    "    zip_ref = zipfile.ZipFile(ARCHIVE_NAME, \"r\")\n",
    "    zip_ref.extractall(\"./\")\n",
    "    zip_ref.close()\n",
    "else:\n",
    "    print(\"File found skipping download\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1d3b4d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(FILE_NAME, sep=\";\", index_col=0, parse_dates=True, decimal=\",\")\n",
    "\n",
    "# Take only one target time series\n",
    "df = df[[\"MT_001\"]]\n",
    "print(df.index.min())\n",
    "print(df.index.max())\n",
    "df.head()\n",
    "\n",
    "df.to_csv(\"train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9a4707d",
   "metadata": {},
   "source": [
    "The dataset happens to span January 01, 2011, to January 01, 2015. We are only going to use about two and a half week's of hourly data to train Amazon Forecast. \n",
    "We will copy the dataset from this local directory to s3 so that SageMaker can access it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11d3d20b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp ./train.csv s3://$default_bucket/forecast_pipeline_example/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7c3ed10",
   "metadata": {},
   "source": [
    "Next, we define parameters that can be set for the execution of the pipeline. They serve as variables. We define the following:\n",
    "\n",
    "- `ProcessingInstanceCount`: The number of processing instances to use for the execution of the pipeline\n",
    "- `ProcessingInstanceType`: The type of processing instances to use for the execution of the pipeline\n",
    "- `TrainingInstanceCount`: The number of training instances to use for the execution of the pipeline\n",
    "- `TrainingInstanceType`: The type of training instances to use for the execution of the pipeline\n",
    "- `TrainData`: Location of the training data in S3\n",
    "- `ModelOutput`: Location of the target S3 path for the Amazon Forecast model artifact\n",
    "\n",
    "Amazon Forecast creates its own validation set when training, so there is no need to provide one.\n",
    "\n",
    "We also define some important parameters to choose, train and evaluate the model \n",
    "- `ForecastHorizon`: The Forecast Horizon (Prediction length) \n",
    "- `ForecastAlgorithm`: What algorithm to use from Amazon Forecast (ex DeepArPlus, CNNQR, ...)\n",
    "- `EvaluationMetric`: The evaluation metric used to select (keep) the model \n",
    "- `MaxScore`: The evaluation metric's threshold to select (keep) the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77c85e30",
   "metadata": {},
   "outputs": [],
   "source": [
    "processing_instance_count = ParameterInteger(name=\"ProcessingInstanceCount\", default_value=1)\n",
    "processing_instance_type = ParameterString(\n",
    "    name=\"ProcessingInstanceType\", default_value=\"ml.m5.large\"\n",
    ")\n",
    "training_instance_type = ParameterString(name=\"TrainingInstanceType\", default_value=\"ml.m5.large\")\n",
    "\n",
    "input_train = ParameterString(\n",
    "    name=\"TrainData\",\n",
    "    default_value=f\"s3://{default_bucket}/forecast_pipeline_example/train.csv\",\n",
    ")\n",
    "model_output = ParameterString(name=\"ModelOutput\", default_value=f\"s3://{default_bucket}/model\")\n",
    "\n",
    "# Model parameters\n",
    "forecast_horizon = ParameterString(name=\"ForecastHorizon\", default_value=\"24\")\n",
    "forecast_algorithm = ParameterString(name=\"ForecastAlgorithm\", default_value=\"NPTS\")\n",
    "maximum_score = ParameterString(name=\"MaxScore\", default_value=\"0.4\")\n",
    "metric = ParameterString(name=\"EvaluationMetric\", default_value=\"WAPE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1bc4f5b",
   "metadata": {},
   "source": [
    "We use an updated [SKLearnProcessor](https://sagemaker.readthedocs.io/en/stable/frameworks/sklearn/sagemaker.sklearn.html#sagemaker.sklearn.processing.SKLearnProcessor) to run Python scripts to build a dataset group and train an Amazon Forecast predictor using `boto3`. In the next chunk, we instantiate an instance of `ScriptProcessor`, which is essentially an SKLearnProcessor with updated `boto3` and `botocore` (as built above) that we use in the next steps. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58dcb8b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "account_id = role_arn.split(\":\")[4]\n",
    "ecr_repository_name = \"sagemaker-sklearn-botocore-updated\"\n",
    "tag = \"latest\"\n",
    "container_image_uri = \"{0}.dkr.ecr.{1}.amazonaws.com/{2}:{3}\".format(\n",
    "    account_id, region, ecr_repository_name, tag\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b681a3a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "sklearn_processor = ScriptProcessor(\n",
    "    image_uri=container_image_uri,\n",
    "    command=[\"python3\"],\n",
    "    instance_type=processing_instance_type,\n",
    "    instance_count=processing_instance_count,\n",
    "    base_job_name=\"forecast-process\",\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    role=role_arn,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e34e02e6",
   "metadata": {},
   "source": [
    "First we preprocess the data using an Amazon SageMaker [ProcessingStep](https://sagemaker.readthedocs.io/en/stable/workflows/pipelines/sagemaker.workflow.pipelines.html?highlight=ProcessingStep#sagemaker.workflow.steps.ProcessingStep) that provides a containerized execution environment to run the `preprocess.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2b3fd55",
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess = ProcessingStep(\n",
    "    name=\"ForecastPreProcess\",\n",
    "    processor=sklearn_processor,\n",
    "    inputs=[\n",
    "        ProcessingInput(source=input_train, destination=\"/opt/ml/processing/input_train\"),\n",
    "    ],\n",
    "    outputs=[\n",
    "        ProcessingOutput(output_name=\"target\", source=\"/opt/ml/processing/target\"),\n",
    "        ProcessingOutput(output_name=\"related\", source=\"/opt/ml/processing/related\"),\n",
    "    ],\n",
    "    job_arguments=[\"--forecast_horizon\", forecast_horizon],\n",
    "    code=\"preprocess.py\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2912dd48",
   "metadata": {},
   "source": [
    "The next step is to train and evaluate the forecasting model calling Amazon Forecast using `boto3`. We instantiate an instance of `SKLearn` estimator that we use in the next `TrainingStep` to run the script `train.py`. \n",
    "Amazon Forecast automatically evaluates the performance on an evaluation set. We will use that score as a condition for deploying the model.\n",
    "The algorithm training is managed by Amazon Forecast. We use a `TrainingStep` instead of a `ProcessingStep` to log the metrics with SageMaker Experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb34964a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the hyperparmeters and the Regex Syntax associated to the metrics\n",
    "hyperparameters = {\n",
    "    \"forecast_horizon\": forecast_horizon,\n",
    "    \"forecast_algorithm\": forecast_algorithm,\n",
    "    \"dataset_frequency\": \"H\",\n",
    "    \"timestamp_format\": \"yyyy-MM-dd hh:mm:ss\",\n",
    "    \"number_of_backtest_windows\": \"1\",\n",
    "    \"s3_directory_target\": preprocess.properties.ProcessingOutputConfig.Outputs[\n",
    "        \"target\"\n",
    "    ].S3Output.S3Uri,\n",
    "    \"s3_directory_related\": preprocess.properties.ProcessingOutputConfig.Outputs[\n",
    "        \"related\"\n",
    "    ].S3Output.S3Uri,\n",
    "    \"role_arn\": role_arn,\n",
    "    \"region\": region,\n",
    "}\n",
    "metric_definitions = [\n",
    "    {\"Name\": \"WAPE\", \"Regex\": \"WAPE=(.*?);\"},\n",
    "    {\"Name\": \"RMSE\", \"Regex\": \"RMSE=(.*?);\"},\n",
    "    {\"Name\": \"MASE\", \"Regex\": \"MASE=(.*?);\"},\n",
    "    {\"Name\": \"MAPE\", \"Regex\": \"MAPE=(.*?);\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "831ad760",
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_model = SKLearn(\n",
    "    entry_point=\"train.py\",\n",
    "    role=role_arn,\n",
    "    image_uri=container_image_uri,\n",
    "    instance_type=training_instance_type,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    base_job_name=\"forecast-train\",\n",
    "    hyperparameters=hyperparameters,\n",
    "    enable_sagemaker_metrics=True,\n",
    "    metric_definitions=metric_definitions,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26d1c58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast_train_and_eval = TrainingStep(name=\"ForecastTrainAndEvaluate\", estimator=forecast_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4474c13",
   "metadata": {},
   "source": [
    "The third step is an Amazon SageMaker ProcessingStep that deletes or keeps the Amazon Forecast model running using the script `conditional_delete.py`. If the error reported after training is higher than a threshold you specify for the metric you specify, this step deletes all the resources created by Amazon Forecast that are related to the pipeline's execution.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7853447d",
   "metadata": {},
   "outputs": [],
   "source": [
    "postprocess = ProcessingStep(\n",
    "    name=\"ForecastCondtionalDelete\",\n",
    "    processor=sklearn_processor,\n",
    "    inputs=[\n",
    "        ProcessingInput(\n",
    "            source=forecast_train_and_eval.properties.ModelArtifacts.S3ModelArtifacts,\n",
    "            destination=\"/opt/ml/processing/model\",\n",
    "        ),\n",
    "    ],\n",
    "    job_arguments=[\n",
    "        \"--metric\",\n",
    "        metric,\n",
    "        \"--maximum-score\",\n",
    "        maximum_score,\n",
    "        \"--region\",\n",
    "        region,\n",
    "    ],\n",
    "    code=\"conditional_delete.py\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cd13b49",
   "metadata": {},
   "source": [
    "Finally, we combine all the steps and define our pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b79c25d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.workflow.pipeline import Pipeline\n",
    "\n",
    "pipeline_name = \"ForecastPipeline\"\n",
    "pipeline = Pipeline(\n",
    "    name=pipeline_name,\n",
    "    parameters=[\n",
    "        processing_instance_type,\n",
    "        processing_instance_count,\n",
    "        training_instance_type,\n",
    "        input_train,\n",
    "        forecast_horizon,\n",
    "        forecast_algorithm,\n",
    "        model_output,\n",
    "        metric,\n",
    "        maximum_score,\n",
    "    ],\n",
    "    steps=[preprocess, forecast_train_and_eval, postprocess],\n",
    "    pipeline_experiment_config=PipelineExperimentConfig(\n",
    "        ExecutionVariables.PIPELINE_NAME,\n",
    "        Join(on=\"-\", values=[\"ForecastTrial\", ExecutionVariables.PIPELINE_EXECUTION_ID]),\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86ef05f",
   "metadata": {},
   "source": [
    "Once the pipeline is successfully defined, we can start the execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3251cc33",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.upsert(role_arn=role_arn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2765ddf",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution = pipeline.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78437af3",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.wait(delay=300, max_attempts=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da0f66e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "execution.list_steps()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d73ecf",
   "metadata": {},
   "source": [
    "## Experiments Tracking\n",
    "\n",
    "Each pipeline execution is tracked by default when using SageMaker Pipelines. To find the experiment tracking in **SageMaker Studio**, you should open **SageMaker Resources** and select **Experiments and Trials**. The experiments and trials are organized as follows:\n",
    "\n",
    "* The Pipeline **ForecastPipeline** is associated with an Experiment. \n",
    "    * Each execution of Pipeline **ForecastPipeline** is associated with a trial.\n",
    "        * Each step within the execution is associated with a trial component within trial.\n",
    "\n",
    "To find the Trial Components and Trial name generated by a pipeline execution in **ForecastPipeline**, you:\n",
    "\n",
    "   1. Open **SageMaker Resources** and select **Experiments and Trials**. \n",
    "   2. Right-Click on your Pipeline’s name (**ForecastPipeline**) and select **Open in trial component list**\n",
    "   3. You can now filter the trial components and customize the table view as presented in [View and Compare Amazon SageMaker Experiments, Trials, and Trial Components](https://docs.aws.amazon.com/sagemaker/latest/dg/experiments-view-compare.html).\n",
    "\n",
    "The AWS documentation for Experiments Tracking can be found under [Manage Machine Learning with Amazon SageMaker Experiments](https://docs.aws.amazon.com/sagemaker/latest/dg/experiments.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d86ae3d",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3673936d",
   "metadata": {},
   "source": [
    "In this notebook we have seen how to create a SageMaker Pipeline to train an Amazon Forecast predictor on your own dataset with a target and related time series."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc51892f",
   "metadata": {},
   "source": [
    "## Clean up\n",
    "\n",
    "Feel free to clean up all related resources (the pipeline, s3 object (train.csv), all Forecast related resources) that could potentially incur costs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "758154af",
   "metadata": {},
   "outputs": [],
   "source": [
    "def wait_till_delete(callback, check_time=5, timeout=100):\n",
    "    elapsed_time = 0\n",
    "    while timeout is None or elapsed_time < timeout:\n",
    "        try:\n",
    "            out = callback()\n",
    "        except botocore.exceptions.ClientError as e:\n",
    "            # When given the resource not found exception, deletion has occured\n",
    "            if e.response[\"Error\"][\"Code\"] == \"ResourceNotFoundException\":\n",
    "                print(\"Successful delete\")\n",
    "                return\n",
    "            else:\n",
    "                raise\n",
    "        time.sleep(check_time)  # units of seconds\n",
    "        elapsed_time += check_time\n",
    "\n",
    "    raise TimeoutError(\"Forecast resource deletion timed-out.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5c04767",
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region)\n",
    "sagemaker = session.client(service_name=\"sagemaker\")\n",
    "\n",
    "steps = execution.list_steps()\n",
    "training_job_name = steps[1][\"Metadata\"][\"TrainingJob\"][\"Arn\"].split(\"/\")[1]\n",
    "response = sagemaker.describe_training_job(TrainingJobName=training_job_name)\n",
    "s3_artifacts = response[\"ModelArtifacts\"][\"S3ModelArtifacts\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "008eb7ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1174af7",
   "metadata": {},
   "outputs": [],
   "source": [
    "! aws s3 cp $s3_artifacts ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee508a66",
   "metadata": {},
   "outputs": [],
   "source": [
    "forecast = session.client(service_name=\"forecast\")\n",
    "\n",
    "session.resource(\"s3\").Bucket(default_bucket).Object(\"train.csv\").delete()\n",
    "\n",
    "with tarfile.open(\"model.tar.gz\") as tar:\n",
    "    tar.extractall(path=\".\")\n",
    "\n",
    "with open(\"model_parameters.json\", \"r\") as f:\n",
    "    model_params = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d18293b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "wait_till_delete(\n",
    "    lambda: forecast.delete_predictor(PredictorArn=model_params[\"forecast_arn_predictor\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "561ceb74",
   "metadata": {},
   "outputs": [],
   "source": [
    "for arn in [\"target_import_job_arn\", \"related_import_job_arn\"]:\n",
    "    wait_till_delete(\n",
    "        lambda: forecast.delete_dataset_import_job(DatasetImportJobArn=model_params[arn])\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24fe7e3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "for arn in [\"target_dataset_arn\", \"related_dataset_arn\"]:\n",
    "    wait_till_delete(lambda: forecast.delete_dataset(DatasetArn=model_params[arn]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35452811",
   "metadata": {},
   "outputs": [],
   "source": [
    "wait_till_delete(\n",
    "    lambda: forecast.delete_dataset_group(DatasetGroupArn=model_params[\"dataset_group_arn\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02e44695",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd4fa036",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/sagemaker-pipelines|time_series_forecasting|amazon_forecast_pipeline|sm_pipeline_with_amazon_forecast.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}