v1/python-sdk/tutorials/automl-with-azureml/forecasting-demand-forecasting-many-models/auto-ml-forecasting-demand-forecasting-many-models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "**Demand Forecasting Using Many Models**\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train Many Models](#TrainManyModels)\n",
    "1. [Train Baseline](#TrainBaseline)\n",
    "1. [Test Set Inference](#TestSetInference)\n",
    "1. [Test Set Evaluation](#TestSetEvaluation)\n",
    "1. [Generate Forecast](#GenerateForecast)\n",
    "1. [Schedule Inference Pipelines](#ScheduleInference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1. Introduction\n",
    "\n",
    "The objective of this notebook is to illustrate how to use the AutoML many models solution accelertor for demand forecasting tasks. It walks you through all stages of model evaluation and production process starting with data ingestion and concluding with scheduling inference runs.\n",
    "\n",
    "We use a subset of UCI electricity data ([link](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014#)) with the objective of predicting electricity demand per consumer 24 hours ahead. The data was preprocessed using the [data prep notebook](https://github.com/Azure/azureml-examples/blob/main/v1/python-sdk/tutorials/automl-with-azureml/forecasting-data-preparation/auto-ml-forecasting-data-preparation.ipynb). Please refer to it for illustration on how to download the data from the source, aggregate to an hourly frequency, convert from wide to long format and upload to the Datastore. Here, we will work with the already uploaded data. \n",
    "\n",
    "Having a problem description such as to generate accurate forecasts 24 hours ahead sounds like a relatively straight forward task. However, there are quite a few steps a user needs to take before the model is put in production. A user needs to prepare the data, partition it into appropriate sets, select the best model, evaluate it against a baseline, and monitor the model in real life to collect enough observations on how it would perform had it been put in production. Some of these steps are time consuming, some require certain expertise in writing code. The steps shown in this notebook follow a typical thought process one follows before the model is put in production.\n",
    "\n",
    "Make sure you have executed the [configuration](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) before running this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from azureml.pipeline.core import StepRun\n",
    "except ModuleNotFoundError:\n",
    "    print(\n",
    "        \"Please install azureml-pipeline-core with the command: \\n! pip install azureml-pipeline-core\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "import os\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.train.automl import AutoMLConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"This notebook was created using version 1.47.0 of the Azure ML SDK\")\n",
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing the Azure ML workspace requires authentication with Azure.\n",
    "\n",
    "The default authentication is interactive authentication using the default tenant. Executing the ws = Workspace.from_config() line in the cell below will prompt for authentication the first time that it is run.\n",
    "\n",
    "If you have multiple Azure tenants, you can specify the tenant by replacing the ws = Workspace.from_config() line in the cell below with the following:\n",
    "```\n",
    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
    "auth = InteractiveLoginAuthentication(tenant_id = 'mytenantid')\n",
    "ws = Workspace.from_config(auth = auth)\n",
    "```\n",
    "If you need to run in an environment where interactive login is not possible, you can use Service Principal authentication by replacing the ws = Workspace.from_config() line in the cell below with the following:\n",
    "```\n",
    "from azureml.core.authentication import ServicePrincipalAuthentication\n",
    "auth = ServicePrincipalAuthentication('mytenantid', 'myappid', 'mypassword')\n",
    "ws = Workspace.from_config(auth = auth)\n",
    "```\n",
    "For more details, see [aka.ms/aml-notebook-auth](aka.ms/aml-notebook-auth)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import uuid\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "datastore = ws.get_default_datastore()\n",
    "\n",
    "# Choose a name for the run history container in the workspace.\n",
    "experiment_name = \"forecasting-many-models-\" + datetime.datetime.now().strftime(\n",
    "    \"%Y%m%d\"\n",
    ")\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "experiment = Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "output[\"Run History Name\"] = experiment_name\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Compute \n",
    "\n",
    "#### Create or Attach existing AmlCompute\n",
    "\n",
    "You will need to create a compute target for your AutoML run. In this tutorial, you will create AmlCompute as your training compute resource.\n",
    "\n",
    "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist.\n",
    "\n",
    "\n",
    "To run deep learning models we recommend to use GPU compute. Here, we use a 5 node cluster of the `STANDARD_DS12_V2` series for illustration purposes. You will need to adjust the compute type and the number of nodes based on your needs which can be driven by the speed needed for model seelction, data size, etc. \n",
    "\n",
    "#### Creation of AmlCompute takes approximately 5 minutes. \n",
    "If the AmlCompute with that name is already in your workspace, this code will skip the creation process.\n",
    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# Choose a name for your CPU cluster\n",
    "amlcompute_cluster_name = \"demand-fcst-mm-cluster\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print(\"Found existing cluster, use it.\")\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_DS12_V2\", max_nodes=5\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the compute cluster, we also need to define the python environment, which will be used for the experiment. We can either use the curated environment, provided by AutoML (recommended), or we can create a new environment from scratch. To create a custom environment, please set the `USE_CURATED_ENV` parameter  to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.runconfig import RunConfiguration\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core import Environment\n",
    "\n",
    "aml_run_config = RunConfiguration()\n",
    "aml_run_config.target = compute_target\n",
    "\n",
    "USE_CURATED_ENV = True\n",
    "if USE_CURATED_ENV:\n",
    "    curated_environment = Environment.get(\n",
    "        workspace=ws, name=\"AzureML-sklearn-0.24-ubuntu18.04-py37-cpu\"\n",
    "    )\n",
    "    aml_run_config.environment = curated_environment\n",
    "else:\n",
    "    aml_run_config.environment.python.user_managed_dependencies = False\n",
    "\n",
    "    # Add some packages relied on by data prep step\n",
    "    aml_run_config.environment.python.conda_dependencies = CondaDependencies.create(\n",
    "        conda_packages=[\"pandas\", \"scikit-learn\"],\n",
    "        pip_packages=[\"azureml-sdk\", \"azureml-dataset-runtime[fuse,pandas]\"],\n",
    "        pin_sdk_version=False,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3. Data\n",
    "If you ran the data [preparation notebook](https://github.com/Azure/azureml-examples/blob/main/v1/python-sdk/tutorials/automl-with-azureml/forecasting-data-preparation/auto-ml-forecasting-data-preparation.ipynb) and want to use the registered data, skip section 3.1 and, instead, uncomment and execute the code in section 3.2. If, on the other hand, you did not run the notebook and want to use the data that we pre-processed and saved in the public blob, execute the code in section 3.1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Loading and registering the data from public blob store\n",
    "\n",
    "Run the code in this section only if you want to use the data that is already available in the blobstore. If you want to use your own data that is already registered in your workspace, skip this section and procceed to run the commented out code in section 3.2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code registers a datastore `autom_fcst_many_models` in your workspace and links the data from the container `automl-sample-notebook-data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Datastore\n",
    "\n",
    "# Please change the following to point to your own blob container and pass in account_key\n",
    "blob_datastore_name = \"autom_fcst_many_models\"\n",
    "container_name = \"automl-sample-notebook-data\"\n",
    "account_name = \"automlsamplenotebookdata\"\n",
    "\n",
    "print(f'Creating datastore \"{blob_datastore_name}\" in your workspace ...\\n---')\n",
    "demand_mm_datastore = Datastore.register_azure_blob_container(\n",
    "    workspace=ws,\n",
    "    datastore_name=blob_datastore_name,\n",
    "    container_name=container_name,\n",
    "    account_name=account_name,\n",
    "    create_if_not_exists=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code registers datasets from the `automl-sample-notebook-data` container in the datastore we just created. Once the datasets are registered, we will be able to use them in our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Dataset\n",
    "\n",
    "print(\"Registering datasets in your workspace ...\\n---\")\n",
    "\n",
    "FOLDER_PREFIX_NAME = \"uci_electro_small_public_mm\"\n",
    "\n",
    "target_path_train = f\"{FOLDER_PREFIX_NAME}_train\"\n",
    "target_path_test = f\"{FOLDER_PREFIX_NAME}_test\"\n",
    "target_path_inference = f\"{FOLDER_PREFIX_NAME}_infer\"\n",
    "\n",
    "train_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=demand_mm_datastore.path(target_path_train + \"/\"),\n",
    "    validate=False,\n",
    "    infer_column_types=True,\n",
    ").register(workspace=ws, name=target_path_train, create_new_version=True)\n",
    "\n",
    "test_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=demand_mm_datastore.path(target_path_test + \"/\"),\n",
    "    validate=False,\n",
    "    infer_column_types=True,\n",
    ").register(workspace=ws, name=target_path_test, create_new_version=True)\n",
    "\n",
    "inference_dataset = Dataset.Tabular.from_delimited_files(\n",
    "    path=demand_mm_datastore.path(target_path_inference + \"/\"),\n",
    "    validate=False,\n",
    "    infer_column_types=True,\n",
    ").register(workspace=ws, name=target_path_inference, create_new_version=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Using data that is registered in your workspace\n",
    "\n",
    "If you ran the [data prep notebook](https://github.com/Azure/azureml-examples/blob/main/v1/python-sdk/tutorials/automl-with-azureml/forecasting-data-preparation/auto-ml-forecasting-data-preparation.ipynb) notebook, the train, test and inference sets are already uploaded and registered in your workspace. Uncomment the following code and change the `DATASET_PREFIX_NAME`, to match the value in the data preparation notebook, and run the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from azureml.core.dataset import Dataset\n",
    "\n",
    "# DATASET_PREFIX_NAME = \"uci_electro_small_mm\"\n",
    "# print(f'Dataset prefix name: {DATASET_PREFIX_NAME}\\n---\\nLoading train, validation, test and inference sets ...\\n---')\n",
    "\n",
    "# target_path_train = f\"{DATASET_PREFIX_NAME}_train\"\n",
    "# target_path_test = f\"{DATASET_PREFIX_NAME}_test\"\n",
    "# target_path_inference = f\"{DATASET_PREFIX_NAME}_inference\"\n",
    "\n",
    "# train_dataset = Dataset.get_by_name(ws, name=target_path_train)\n",
    "# test_dataset = Dataset.get_by_name(ws, name=target_path_test)\n",
    "# inference_dataset = Dataset.get_by_name(ws, name=target_path_inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Test and inference sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have *test* and *inference* sets. The difference between the two is the presence of the target column. The test set contains the target column and is used to evaluate model performance using [rolling forecast](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-auto-train-forecast-v1#evaluating-model-accuracy-with-a-rolling-forecast). On the other hand, the target column is not present in the inference set to illustrate how to generate an actual forecast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The first few rows of the test set ...\\n---\")\n",
    "print(test_dataset.take(5).to_pandas_dataframe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The first few rows of the inference set ...\\n---\")\n",
    "print(inference_dataset.take(5).to_pandas_dataframe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set up what we know about the dataset.\n",
    "\n",
    "- **Target column** is what we want to forecast. In our case it is electricity consumption per customer measured in kilowatt hours (kWh).\n",
    "- **Time column** is the time axis along which to predict.\n",
    "- **Time series identifier columns** are identified by values of the columns listed `time_series_id_column_names`. In our case all unique time series are identified by a single column `customer_id`. However, it is quite common to have multiple columns identifying unique time series. See the [link](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-forecast#configuration-settings) for a more detailed explanation on this topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column_name = \"usage\"\n",
    "time_column_name = \"datetime\"\n",
    "GRAIN_COL = \"customer_id\"\n",
    "time_series_id_column_names = [GRAIN_COL]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we download training data from the Datastore to make sure it looks as expected. If your dataset is large, there is no need to store it in the memory. In this case, skip the next block of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = train_dataset.to_pandas_dataframe()\n",
    "\n",
    "nseries = train_df.groupby(time_series_id_column_names).ngroups\n",
    "print(\n",
    "    f\"Data contains {nseries} individual time-series:\\n{list(train_df[GRAIN_COL].unique())}\\n---\"\n",
    ")\n",
    "print(\"Printing the first few rows of the training data ...\\n---\")\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Train Many Models\n",
    "\n",
    "In this section we will train many models as well as the baseline model. The baseline model will be used as a reference point to understand many models' accuracy performance. The goal of forecasting is to have the most accurate predictions measured by some accuracy metric. What is considered an accurate prediction is fairly subjective. Take, for example, the MAPE (mean absolute percentage error) metric. A perfect forecast will result in the MAPE value of zero, which is not achievable using business data. For this reason it is imperative to have a baseline model to compare AutoML results against. Doing this adds objectivity to the model acceptance criteria. \n",
    "\n",
    "The baseline model can be the model that is currently in production. Oftentimes, the baseline is set to be a Naive forecast, which we will use in this notebook. The choice of the baseline is also specific to the data. For example, if there is a clear trend in the data one may not want to use a Naive model.  Instead, one can use an ARIMA model. Please see this [document](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-configure-auto-train-v1#supported-models) for a list of AutoML models one can chose from to use as a baseline model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following 2 parameters allow us to re-use training runs for the many-models and baseline models, respectively. This can be helpful it you need to experiment with the post model training steps thus avoiding the need to kick off a new training run which can be computationally expensive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IS_MANY_MODELS_TRAINED = False\n",
    "IS_BASE_MODEL_TRAINED = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Train AutoML model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1.1 Set up training parameters\n",
    "We need to provide the `ForecastingParameters`, `AutoMLConfig` and `ManyModelsTrainParameters` objects. For the forecasting task we also need to define several settings including the name of the time column, the maximum forecast horizon, and the partition column name(s) definition.\n",
    "\n",
    "#### Forecasting Parameters\n",
    "To define forecasting parameters for your experiment training, you can leverage the `ForecastingParameters` class. The table below details the forecasting parameters we will be passing into our experiment.\n",
    "\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**time_column_name**|The name of the time column in the data.|\n",
    "|**forecast_horizon**|The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly).|\n",
    "|**time_series_id_column_names**|The column names used to uniquely identify the time series in data that has multiple rows with the same timestamp. If the time series identifiers are not defined, the data set is assumed to be one time series.|\n",
    "| **cv_step_size**| Number of periods between two consecutive cross-validation folds. The default value is \"auto\", in which case AutoMl determines the cross-validation step size automatically, if a validation set is not provided. Or users could specify an integer value. |\n",
    "|**freq**|Forecast frequency. This optional parameter represents the period for which the forecast is desired, for example, daily, weekly, yearly, etc. Use this parameter for the correction of time series containing irregular data points or for padding of short time series. The frequency needs to be a pandas offset alias. Please refer to [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#dateoffset-objects) for more information.\n",
    "\n",
    "\n",
    "#### AutoMLConfig arguments\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "| **task**                           | forecasting |\n",
    "| **primary_metric**                 | This is the metric that you want to optimize. Forecasting supports the following primary metrics<ul><li>`normalized_root_mean_squared_error`</li><li>`normalized_mean_absolute_error`</li><li>`spearman_correlation`</li><li>`r2_score`</li></ul> We recommend using either the normalized root mean squared error or normalized mean absolute erorr as a primary metric because they measure forecast accuracy. See the [link](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-automl-forecasting-faq#how-do-i-choose-the-primary-metric) for a more detailed discussion on this topic. |\n",
    "| **experiment_timeout_hours**       | Maximum amount of time in hours that each experiment can take before it terminates. This is optional but provides customers with greater control on exit criteria. When setting this criteria we advise to take into account the number of desired iterations parameter and set experiment timeout setting such that the desired number of iterations will be completed.|\n",
    "| **iterations**                     | Number of models to train. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **label_column_name**              | The name of the target column we are trying to predict. |\n",
    "| **n_cross_validations**            | Number of cross validation splits. The default value is \"auto\", in which case AutoMl determines the number of cross-validations automatically. Or users could specify an integer value. Rolling Origin Validation is used to split time-series in a temporally consistent way.|\n",
    "| **enable_early_stopping**          | Flag to enable early termination if the primary metric is no longer improving. |\n",
    "| **blocked_models**                 | List of models we want to block. For illustration purposes and to reduce the runtime, we block all time series specific models. The defaule value is None or an empty list.|\n",
    "\n",
    "#### ManyModelsTrainParameters arguments\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "| **automl_settings** | The `AutoMLConfig` object defined above. |\n",
    "| **partition_column_names** | The names of columns used to group your models. For timeseries, the groups must not split up individual time-series. That is, each group must contain one or more whole time-series. |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, that the `time_series_id_column_names` and `partition_column_names` do not have to be the same. In our scenario, they are the same since we are interested in training one model per customer. We have 10 customers in our dataset and there will be 10 models trained. Say, you decide to cluster customers into groups, for example, each group has 2 customers and you want to train one model per group of customers. In such scenario, you will partition the data by groups, and the `time_series_id_column_names` will be different from the `partition_column_names`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
    "from azureml.train.automl.automlconfig import AutoMLConfig\n",
    "from azureml.train.automl.runtime._many_models.many_models_parameters import (\n",
    "    ManyModelsTrainParameters,\n",
    ")\n",
    "\n",
    "forecast_horizon = 24\n",
    "partition_column_names = time_series_id_column_names\n",
    "\n",
    "BLOCKED_MODELS = [\n",
    "    \"Naive\",\n",
    "    \"SeasonalNaive\",\n",
    "    \"Average\",\n",
    "    \"SeasonalAverage\",\n",
    "    \"Prophet\",\n",
    "    \"ExponentialSmoothing\",\n",
    "    \"ExtremeRandomTrees\",\n",
    "    \"AutoArima\",\n",
    "    \"Arimax\",\n",
    "]\n",
    "EXPERIMENT_TIMEOUT_HOURS = 1\n",
    "\n",
    "forecasting_parameters = ForecastingParameters(\n",
    "    time_column_name=time_column_name,\n",
    "    forecast_horizon=forecast_horizon,\n",
    "    time_series_id_column_names=time_series_id_column_names,\n",
    "    cv_step_size=\"auto\",\n",
    "    freq=\"H\",\n",
    ")\n",
    "\n",
    "automl_settings = AutoMLConfig(\n",
    "    task=\"forecasting\",\n",
    "    primary_metric=\"normalized_root_mean_squared_error\",\n",
    "    iteration_timeout_minutes=20,\n",
    "    iterations=25,\n",
    "    experiment_timeout_hours=EXPERIMENT_TIMEOUT_HOURS,\n",
    "    label_column_name=target_column_name,\n",
    "    n_cross_validations=\"auto\",  # Feel free to set to a small integer (>=2) if runtime is an issue.\n",
    "    blocked_models=BLOCKED_MODELS,\n",
    "    track_child_runs=False,\n",
    "    forecasting_parameters=forecasting_parameters,\n",
    ")\n",
    "\n",
    "mm_paramters = ManyModelsTrainParameters(\n",
    "    automl_settings=automl_settings, partition_column_names=partition_column_names\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up many models pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parallel run step is leveraged to train multiple models at once. To configure the ParallelRunConfig you will need to determine the appropriate number of workers and nodes for your use case. The `process_count_per_node` is based off the number of cores of the compute VM. The node_count will determine the number of master nodes to use, increasing the node count will speed up the training process.\n",
    "\n",
    "| Property                           | Description|\n",
    "|-|-|\n",
    "| **experiment**                     | The experiment used for training. |\n",
    "| **train_data**                     | The file dataset to be used as input to the training run. |\n",
    "| **node_count**                     | The number of compute nodes to be used for running the user script. We recommend to start with 3 and increase the node_count if the training time is taking too long. |\n",
    "| **process_count_per_node**         | Process count per node, we recommend 2:1 ratio for number of cores: number of processes per node. eg. If node has 16 cores then configure 8 or less process count per node for optimal performance. |\n",
    "| **train_pipeline_parameters**      | The set of configuration parameters defined in the previous section. |\n",
    "| **run_invocation_timeout**         | Maximum amount of time in seconds that the `ParallelRunStep` class is allowed. This is optional but provides customers with greater control on exit criteria. This must be greater than `experiment_timeout_hours` by at least 300 seconds. Here, we we add a buffer of 1000 seconds. |\n",
    "| **arguments**                      | Arguments to be passed to training script. Here, we pass the parameter `retrain_failed_models` and set it to True. If training a model for any partition fails, AutoML will kick off a new child run for that partition.|\n",
    "\n",
    "**Note**: Total time it takes for the **training step** in the pipeline to complete  equals to \n",
    "\n",
    "$$\n",
    "\\left( \\frac{t}{ p \\times n } \\right) \\times k\n",
    "$$\n",
    "\n",
    "where\n",
    "- $ t $ is time it takes to train one partition (can be viewed in the training logs)\n",
    "- $ p $ is the process count per node\n",
    "- $ n $ is the node count\n",
    "- $ k $ is total number of partitions in time series based on `partition_column_names`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder\n",
    "\n",
    "\n",
    "training_pipeline_steps = AutoMLPipelineBuilder.get_many_models_train_steps(\n",
    "    experiment=experiment,\n",
    "    train_data=train_dataset,\n",
    "    compute_target=compute_target,\n",
    "    node_count=5,\n",
    "    process_count_per_node=2,\n",
    "    run_invocation_timeout=(EXPERIMENT_TIMEOUT_HOURS * 3600 + 1000),\n",
    "    train_pipeline_parameters=mm_paramters,\n",
    "    arguments=[\"--retrain_failed_models\", \"True\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, the output of the previous cell prints out the name of the partitioned dataset. This allows you to run a new experiment on the already partiitoned dataset. What this does is it skips a data partitioning step, and reduces the runtime. To use already partioned dataset, uncomment and execute the following code _**before**_ building the `training_pipeline_steps`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from azureml.core.dataset import Dataset\n",
    "\n",
    "# PARTITIONED_TRAIN_DATASET_NAME = \"<Paste the output of the previous window here.>\"\n",
    "# train_dataset = Dataset.get_by_name(ws, name=PARTITIONED_TRAIN_DATASET_NAME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit the pipeline to run\n",
    "Next we submit our pipeline to run. The whole training pipeline takes about 20 minutes on a 5 node STANDARD_DS15_V2  cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "\n",
    "training_pipeline = Pipeline(ws, steps=training_pipeline_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Are many models trained? {IS_MANY_MODELS_TRAINED}\\n---\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not IS_MANY_MODELS_TRAINED:\n",
    "    print(\"Training new AutoML model ...\\n---\")\n",
    "    training_run = experiment.submit(training_pipeline)\n",
    "    training_run.wait_for_completion(show_output=False)\n",
    "    IS_MODEL_TRAINED = True\n",
    "else:\n",
    "    from azureml.train.automl.run import AutoMLRun\n",
    "    from azureml.pipeline.core.run import PipelineRun\n",
    "\n",
    "    PIPELINE_RUN_ID = \"<Replace with the PipelineRunId used for training many models>\"  # Copy the output of Submitted PipelineRun to re-use trained models\n",
    "    training_run = PipelineRun(experiment=experiment, run_id=PIPELINE_RUN_ID)\n",
    "    print(f\"Using previously trained model. Pipeline run ID: {PIPELINE_RUN_ID}\\n---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Train the baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use Naive model as our baseline. To train it, we kick off another automl experiment with the following settings. Please note that we added the Naive model to the allowed models list. To reduce the training time, we set the number of cross validations to 2. Read the following [document](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-auto-train-forecast-v1#training-and-validation-data) for more information on this topic.\n",
    "\n",
    "The only `AutoMLConfig` settings you might consider changing are the `experiment_timeout_hours` and `allowed_models`. You might want to increase the experiment timeout if your data has lots of unique time series. The allowed model list can be modified to refect a different choice of the baseline model and can be selected from the supported [forecasting models](https://learn.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.forecasting) and [regression models](https://learn.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.constants.supportedmodels.regression)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.automl.core.forecasting_parameters import ForecastingParameters\n",
    "\n",
    "forecasting_parameters = ForecastingParameters(\n",
    "    time_column_name=time_column_name,\n",
    "    forecast_horizon=forecast_horizon,\n",
    "    time_series_id_column_names=time_series_id_column_names,\n",
    "    cv_step_size=1,\n",
    "    freq=\"H\",\n",
    ")\n",
    "\n",
    "automl_config = AutoMLConfig(\n",
    "    task=\"forecasting\",\n",
    "    debug_log=\"baseline.log\",\n",
    "    primary_metric=\"normalized_root_mean_squared_error\",\n",
    "    experiment_timeout_hours=1,\n",
    "    training_data=train_dataset,\n",
    "    label_column_name=target_column_name,\n",
    "    compute_target=compute_target,\n",
    "    enable_early_stopping=True,\n",
    "    n_cross_validations=2,\n",
    "    verbosity=logging.INFO,\n",
    "    max_cores_per_iteration=-1,\n",
    "    enable_dnn=False,\n",
    "    allowed_models=[\"Naive\"],\n",
    "    forecasting_parameters=forecasting_parameters,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Is the baseline model trained? {IS_BASE_MODEL_TRAINED}\\n---\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if not IS_BASE_MODEL_TRAINED:\n",
    "    remote_base_run = experiment.submit(automl_config, show_output=False)\n",
    "    remote_base_run.wait_for_completion(show_output=False)\n",
    "    IS_BASE_MODEL_TRAINED = True\n",
    "else:\n",
    "    from azureml.train.automl.run import AutoMLRun\n",
    "\n",
    "    BASE_RUN_ID = \"<Replace with the run ID used for training the baseline model>\"\n",
    "    # during the initial training run copy-paste the run id to be utilized later if needed.\n",
    "    remote_base_run = AutoMLRun(experiment=experiment, run_id=BASE_RUN_ID)\n",
    "    print(f\"Using previously trained model. Run ID: {BASE_RUN_ID}\\n---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Test set inference "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Many models inferences\n",
    "\n",
    "We create an output folder which will be used to save the output of our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an output folder\n",
    "OUTPUT_DIR = \"forecast_output\"\n",
    "os.makedirs(OUTPUT_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_experiment = Experiment(ws, experiment_name + \"_inference\")\n",
    "test_experiment_base = Experiment(ws, experiment_name + \"_inference_base\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1.1 Set up output dataset for inference data\n",
    "Output of inference can be represented as [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.output_dataset_config.outputdatasetconfig?view=azure-ml-py) object and OutputFileDatasetConfig can be registered as a dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.data import OutputFileDatasetConfig\n",
    "\n",
    "output_test_data_ds = OutputFileDatasetConfig(\n",
    "    name=\"many_models_inference_output\",\n",
    "    destination=(datastore, \"uci_electro_small/test_set_output/\"),\n",
    ").register_on_complete(name=\"uci_electro_small_test_data_ds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many models we need to provide the ManyModelsInferenceParameters object.\n",
    "\n",
    "#### `ManyModelsInferenceParameters` arguments\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **partition_column_names**         | List of column names that identifies groups. |\n",
    "| **target_column_name**             | \\[Optional\\] Column name only if the inference dataset has the target. |\n",
    "| **time_column_name**               | \\[Optional\\] Time column name only if it is timeseries. |\n",
    "| **inference_type**                 | \\[Optional\\] Which inference method to use on the model. For the forecasting tasks set this value to 'forecast'. |\n",
    "| **forecast_mode**                  | \\[Optional\\] The type of forecast to be used, either 'rolling' or 'recursive'; defaults to 'recursive'. |\n",
    "| **step**                           | \\[Optional\\] Number of periods to advance the forecasting window in each iteration **(for rolling forecast only)**; defaults to 1. |\n",
    "\n",
    "#### `get_many_models_batch_inference_steps` arguments\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **experiment**                     | The experiment used for inference run. |\n",
    "| **inference_data**                 | The data to use for inferencing. It should be the same schema as used for training.\n",
    "| **compute_target**                 | The compute target that runs the inference pipeline. |\n",
    "| **node_count**                     | The number of compute nodes to be used for running the user script. We recommend to start with the number of cores per node (varies by compute sku). |\n",
    "| **process_count_per_node**         | \\[Optional\\] The number of processes per node. By default it's 2 (should be at most half of the number of cores in a single node of the compute cluster that will be used for the experiment).\n",
    "| **inference_pipeline_parameters**  | \\[Optional\\] The `ManyModelsInferenceParameters` object defined above. |\n",
    "| **append_row_file_name**           | \\[Optional\\] The name of the output file (optional, default value is 'parallel_run_step.txt'). Supports 'txt' and 'csv' file extension. A 'txt' file extension generates the output in 'txt' format with space as separator without column names. A 'csv' file extension generates the output in 'csv' format with comma as separator and with column names. |\n",
    "| **train_run_id**                   | \\[Optional\\] The run id of the **training pipeline**. By default it is the latest successful training pipeline run in the experiment. |\n",
    "| **train_experiment_name**          | \\[Optional\\] The train experiment that contains the train pipeline. This one is only needed when the train pipeline is not in the same experiement as the inference pipeline. |\n",
    "| **run_invocation_timeout**         | \\[Optional\\] Maximum amount of time in seconds that the `ParallelRunStep` class is allowed. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **output_datastore**               | \\[Optional\\] The `Datastore` or `OutputDatasetConfig` to be used for output. If specified any pipeline output will be written to that location. If unspecified the default datastore will be used. |\n",
    "| **arguments**                      | \\[Optional\\] Arguments to be passed to inference script. Possible argument is '--forecast_quantiles' followed by quantile values. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder\n",
    "from azureml.train.automl.runtime._many_models.many_models_parameters import (\n",
    "    ManyModelsInferenceParameters,\n",
    ")\n",
    "\n",
    "output_file_name = \"parallel_run_step.csv\"\n",
    "\n",
    "mm_parameters = ManyModelsInferenceParameters(\n",
    "    partition_column_names=time_series_id_column_names,\n",
    "    time_column_name=time_column_name,\n",
    "    target_column_name=target_column_name,\n",
    ")\n",
    "\n",
    "inference_steps = AutoMLPipelineBuilder.get_many_models_batch_inference_steps(\n",
    "    experiment=experiment,\n",
    "    inference_data=test_dataset,\n",
    "    node_count=2,\n",
    "    process_count_per_node=8,\n",
    "    compute_target=compute_target,\n",
    "    run_invocation_timeout=300,\n",
    "    output_datastore=output_test_data_ds,\n",
    "    train_run_id=training_run.id,\n",
    "    train_experiment_name=training_run.experiment.name,\n",
    "    inference_pipeline_parameters=mm_parameters,\n",
    "    append_row_file_name=output_file_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "\n",
    "inference_pipeline = Pipeline(ws, steps=inference_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1.2 Run the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "inference_run = experiment.submit(inference_pipeline)\n",
    "inference_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Retreive the test set predictions from the many models\n",
    "\n",
    "The forecasting pipeline forecasts the orange juice quantity for a Store by Brand. The pipeline returns one file with the predictions for each store and outputs the result to the forecasting_output Blob container. The details of the blob container is listed in 'forecasting_output.txt' under Outputs+logs. \n",
    "\n",
    "The following code snippet:\n",
    "1. Downloads the contents of the output folder that is passed in the parallel run step \n",
    "2. Reads the output file that has the predictions as pandas dataframe \n",
    "3. Displays the top 5 rows of the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from azureml.pipeline.core import StepRun\n",
    "# ! pip show azureml-pipeline-core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps.utilities import get_output_from_mm_pipeline\n",
    "\n",
    "forecasting_results_name = \"forecasting_results\"\n",
    "forecasting_output_name = \"many_models_inference_output\"\n",
    "forecast_file = get_output_from_mm_pipeline(\n",
    "    inference_run, forecasting_results_name, forecasting_output_name, output_file_name\n",
    ")\n",
    "df = pd.read_csv(forecast_file)\n",
    "print(\n",
    "    \"Prediction has \", df.shape[0], \" rows. Here the first 5 rows are being displayed.\"\n",
    ")\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns={\"Predictions\": \"predicted\"}, inplace=True)\n",
    "df.to_csv(os.path.join(OUTPUT_DIR, \"test-set-predictions-many-models.csv\"), index=False)\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 Inference the baseline model\n",
    "\n",
    "Next, we perform a rolling evaluation on the test set for the baseline model. To do this, we use the `run_remote_inference` method which downloads the pickle file of the model into the temporary folder `forecast_naive` and copies the `inference_script_naive.py` file to it. This folder is then uploaded on the compute cluster where inference is performed. The `inference_script_naive.py` script performs a rolling evaluation on the test set, similarly to what we have done in section 6.2. Upon completion of this step, we delete the newly created `forecast_naive` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_run = remote_base_run.get_best_child()\n",
    "baseline_model_name = baseline_run.properties[\"model_name\"]\n",
    "baseline_run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import shutil\n",
    "from scripts.helper_scripts import run_remote_inference_naive\n",
    "\n",
    "remote_base_run_test = run_remote_inference_naive(\n",
    "    test_experiment=test_experiment_base,\n",
    "    compute_target=compute_target,\n",
    "    train_run=baseline_run,\n",
    "    test_dataset=test_dataset,\n",
    "    target_column_name=target_column_name,\n",
    "    rolling_evaluation_step_size=forecast_horizon,\n",
    "    inference_folder=\"./forecast_naive\",\n",
    ")\n",
    "remote_base_run_test.wait_for_completion(show_output=False)\n",
    "\n",
    "# download the forecast file to the local machine\n",
    "print(\"Downloading test data with prediction ...\\n---\")\n",
    "remote_base_run_test.download_file(\n",
    "    \"outputs/predictions.csv\",\n",
    "    os.path.join(OUTPUT_DIR, \"test-set-predictions-base.csv\"),\n",
    ")\n",
    "\n",
    "# delete downloaded scripts\n",
    "print(\"Removing auxiliary files ...\\n---\")\n",
    "shutil.rmtree(\"./forecast_naive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Test set model evaluation\n",
    "\n",
    "In this section we will evaluate the test set performance for many models and compare it with the baseline. We will generate time series plots for forecasts and actuals, calculate accuracy metrics and plot the evolution of metrics for each model over time. All output from this section will be stored in the `forecast_output` folder and can be referenced any time you need it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 Load test set results\n",
    "\n",
    "Here, we will import test set results for both many-models and baseline experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "backtest_automl = pd.read_csv(\n",
    "    os.path.join(OUTPUT_DIR, \"test-set-predictions-many-models.csv\"),\n",
    "    parse_dates=[time_column_name],\n",
    ")\n",
    "backtest_base = pd.read_csv(\n",
    "    os.path.join(OUTPUT_DIR, \"test-set-predictions-base.csv\"),\n",
    "    parse_dates=[time_column_name],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we combine outputs into a single dataframe which will be used for plotting and scoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "backtest = backtest_automl.merge(\n",
    "    backtest_base.drop(target_column_name, axis=1),\n",
    "    on=[\"customer_id\", \"datetime\"],\n",
    "    how=\"inner\",\n",
    "    suffixes=[\"\", \"_base\"],\n",
    ")\n",
    "backtest.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f\"N model: {backtest_automl.shape[0]}. N baseline: {backtest_base.shape[0]}. N merged: {backtest.shape[0]}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data we are working with has an hourly frequency and we plan to generate the forecasts every 24 hours. If the model were to be put in production such that the forecasts are generated and model's performance is monitored every 24 hours, we will mimic the scoring process on the test set by generating daily accuracy metrics. To do this, we create a date column (\"ymd\"). If you want to score the output at any other frequency, say, weekly, just change the frequency parameter to the desired frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PERIOD_COLUMN = \"ymd\"\n",
    "\n",
    "backtest[PERIOD_COLUMN] = backtest[time_column_name].dt.to_period(\n",
    "    \"D\"\n",
    ")  # year-month-day to be used for daily metrics computation\n",
    "backtest.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2 Generate time series plots\n",
    "\n",
    "Here, we generate forecast versus actuals plot for the test set for both the best many models and the baseline. Since we use rolling evaluation with the step size of 24 hours, this mimics the behavior of putting both models in production and monitoring their behavior for the duration of the test set. This step allows users to make informed decisons about model performance and saves numerous costs associated with productionalizing the model and monitoring its performance in real life. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.helper_scripts import draw_one_plot\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib.backends.backend_pdf import PdfPages\n",
    "\n",
    "plot_filename = \"forecast_vs_actual.pdf\"\n",
    "\n",
    "pdf = PdfPages(os.path.join(os.getcwd(), OUTPUT_DIR, plot_filename))\n",
    "for _, one_forecast in backtest.groupby(GRAIN_COL):\n",
    "    one_forecast[time_column_name] = pd.to_datetime(one_forecast[time_column_name])\n",
    "    one_forecast.sort_values(time_column_name, inplace=True)\n",
    "    draw_one_plot(\n",
    "        one_forecast,\n",
    "        time_column_name,\n",
    "        target_column_name,\n",
    "        [GRAIN_COL],\n",
    "        [target_column_name, \"predicted\", \"predicted_base\"],\n",
    "        pdf,\n",
    "        plot_predictions=True,\n",
    "    )\n",
    "pdf.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3 Calculate metrics\n",
    "Here, we will calculate the metric of interest for each day. For illustration purposes we use root mean squared error as the metric of choice. However, the `compute_all_metrics` method calculated all primary and secondary metrics for AutoML runs. Please refer to this <u>*Regression/forecasting metrics*</u> section in this [document](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-understand-automated-ml#regressionforecasting-metrics) for the list of available metrics. We will calculate the distribution of this metric for each time series in our dataset. Looking at the descrptive stats of such metrics can be more informative than calculating a single metric such as the mean for each time series. As an example, we are looking at the RMSE (root mean squared error) metric, but you can choose any other metric computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.helper_scripts import compute_all_metrics\n",
    "\n",
    "metrics_per_grain_day = compute_all_metrics(\n",
    "    fcst_df=backtest,\n",
    "    actual_col=target_column_name,\n",
    "    fcst_col=\"predicted\",\n",
    "    ts_id_colnames=[GRAIN_COL, PERIOD_COLUMN],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    f'List of available metrics: {metrics_per_grain_day[\"metric_name\"].unique()}\\n---'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DESIRED_METRIC_NAME = \"root_mean_squared_error\"\n",
    "\n",
    "metrics_per_grain_day = metrics_per_grain_day.query(\n",
    "    f'metric_name == \"{DESIRED_METRIC_NAME}\"'\n",
    ")\n",
    "metrics_per_grain_day[[GRAIN_COL, PERIOD_COLUMN]] = metrics_per_grain_day[\n",
    "    \"time_series_id\"\n",
    "].str.split(\"|\", 1, expand=True)\n",
    "metrics_per_grain_day.to_csv(\n",
    "    os.path.join(OUTPUT_DIR, \"metrics-automl.csv\"), index=False\n",
    ")\n",
    "metrics_per_grain_day.groupby(GRAIN_COL)[\"metric\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_per_grain_day_base = compute_all_metrics(\n",
    "    fcst_df=backtest,\n",
    "    actual_col=target_column_name,\n",
    "    fcst_col=\"predicted_base\",\n",
    "    ts_id_colnames=[GRAIN_COL, PERIOD_COLUMN],\n",
    ")\n",
    "metrics_per_grain_day_base = metrics_per_grain_day_base.query(\n",
    "    f'metric_name == \"{DESIRED_METRIC_NAME}\"'\n",
    ")\n",
    "metrics_per_grain_day_base[[GRAIN_COL, PERIOD_COLUMN]] = metrics_per_grain_day[\n",
    "    \"time_series_id\"\n",
    "].str.split(\"|\", 1, expand=True)\n",
    "metrics_per_grain_day_base.to_csv(\n",
    "    os.path.join(OUTPUT_DIR, \"metrics-base.csv\"), index=False\n",
    ")\n",
    "metrics_per_grain_day_base.groupby(GRAIN_COL)[\"metric\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.4  Visualize metrics\n",
    "\n",
    "In this section we plot metric evolution over time for the best AutoML and the baseline models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_df = metrics_per_grain_day.drop(\"time_series_id\", axis=1).merge(\n",
    "    metrics_per_grain_day_base.drop(\"time_series_id\", axis=1),\n",
    "    on=[\"metric_name\", GRAIN_COL, PERIOD_COLUMN],\n",
    "    how=\"inner\",\n",
    "    suffixes=[\"\", \"_base\"],\n",
    ")\n",
    "metrics_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Save the metrics evolution plots to pdf file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grain = [GRAIN_COL]\n",
    "plot_filename = \"metrics_plot.pdf\"\n",
    "\n",
    "pdf = PdfPages(os.path.join(os.getcwd(), OUTPUT_DIR, plot_filename))\n",
    "for _, one_forecast in metrics_df.groupby(grain):\n",
    "    one_forecast[PERIOD_COLUMN] = pd.to_datetime(one_forecast[PERIOD_COLUMN])\n",
    "    one_forecast.sort_values(PERIOD_COLUMN, inplace=True)\n",
    "    draw_one_plot(\n",
    "        one_forecast,\n",
    "        PERIOD_COLUMN,\n",
    "        target_column_name,\n",
    "        grain,\n",
    "        [\"metric\", \"metric_base\"],\n",
    "        pdf,\n",
    "        plot_predictions=True,\n",
    "    )\n",
    "pdf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame(os.path.join(\"./forecast_output/metrics_plot.pdf\"), width=800, height=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Inference\n",
    "\n",
    "In this step, we generate an actual forecast by providing an inference set that does not contain actual values. This illustrates how to generate production forecasts in real life. The code in this section is pretty much identical to the one in section 6.1 with one exception, we set the `run_rolling_evaluation` argument to `False`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.1 Set up output dataset for inference data\n",
    "\n",
    "Output of inference step can be represented as [OutputFileDatasetConfig](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.output_dataset_config.outputdatasetconfig?view=azure-ml-py) object which, in turn, will be registered as a dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data import OutputFileDatasetConfig\n",
    "\n",
    "output_inference_data_ds = OutputFileDatasetConfig(\n",
    "    name=\"many_models_inference_output\",\n",
    "    destination=(datastore, \"uci_electro_small/inference_output/\"),\n",
    ").register_on_complete(name=\"uci_electro_small_inference_data_ds\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_dataset.take(5).to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For many models we need to provide the ManyModelsInferenceParameters object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps import AutoMLPipelineBuilder\n",
    "from azureml.train.automl.runtime._many_models.many_models_parameters import (\n",
    "    ManyModelsInferenceParameters,\n",
    ")\n",
    "\n",
    "output_file_name = \"parallel_run_step.csv\"\n",
    "inference_ds_small = inference_dataset\n",
    "\n",
    "mm_parameters = ManyModelsInferenceParameters(\n",
    "    partition_column_names=time_series_id_column_names,\n",
    "    time_column_name=time_column_name,\n",
    ")\n",
    "\n",
    "inference_steps = AutoMLPipelineBuilder.get_many_models_batch_inference_steps(\n",
    "    experiment=experiment,\n",
    "    inference_data=inference_ds_small,\n",
    "    node_count=2,\n",
    "    process_count_per_node=8,\n",
    "    compute_target=compute_target,\n",
    "    run_invocation_timeout=300,\n",
    "    output_datastore=output_inference_data_ds,\n",
    "    train_run_id=training_run.id,\n",
    "    train_experiment_name=training_run.experiment.name,\n",
    "    inference_pipeline_parameters=mm_parameters,\n",
    "    append_row_file_name=output_file_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.pipeline.core import Pipeline\n",
    "\n",
    "inference_pipeline = Pipeline(ws, steps=inference_steps)\n",
    "inference_run = experiment.submit(inference_pipeline)\n",
    "inference_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8.2 Get the predicted data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.automl.pipeline.steps.utilities import get_output_from_mm_pipeline\n",
    "\n",
    "forecasting_results_name = \"forecasting_results\"\n",
    "forecasting_output_name = \"many_models_inference_output\"\n",
    "forecast_file = get_output_from_mm_pipeline(\n",
    "    inference_run, forecasting_results_name, forecasting_output_name, output_file_name\n",
    ")\n",
    "inference_df = pd.read_csv(forecast_file)\n",
    "print(\n",
    "    \"Prediction has \",\n",
    "    inference_df.shape[0],\n",
    "    \" rows. Here the first 5 rows are being displayed.\",\n",
    ")\n",
    "inference_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_df.rename(columns={\"Predictions\": \"predicted\"}, inplace=True)\n",
    "inference_df.to_csv(\n",
    "    os.path.join(OUTPUT_DIR, \"inference-set-predictions-many-models.csv\"), index=False\n",
    ")\n",
    "inference_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Schedule Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section is about how to schedule a pipeline for periodically predictions. For more info about pipeline schedule and pipeline endpoint, please follow this [notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-setup-schedule-for-a-published-pipeline.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_published_pipeline = inference_pipeline.publish(\n",
    "    name=\"automl_forecast_many_models\",\n",
    "    description=\"forecast many models\",\n",
    "    version=\"1\",\n",
    "    continue_on_step_failure=False,\n",
    ")\n",
    "print(\"Newly published pipeline id: {}\".format(inference_published_pipeline.id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `inference_dataset` is going to refresh every 24 hours and we want to predict every 24 hours (forecast_horizon), we can schedule our pipeline to run every day at 11 pm to get daily inference results. You can refresh your test dataset (a newer version will be created) periodically when new data is available (i.e. target column in test dataset would have values in the beginning as context data, and followed by NaNs to be predicted). The inference pipeline will pick up context to further improve the forecast accuracy. See the <u><i>Forecasting away from training data</i></u> in this [notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/automated-machine-learning/forecasting-forecast-function/auto-ml-forecasting-function.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `test_dataset` is going to refresh every 4 weeks before Friday 16:00 and we want to predict every 4 weeks (forecast_horizon), we can schedule our pipeline to run every 4 weeks at 16:00 to get daily inference results. You can refresh your test dataset (a newer version will be created) periodically when new data is available (i.e. target column in test dataset would have values in the beginning as context data, and followed by NaNs to be predicted). The inference pipeline will pick up context to further improve the forecast accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.pipeline.core.schedule import ScheduleRecurrence, Schedule\n",
    "\n",
    "recurrence = ScheduleRecurrence(\n",
    "    frequency=\"Day\", interval=1, hours=[23], minutes=[00]  # Runs every day at 11:00 pm\n",
    ")\n",
    "\n",
    "schedule = Schedule.create(\n",
    "    workspace=ws,\n",
    "    name=\"many_models_inference_schedule\",\n",
    "    pipeline_id=inference_published_pipeline.id,\n",
    "    experiment_name=\"schedule-run-mm-uci-electro\",\n",
    "    recurrence=recurrence,\n",
    "    wait_for_provisioning=True,\n",
    "    description=\"Schedule Run\",\n",
    ")\n",
    "\n",
    "# You may want to make sure that the schedule is provisioned properly\n",
    "# before making any further changes to the schedule\n",
    "\n",
    "print(\"Created schedule with id: {}\\n---\".format(schedule.id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1 [Optional] Disable schedule"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schedule.disable()"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "jialiu"
   }
  ],
  "category": "tutorial",
  "celltoolbar": "Raw Cell Format",
  "compute": [
   "Remote"
  ],
  "datasets": [
   "Orange Juice Sales"
  ],
  "deployment": [
   "Azure Container Instance"
  ],
  "exclude_from_index": false,
  "framework": [
   "Azure ML AutoML"
  ],
  "friendly_name": "Forecasting orange juice sales with deployment",
  "index_order": 1,
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "tags": [
   "None"
  ],
  "task": "Forecasting",
  "vscode": {
   "interpreter": {
    "hash": "6bd77c88278e012ef31757c15997a7bea8c943977c43d6909403c00ae11d43ca"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}