examples/grocery_sales/python/03_model_tune_deploy/azure_hyperdrive_lightgbm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Tuning and  Deployment using Azure Machine Learning Service\n",
    "\n",
    "In this notebook, we perform hyperparameter tuning of a LightGBM retail sales forecast model using HyperDrive in Azure Machine Learning (AzureML). After the optimal hyperparameters are found, we further deploy the best model as a web service on Azure.\n",
    "\n",
    "To tune the hyperparameters, we carry out cross-validation with the Orange Juice data from week 40 to week 135. Specifically, we split the data into a training set and a validation set. Then, we train LightGBM models with different sets of hyperparameters on the training set and evaluate the accuracy of each model on the validation set. The set of hyperparameters that yield the best validation accuracy will be used to train forecast models when the data beyond week 135 is available, e.g., in the multi-round training examples provided in [examples/02_model](../02_model).\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "To run this notebook, you need to start from a conda environment where AzureML SDK is installed. In our case, we can first activate `forecasting_env` environment by\n",
    "```\n",
    "conda activate forecasting_env\n",
    "```\n",
    "as we have installed AzureML SDK in this environment. Then, we can start the notebook via\n",
    "```\n",
    "jupyter notebook\n",
    "```\n",
    "In addition, you need to install and enable AzureML widget extension in your environment by running the following commands. For your convenience, we will do this in the beginning of this notebook.\n",
    "```\n",
    "jupyter nbextension install --py --user azureml.widgets\n",
    "jupyter nbextension enable --py --user azureml.widgets\n",
    "```\n",
    "\n",
    "Besides, you need to create an AzureML workspace and download its configuration file (`config.json`) by following the instructions in [configuration.ipynb](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Note: If you run into any issue with installing and enabling the AzureML widgets below, please *uncomment* the first line in the following cell to manually install `azureml-widgets`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Installing /data/anaconda/envs/forecasting_env/lib/python3.6/site-packages/azureml/widgets/static -> azureml_widgets\n",
      "Up to date: /data/home/chenhui/.local/share/jupyter/nbextensions/azureml_widgets/index.js\n",
      "Up to date: /data/home/chenhui/.local/share/jupyter/nbextensions/azureml_widgets/extension.js\n",
      "Up to date: /data/home/chenhui/.local/share/jupyter/nbextensions/azureml_widgets/packages/labextension/azureml_widgets-1.1.0.tgz\n",
      "- Validating: \u001b[32mOK\u001b[0m\n",
      "\n",
      "    To initialize this nbextension in the browser every time the notebook (or other app) loads:\n",
      "    \n",
      "          jupyter nbextension enable azureml.widgets --user --py\n",
      "    \n",
      "Enabling notebook extension azureml_widgets/extension...\n",
      "      - Validating: \u001b[32mOK\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# Install and enable AzureML widgets\n",
    "#!pip -q install azureml-widgets==1.0.85\n",
    "!jupyter nbextension install --py --user azureml.widgets\n",
    "!jupyter nbextension enable --py --user azureml.widgets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Turning diagnostics collection on. \n",
      "Azure ML SDK Version:  1.0.85\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import json\n",
    "import shutil\n",
    "import azureml\n",
    "import requests\n",
    "import numpy as np\n",
    "from azureml.core import (\n",
    "    Experiment,\n",
    "    ScriptRunConfig,\n",
    ")\n",
    "from azureml.telemetry import set_diagnostics_collection\n",
    "from azureml.core.runconfig import (\n",
    "    RunConfiguration,\n",
    "    EnvironmentDefinition,\n",
    "    CondaDependencies,\n",
    ")\n",
    "from azureml.train.estimator import Estimator\n",
    "from azureml.widgets import RunDetails\n",
    "from azureml.train.hyperdrive import (\n",
    "    BayesianParameterSampling,\n",
    "    HyperDriveConfig,\n",
    "    quniform,\n",
    "    uniform,\n",
    "    choice,\n",
    "    PrimaryMetricGoal,\n",
    ")\n",
    "from azureml.core.webservice import AciWebservice\n",
    "from azureml.core.model import Model, InferenceConfig\n",
    "from fclib.common.utils import git_repo_path, module_path\n",
    "from fclib.azureml.azureml_utils import (\n",
    "    get_or_create_workspace,\n",
    "    get_or_create_amlcompute,\n",
    ")\n",
    "from fclib.dataset.ojdata import download_ojdata, split_train_test\n",
    "\n",
    "cur_dir = os.getcwd()\n",
    "if cur_dir not in sys.path:\n",
    "    sys.path.append(cur_dir)\n",
    "from aml_scripts.train_validate import create_features\n",
    "\n",
    "# Opt-in diagnostics for better experience of future releases\n",
    "set_diagnostics_collection(send_diagnostics=True)\n",
    "\n",
    "# Check core SDK version number\n",
    "print(\"Azure ML SDK Version: \", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use False if you've already downloaded and split the data\n",
    "DOWNLOAD_SPLIT_DATA = True\n",
    "\n",
    "# Get data directory\n",
    "DATA_DIR = os.path.join(git_repo_path(), \"ojdata\")\n",
    "\n",
    "# Forecasting settings\n",
    "N_SPLITS = 1\n",
    "HORIZON = 2\n",
    "GAP = 2\n",
    "FIRST_WEEK = 40\n",
    "LAST_WEEK = 138"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize Workspace & Create an AzureML Experiment\n",
    "\n",
    "Initialize a [Machine Learning Workspace](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace) object from the workspace you created in the Prerequisites step. `get_or_create_workspace()` below creates a workspace object from the details stored in `config.json` that you have downloaded. We assume that you store this config file to a directory `./.azureml`. In case the existing workspace cannot be loaded, the following cell will try to create a new workspace with the subscription ID, resource group, and workspace name as specified at the beginning of the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workspace name: chhamlws\n",
      "Azure region: westcentralus\n",
      "Resource group: chhamlwsrg\n"
     ]
    }
   ],
   "source": [
    "# Please specify the AzureML workspace attributes below if you want to create a new one.\n",
    "subscription_id = \"<subscription-id>\"\n",
    "resource_group = \"<resource-group>\"\n",
    "workspace_name = \"<workspace-name>\"\n",
    "workspace_region = \"<workspace-region>\"\n",
    "\n",
    "# Connect to a workspace\n",
    "ws = get_or_create_workspace(\n",
    "    config_path=\"./.azureml\",\n",
    "    subscription_id=subscription_id,\n",
    "    resource_group=resource_group,\n",
    "    workspace_name=workspace_name,\n",
    "    workspace_region=workspace_region,\n",
    ")\n",
    "print(\n",
    "    \"Workspace name: \" + ws.name,\n",
    "    \"Azure region: \" + ws.location,\n",
    "    \"Resource group: \" + ws.resource_group,\n",
    "    sep=\"\\n\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an experiment\n",
    "exp = Experiment(workspace=ws, name=\"tune-lgbm-forecast\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "\n",
    "We need to download the Orange Juice data and split it into training and test sets. By default, the following cell will download and spit the data. If you've already done so, you may skip this part by switching `DOWNLOAD_SPLIT_DATA` to False. \n",
    "\n",
    "By passing `write_csv=True` to `split_train_test()` below, this function will write the training data and test data to three csv files: `train.csv`, `auxi.csv` and `test.csv`. The first two csv files contain the historical sales up to week 135 as well as auxiliary information such as future price and promotion. Here we assume that future price and promotion information up to a certain number of weeks ahead is predetermined and known. We will use these two files to implement cross-validation and search for the best model with HyperDrive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data already exists at the specified location.\n"
     ]
    }
   ],
   "source": [
    "if DOWNLOAD_SPLIT_DATA:\n",
    "    download_ojdata(DATA_DIR)\n",
    "    split_train_test(\n",
    "        DATA_DIR,\n",
    "        n_splits=N_SPLITS,\n",
    "        horizon=HORIZON,\n",
    "        gap=GAP,\n",
    "        first_week=FIRST_WEEK,\n",
    "        last_week=LAST_WEEK,\n",
    "        write_csv=True,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate Script Locally\n",
    "\n",
    "A good practice is to test the model training and validation script on your local machine before you run the hyperparameter tuning job on a remote compute. To run the script locally, we need to correctly specify the path of the Python interpreter that has been installed in `forecasting_env` conda environment. We will try to find the Python interpreter path in the cell below. In case it cannot be retrieved automatically, you can also look for the path by executing `which python` (for Linux) or `where python` (for Windows) from a terminal where `forecasting_env` is activated and set `python_path` as the path that you find.\n",
    "\n",
    "In what follows, the script `train_validate.py` trains a model on the training set with the input arguments as specified in `ScriptRunConfig()` and computes the accuracy of the model on the validation set. Here we evaluate the model accuracy using mean-absolute-percentage-error (MAPE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get Python interpreter path\n",
    "python_path = module_path(\"forecasting_env\", \"python\")\n",
    "\n",
    "# Configure local, user managed environment\n",
    "run_config_user_managed = RunConfiguration()\n",
    "run_config_user_managed.environment.python.user_managed_dependencies = True\n",
    "run_config_user_managed.environment.python.interpreter_path = python_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Directory of the local scripts\n",
    "script_folder = \"./aml_scripts\"\n",
    "\n",
    "# Copy feature engineering utils\n",
    "src_dir = os.path.join(git_repo_path(), \"fclib\", \"fclib\", \"feature_engineering\")\n",
    "lib_dir = os.path.join(script_folder, \"fclib\")\n",
    "des_dir = os.path.join(lib_dir, \"feature_engineering\")\n",
    "if os.path.isdir(lib_dir):\n",
    "    shutil.rmtree(lib_dir)\n",
    "shutil.copytree(src_dir, des_dir)\n",
    "\n",
    "# Training script name and path\n",
    "train_script_name = \"train_validate.py\"\n",
    "train_script_path = os.path.join(script_folder, train_script_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will submit a local run if the Python interpreter path is valid and wait until the local run finishes. Then, we print out the validation metric. Moreover, you can also use `run_local.get_details()` to get detailed information about this run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'MAPE': 66.59144474679267}\n"
     ]
    }
   ],
   "source": [
    "if python_path:\n",
    "    # Specify script run config\n",
    "    src = ScriptRunConfig(\n",
    "        source_directory=\"./\",\n",
    "        script=train_script_path,\n",
    "        arguments=[\"--data-folder\", DATA_DIR, \"--bagging-fraction\", \"0.8\"],\n",
    "        run_config=run_config_user_managed,\n",
    "    )\n",
    "    # Submit local run\n",
    "    run_local = exp.submit(src)\n",
    "\n",
    "    # Check results\n",
    "    run_local.wait_for_completion()\n",
    "    metric = run_local.get_metrics()\n",
    "    print(metric)\n",
    "else:\n",
    "    print(\"Can't find Python interpreter path. Local validation will be skipped.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Script on Remote Compute\n",
    "\n",
    "After validating model training script locally, we can create a remote compute and further test the script on the remote compute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a CPU cluster as compute target\n",
    "\n",
    "In the next cell, we create an AmlCompute target with a specific cluster name, VM size, and maximum number of nodes if the cluster does not exist. Otherwise, we will reuse an existing one. For more options of VM sizes, you can check information in this [link](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-general)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found compute target: cpu-cluster\n"
     ]
    }
   ],
   "source": [
    "# Choose a name for your cluster\n",
    "cluster_name = \"cpu-cluster\"\n",
    "# VM Size\n",
    "vm_size = \"STANDARD_D2_V2\"\n",
    "# Maximum number of nodes of the cluster\n",
    "max_nodes = 4\n",
    "\n",
    "# Create a new AmlCompute if it does not exist or reuse an existing one\n",
    "compute_target = get_or_create_amlcompute(\n",
    "    workspace=ws,\n",
    "    compute_name=cluster_name,\n",
    "    vm_size=vm_size,\n",
    "    min_nodes=0,\n",
    "    max_nodes=max_nodes,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Docker environment\n",
    "\n",
    "The remote compute will need to create a [Docker image](https://docs.docker.com/get-started/) for running the script. The Docker image is an encapsulated environment with necessary dependencies installed. In the following cell, we specify the conda packages and Python version that are needed for running the script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = EnvironmentDefinition()\n",
    "env.python.user_managed_dependencies = False\n",
    "env.python.conda_dependencies = CondaDependencies.create(\n",
    "    conda_packages=[\"pandas\", \"numpy\", \"scipy\", \"scikit-learn\", \"lightgbm\", \"joblib\"],\n",
    "    python_version=\"3.6.2\",\n",
    ")\n",
    "env.python.conda_dependencies.add_channel(\"conda-forge\")\n",
    "env.docker.enabled = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload data to default datastore\n",
    "\n",
    "Each workspace comes with a default datastore. In the following, we upload the Orange Juice dataset to the workspace's default datastore, which will later be mounted on the cluster for model training and validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datastore type: AzureBlob\n",
      "Account name: chhamlws4931040064\n",
      "Container name: azureml-blobstore-f799a640-1ca3-4877-ad24-08eef7bd307e\n"
     ]
    }
   ],
   "source": [
    "ds = ws.get_default_datastore()\n",
    "print(\n",
    "    \"Datastore type: \" + ds.datastore_type,\n",
    "    \"Account name: \" + ds.account_name,\n",
    "    \"Container name: \" + ds.container_name,\n",
    "    sep=\"\\n\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "$AZUREML_DATAREFERENCE_0fd85d0787ad41e7b99139d1933ea9b9"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remote data path\n",
    "path_on_datastore = \"data\"\n",
    "ds.upload(\n",
    "    src_dir=DATA_DIR,\n",
    "    target_path=path_on_datastore,\n",
    "    overwrite=True,\n",
    "    show_progress=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$AZUREML_DATAREFERENCE_00b162eda80e4ee9a050e69c71311ecc\n"
     ]
    }
   ],
   "source": [
    "# Get data reference object for the data path\n",
    "ds_data = ds.path(path_on_datastore)\n",
    "print(ds_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create estimator\n",
    "\n",
    "Next, we will check if the remote compute target is successfully created by submitting a job to the target. This compute target will be used by HyperDrive for hyperparameter tuning later. Note that you may skip this part and directly go to [Tune Hyperparameters using HyperDrive](#tune-hyperparameters-using-hyperdrive) if you want.\n",
    "\n",
    "In the following cells, we first create an estimator to specify details of the job. Then we sumbit the job to the remote compute and check the status of the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_params = {\"--data-folder\": ds_data.as_mount(), \"--bagging-fraction\": 0.8}\n",
    "est = Estimator(\n",
    "    source_directory=script_folder,\n",
    "    script_params=script_params,\n",
    "    compute_target=compute_target,\n",
    "    use_docker=True,\n",
    "    entry_script=train_script_name,\n",
    "    environment_definition=env,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Submit job to remote compute\n",
    "run_remote = exp.submit(config=est)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check job status\n",
    "\n",
    "You can monitor the status of the remote run using the AzureML widgets. After the job is done, the following cell will display a dashboard similar to the snapshot below. You will see the dashboard when you run the notebook. However, the Jupyter widget state cannot be saved in the notebook. That's why we see the current output message of the following cell.  \n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/20047467/76150936-67fc7c00-607d-11ea-9354-418bd5f733d6.png\" width=\"900\" height=\"360\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f86dd06da77e43e8975bd8a695afcb8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_UserRunWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': True, 'log_level': 'INFO', 's…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/tune-lgbm-forecast/runs/tune-lgbm-forecast_1585967817_b60bbc6e?wsid=/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourcegroups/chhamlwsrg/workspaces/chhamlws\", \"run_id\": \"tune-lgbm-forecast_1585967817_b60bbc6e\", \"run_properties\": {\"run_id\": \"tune-lgbm-forecast_1585967817_b60bbc6e\", \"created_utc\": \"2020-04-04T02:37:04.210372Z\", \"properties\": {\"_azureml.ComputeTargetType\": \"amlcompute\", \"ContentSnapshotId\": \"f53e530a-0dc6-4573-af3b-fa3e6b6f9991\", \"azureml.git.repository_uri\": \"git@github.com:microsoft/forecasting.git\", \"mlflow.source.git.repoURL\": \"git@github.com:microsoft/forecasting.git\", \"azureml.git.branch\": \"chenhui/enhancement\", \"mlflow.source.git.branch\": \"chenhui/enhancement\", \"azureml.git.commit\": \"93da5e12f190bb96bcb7de6bad0280073e43b984\", \"mlflow.source.git.commit\": \"93da5e12f190bb96bcb7de6bad0280073e43b984\", \"azureml.git.dirty\": \"True\", \"AzureML.DerivedImageName\": \"azureml/azureml_7842fd2c5e99a43f1cca1341b66a0ecb\", \"ProcessInfoFile\": \"azureml-logs/process_info.json\", \"ProcessStatusFile\": \"azureml-logs/process_status.json\"}, \"tags\": {\"_aml_system_ComputeTargetStatus\": \"{\\\"AllocationState\\\":\\\"steady\\\",\\\"PreparingNodeCount\\\":0,\\\"RunningNodeCount\\\":0,\\\"CurrentNodeCount\\\":4}\"}, \"script_name\": null, \"arguments\": null, \"end_time_utc\": \"2020-04-04T02:39:25.162168Z\", \"status\": \"Completed\", \"log_files\": {\"azureml-logs/55_azureml-execution-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/azureml-logs/55_azureml-execution-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt?sv=2019-02-02&sr=b&sig=DduXomTRGTxuMQ4V6n2qdTWAzqWobPAa%2BgfUb3XTtpU%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"azureml-logs/65_job_prep-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/azureml-logs/65_job_prep-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt?sv=2019-02-02&sr=b&sig=9SL92v6oBKvs3U6v7wpc2i6bnhSriBFKDYFOOncm79U%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"azureml-logs/70_driver_log.txt\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/azureml-logs/70_driver_log.txt?sv=2019-02-02&sr=b&sig=Wgd4KfIzVCciOILVowjVT57w9Mo7uEBvNZVag8d3%2BWw%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"azureml-logs/75_job_post-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/azureml-logs/75_job_post-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt?sv=2019-02-02&sr=b&sig=Tf%2ByHLidR7mUEan%2BHAkh7kMmre5cD1bh8EqayfX%2BQTE%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"azureml-logs/process_info.json\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/azureml-logs/process_info.json?sv=2019-02-02&sr=b&sig=WtGEJUAQq925I8htjL8b%2FIM9y2Mp3Kh%2FDR4URRf4JZ0%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"azureml-logs/process_status.json\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/azureml-logs/process_status.json?sv=2019-02-02&sr=b&sig=q0y4E0REUTBa5t1TPhKKD3oyJS8xpl2GcAHajFNLgYA%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"logs/azureml/138_azureml.log\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/logs/azureml/138_azureml.log?sv=2019-02-02&sr=b&sig=C7SgQUqoJNVXDnEB7F6eUKNXyO8dRZlP5V24waycRtY%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"logs/azureml/job_prep_azureml.log\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/logs/azureml/job_prep_azureml.log?sv=2019-02-02&sr=b&sig=ybVIQN040pWWLpFQeGitV%2BrzuO9ojxErW%2B1ijX%2F8VV0%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\", \"logs/azureml/job_release_azureml.log\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.tune-lgbm-forecast_1585967817_b60bbc6e/logs/azureml/job_release_azureml.log?sv=2019-02-02&sr=b&sig=aV3H31e%2FdOnXtR%2BJmmRwSR7HLGJdeMeZ7vtla%2BNLgY0%3D&st=2020-04-04T02%3A29%3A27Z&se=2020-04-04T10%3A39%3A27Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/process_info.json\", \"azureml-logs/process_status.json\", \"logs/azureml/job_prep_azureml.log\", \"logs/azureml/job_release_azureml.log\"], [\"azureml-logs/55_azureml-execution-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt\"], [\"azureml-logs/65_job_prep-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt\"], [\"azureml-logs/70_driver_log.txt\"], [\"azureml-logs/75_job_post-tvmps_64205f30874f615b57c2717938d4039e14020b51244803148cd50a8d15cabd4c_d.txt\"], [\"logs/azureml/138_azureml.log\"]], \"run_duration\": \"0:02:20\"}, \"child_runs\": [], \"children_metrics\": {}, \"run_metrics\": [{\"name\": \"MAPE\", \"run_id\": \"tune-lgbm-forecast_1585967817_b60bbc6e\", \"categories\": [0], \"series\": [{\"data\": [66.59144474679267]}]}], \"run_logs\": \"2020-04-04 02:38:15,684|azureml|DEBUG|Inputs:: kwargs: {'OutputCollection': True, 'snapshotProject': True, 'only_in_process_features': True, 'skip_track_logs_dir': True}, track_folders: None, deny_list: None, directories_to_watch: []\\n2020-04-04 02:38:15,684|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Execution target type: batchai\\n2020-04-04 02:38:15,691|azureml.history._tracking.PythonWorkingDirectory|DEBUG|Failed to import pyspark with error: No module named 'pyspark'\\n2020-04-04 02:38:15,691|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Pinning working directory for filesystems: ['pyfs']\\n2020-04-04 02:38:15,914|azureml._base_sdk_common.user_agent|DEBUG|Fetching client info from /root/.azureml/clientinfo.json\\n2020-04-04 02:38:15,914|azureml._base_sdk_common.user_agent|DEBUG|Error loading client info: [Errno 2] No such file or directory: '/root/.azureml/clientinfo.json'\\n2020-04-04 02:38:16,226|azureml.core.run|DEBUG|Adding new factory <function ScriptRun._from_run_dto at 0x7fdf748c0488> for run source azureml.scriptrun\\n2020-04-04 02:38:16,226|azureml.core.authentication.TokenRefresherDaemon|DEBUG|Starting daemon and triggering first instance\\n2020-04-04 02:38:16,233|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:16,233|azureml._restclient.clientbase|INFO|Created a worker pool for first use\\n2020-04-04 02:38:16,233|azureml.core.authentication|DEBUG|Time to expire 1814327.766101 seconds\\n2020-04-04 02:38:16,234|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,234|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,234|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,234|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,234|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,265|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,265|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,266|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:16,271|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:16,280|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:16,286|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:16,293|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:16,299|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:16,299|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.RunClient.get-async:False|DEBUG|[START]\\n2020-04-04 02:38:16,299|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|Request URL: 'https://westcentralus.experiments.azureml.net/history/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runs/tune-lgbm-forecast_1585967817_b60bbc6e'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|Request method: 'GET'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|Request headers:\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '881085ef-cb0d-4095-a8a2-6dfd8db90588'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|    'request-id': '881085ef-cb0d-4095-a8a2-6dfd8db90588'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1067-azure-x86_64-with-debian-stretch-sid) msrest/0.6.11 azureml._restclient/core.1.0.85'\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|Request body:\\n2020-04-04 02:38:16,300|msrest.http_logger|DEBUG|None\\n2020-04-04 02:38:16,300|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2020-04-04 02:38:16,301|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2020-04-04 02:38:16,301|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2020-04-04 02:38:16,301|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2020-04-04 02:38:16,340|msrest.http_logger|DEBUG|Response status: 200\\n2020-04-04 02:38:16,340|msrest.http_logger|DEBUG|Response headers:\\n2020-04-04 02:38:16,340|msrest.http_logger|DEBUG|    'Date': 'Sat, 04 Apr 2020 02:38:16 GMT'\\n2020-04-04 02:38:16,340|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2020-04-04 02:38:16,340|msrest.http_logger|DEBUG|    'Transfer-Encoding': 'chunked'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'Vary': 'Accept-Encoding'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '881085ef-cb0d-4095-a8a2-6dfd8db90588'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'x-request-time': '0.020'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|    'Content-Encoding': 'gzip'\\n2020-04-04 02:38:16,341|msrest.http_logger|DEBUG|Response content:\\n2020-04-04 02:38:16,342|msrest.http_logger|DEBUG|{\\n  \\\"runNumber\\\": 302,\\n  \\\"rootRunId\\\": \\\"tune-lgbm-forecast_1585967817_b60bbc6e\\\",\\n  \\\"experimentId\\\": \\\"3199ea18-1505-42f9-9092-777f27df73e5\\\",\\n  \\\"createdUtc\\\": \\\"2020-04-04T02:37:04.210372+00:00\\\",\\n  \\\"createdBy\\\": {\\n    \\\"userObjectId\\\": \\\"8157bc92-2d12-4bc9-9270-bab51c673493\\\",\\n    \\\"userPuId\\\": \\\"10033FFF97A21586\\\",\\n    \\\"userIdp\\\": null,\\n    \\\"userAltSecId\\\": null,\\n    \\\"userIss\\\": \\\"https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/\\\",\\n    \\\"userTenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\",\\n    \\\"userName\\\": \\\"Chenhui Hu\\\"\\n  },\\n  \\\"userId\\\": \\\"8157bc92-2d12-4bc9-9270-bab51c673493\\\",\\n  \\\"token\\\": null,\\n  \\\"tokenExpiryTimeUtc\\\": null,\\n  \\\"error\\\": null,\\n  \\\"warnings\\\": null,\\n  \\\"revision\\\": 9,\\n  \\\"runUuid\\\": \\\"f667e716-d546-4c3e-83e8-c0db0a607d76\\\",\\n  \\\"parentRunUuid\\\": null,\\n  \\\"rootRunUuid\\\": \\\"f667e716-d546-4c3e-83e8-c0db0a607d76\\\",\\n  \\\"runId\\\": \\\"tune-lgbm-forecast_1585967817_b60bbc6e\\\",\\n  \\\"parentRunId\\\": null,\\n  \\\"status\\\": \\\"Running\\\",\\n  \\\"startTimeUtc\\\": \\\"2020-04-04T02:37:59.4792144+00:00\\\",\\n  \\\"endTimeUtc\\\": null,\\n  \\\"heartbeatEnabled\\\": false,\\n  \\\"options\\\": {\\n    \\\"generateDataContainerIdIfNotSpecified\\\": true\\n  },\\n  \\\"name\\\": null,\\n  \\\"dataContainerId\\\": \\\"dcid.tune-lgbm-forecast_1585967817_b60bbc6e\\\",\\n  \\\"description\\\": null,\\n  \\\"hidden\\\": false,\\n  \\\"runType\\\": \\\"azureml.scriptrun\\\",\\n  \\\"properties\\\": {\\n    \\\"_azureml.ComputeTargetType\\\": \\\"amlcompute\\\",\\n    \\\"ContentSnapshotId\\\": \\\"f53e530a-0dc6-4573-af3b-fa3e6b6f9991\\\",\\n    \\\"azureml.git.repository_uri\\\": \\\"git@github.com:microsoft/forecasting.git\\\",\\n    \\\"mlflow.source.git.repoURL\\\": \\\"git@github.com:microsoft/forecasting.git\\\",\\n    \\\"azureml.git.branch\\\": \\\"chenhui/enhancement\\\",\\n    \\\"mlflow.source.git.branch\\\": \\\"chenhui/enhancement\\\",\\n    \\\"azureml.git.commit\\\": \\\"93da5e12f190bb96bcb7de6bad0280073e43b984\\\",\\n    \\\"mlflow.source.git.commit\\\": \\\"93da5e12f190bb96bcb7de6bad0280073e43b984\\\",\\n    \\\"azureml.git.dirty\\\": \\\"True\\\",\\n    \\\"AzureML.DerivedImageName\\\": \\\"azureml/azureml_7842fd2c5e99a43f1cca1341b66a0ecb\\\",\\n    \\\"ProcessInfoFile\\\": \\\"azureml-logs/process_info.json\\\",\\n    \\\"ProcessStatusFile\\\": \\\"azureml-logs/process_status.json\\\"\\n  },\\n  \\\"scriptName\\\": \\\"train_validate.py\\\",\\n  \\\"target\\\": \\\"cpu-cluster\\\",\\n  \\\"uniqueChildRunComputeTargets\\\": [],\\n  \\\"tags\\\": {\\n    \\\"_aml_system_ComputeTargetStatus\\\": \\\"{\\\\\\\"AllocationState\\\\\\\":\\\\\\\"steady\\\\\\\",\\\\\\\"PreparingNodeCount\\\\\\\":0,\\\\\\\"RunningNodeCount\\\\\\\":0,\\\\\\\"CurrentNodeCount\\\\\\\":4}\\\"\\n  },\\n  \\\"inputDatasets\\\": [],\\n  \\\"runDefinition\\\": null,\\n  \\\"createdFrom\\\": {\\n    \\\"type\\\": \\\"Notebook\\\",\\n    \\\"locationType\\\": \\\"ArtifactId\\\",\\n    \\\"location\\\": \\\"LocalUpload/tune-lgbm-forecast_1585967817_b60bbc6e/azure_hyperdrive_lightgbm.ipynb\\\"\\n  },\\n  \\\"cancelUri\\\": \\\"https://westcentralus.experiments.azureml.net/execution/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runId/tune-lgbm-forecast_1585967817_b60bbc6e/cancel\\\",\\n  \\\"completeUri\\\": null,\\n  \\\"diagnosticsUri\\\": \\\"https://westcentralus.experiments.azureml.net/execution/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runId/tune-lgbm-forecast_1585967817_b60bbc6e/diagnostics\\\",\\n  \\\"computeRequest\\\": {\\n    \\\"nodeCount\\\": 1\\n  },\\n  \\\"retainForLifetimeOfWorkspace\\\": false,\\n  \\\"queueingInfo\\\": null\\n}\\n2020-04-04 02:38:16,349|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.RunClient.get-async:False|DEBUG|[STOP]\\n2020-04-04 02:38:16,351|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': 'f53e530a-0dc6-4573-af3b-fa3e6b6f9991', 'azureml.git.repository_uri': 'git@github.com:microsoft/forecasting.git', 'mlflow.source.git.repoURL': 'git@github.com:microsoft/forecasting.git', 'azureml.git.branch': 'chenhui/enhancement', 'mlflow.source.git.branch': 'chenhui/enhancement', 'azureml.git.commit': '93da5e12f190bb96bcb7de6bad0280073e43b984', 'mlflow.source.git.commit': '93da5e12f190bb96bcb7de6bad0280073e43b984', 'azureml.git.dirty': 'True', 'AzureML.DerivedImageName': 'azureml/azureml_7842fd2c5e99a43f1cca1341b66a0ecb', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2020-04-04 02:38:16,351|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2020-04-04 02:38:16,351|azureml|WARNING|Could not import azureml.mlflow or azureml.contrib.mlflow mlflow APIs will not run against AzureML services.  Add azureml-mlflow as a conda dependency for the run if this behavior is desired\\n2020-04-04 02:38:16,351|azureml.WorkerPool|DEBUG|[START]\\n2020-04-04 02:38:16,352|azureml.SendRunKillSignal|DEBUG|[START]\\n2020-04-04 02:38:16,352|azureml.RunStatusContext|DEBUG|[START]\\n2020-04-04 02:38:16,352|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunContextManager.RunStatusContext|DEBUG|[START]\\n2020-04-04 02:38:16,352|azureml.WorkingDirectoryCM|DEBUG|[START]\\n2020-04-04 02:38:16,352|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[START]\\n2020-04-04 02:38:16,352|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/chhamlws/2de7e9cc412442c590a3a3c62880b1e9/tune-lgbm-forecast_1585967817_b60bbc6e/mounts/workspaceblobstore/azureml/tune-lgbm-forecast_1585967817_b60bbc6e\\n2020-04-04 02:38:16,353|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2020-04-04 02:38:16,353|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Storing working dir for pyfs as /mnt/batch/tasks/shared/LS_root/jobs/chhamlws/2de7e9cc412442c590a3a3c62880b1e9/tune-lgbm-forecast_1585967817_b60bbc6e/mounts/workspaceblobstore/azureml/tune-lgbm-forecast_1585967817_b60bbc6e\\n2020-04-04 02:38:17,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,195|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,196|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,196|azureml._base_sdk_common.service_discovery|DEBUG|Found history service url in environment variable AZUREML_SERVICE_ENDPOINT, history service url: https://westcentralus.experiments.azureml.net.\\n2020-04-04 02:38:17,201|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:17,202|azureml._run_impl.run_history_facade|DEBUG|Created a static thread pool for RunHistoryFacade class\\n2020-04-04 02:38:17,207|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:17,211|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:17,216|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:17,221|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:38:17,221|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.RunClient.get-async:False|DEBUG|[START]\\n2020-04-04 02:38:17,221|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|Request URL: 'https://westcentralus.experiments.azureml.net/history/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runs/tune-lgbm-forecast_1585967817_b60bbc6e'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|Request method: 'GET'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|Request headers:\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'a7569271-846f-4750-998f-1dad026f9b7d'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|    'request-id': 'a7569271-846f-4750-998f-1dad026f9b7d'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1067-azure-x86_64-with-debian-stretch-sid) msrest/0.6.11 azureml._restclient/core.1.0.85'\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|Request body:\\n2020-04-04 02:38:17,222|msrest.http_logger|DEBUG|None\\n2020-04-04 02:38:17,222|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2020-04-04 02:38:17,222|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2020-04-04 02:38:17,222|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2020-04-04 02:38:17,222|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2020-04-04 02:38:17,265|msrest.http_logger|DEBUG|Response status: 200\\n2020-04-04 02:38:17,265|msrest.http_logger|DEBUG|Response headers:\\n2020-04-04 02:38:17,265|msrest.http_logger|DEBUG|    'Date': 'Sat, 04 Apr 2020 02:38:17 GMT'\\n2020-04-04 02:38:17,265|msrest.http_logger|DEBUG|    'Content-Type': 'application/json; charset=utf-8'\\n2020-04-04 02:38:17,265|msrest.http_logger|DEBUG|    'Transfer-Encoding': 'chunked'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'Vary': 'Accept-Encoding'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'x-ms-client-request-id': 'a7569271-846f-4750-998f-1dad026f9b7d'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'x-request-time': '0.022'\\n2020-04-04 02:38:17,266|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2020-04-04 02:38:17,267|msrest.http_logger|DEBUG|    'Content-Encoding': 'gzip'\\n2020-04-04 02:38:17,267|msrest.http_logger|DEBUG|Response content:\\n2020-04-04 02:38:17,267|msrest.http_logger|DEBUG|{\\n  \\\"runNumber\\\": 302,\\n  \\\"rootRunId\\\": \\\"tune-lgbm-forecast_1585967817_b60bbc6e\\\",\\n  \\\"experimentId\\\": \\\"3199ea18-1505-42f9-9092-777f27df73e5\\\",\\n  \\\"createdUtc\\\": \\\"2020-04-04T02:37:04.210372+00:00\\\",\\n  \\\"createdBy\\\": {\\n    \\\"userObjectId\\\": \\\"8157bc92-2d12-4bc9-9270-bab51c673493\\\",\\n    \\\"userPuId\\\": \\\"10033FFF97A21586\\\",\\n    \\\"userIdp\\\": null,\\n    \\\"userAltSecId\\\": null,\\n    \\\"userIss\\\": \\\"https://sts.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/\\\",\\n    \\\"userTenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\",\\n    \\\"userName\\\": \\\"Chenhui Hu\\\"\\n  },\\n  \\\"userId\\\": \\\"8157bc92-2d12-4bc9-9270-bab51c673493\\\",\\n  \\\"token\\\": null,\\n  \\\"tokenExpiryTimeUtc\\\": null,\\n  \\\"error\\\": null,\\n  \\\"warnings\\\": null,\\n  \\\"revision\\\": 9,\\n  \\\"runUuid\\\": \\\"f667e716-d546-4c3e-83e8-c0db0a607d76\\\",\\n  \\\"parentRunUuid\\\": null,\\n  \\\"rootRunUuid\\\": \\\"f667e716-d546-4c3e-83e8-c0db0a607d76\\\",\\n  \\\"runId\\\": \\\"tune-lgbm-forecast_1585967817_b60bbc6e\\\",\\n  \\\"parentRunId\\\": null,\\n  \\\"status\\\": \\\"Running\\\",\\n  \\\"startTimeUtc\\\": \\\"2020-04-04T02:37:59.4792144+00:00\\\",\\n  \\\"endTimeUtc\\\": null,\\n  \\\"heartbeatEnabled\\\": false,\\n  \\\"options\\\": {\\n    \\\"generateDataContainerIdIfNotSpecified\\\": true\\n  },\\n  \\\"name\\\": null,\\n  \\\"dataContainerId\\\": \\\"dcid.tune-lgbm-forecast_1585967817_b60bbc6e\\\",\\n  \\\"description\\\": null,\\n  \\\"hidden\\\": false,\\n  \\\"runType\\\": \\\"azureml.scriptrun\\\",\\n  \\\"properties\\\": {\\n    \\\"_azureml.ComputeTargetType\\\": \\\"amlcompute\\\",\\n    \\\"ContentSnapshotId\\\": \\\"f53e530a-0dc6-4573-af3b-fa3e6b6f9991\\\",\\n    \\\"azureml.git.repository_uri\\\": \\\"git@github.com:microsoft/forecasting.git\\\",\\n    \\\"mlflow.source.git.repoURL\\\": \\\"git@github.com:microsoft/forecasting.git\\\",\\n    \\\"azureml.git.branch\\\": \\\"chenhui/enhancement\\\",\\n    \\\"mlflow.source.git.branch\\\": \\\"chenhui/enhancement\\\",\\n    \\\"azureml.git.commit\\\": \\\"93da5e12f190bb96bcb7de6bad0280073e43b984\\\",\\n    \\\"mlflow.source.git.commit\\\": \\\"93da5e12f190bb96bcb7de6bad0280073e43b984\\\",\\n    \\\"azureml.git.dirty\\\": \\\"True\\\",\\n    \\\"AzureML.DerivedImageName\\\": \\\"azureml/azureml_7842fd2c5e99a43f1cca1341b66a0ecb\\\",\\n    \\\"ProcessInfoFile\\\": \\\"azureml-logs/process_info.json\\\",\\n    \\\"ProcessStatusFile\\\": \\\"azureml-logs/process_status.json\\\"\\n  },\\n  \\\"scriptName\\\": \\\"train_validate.py\\\",\\n  \\\"target\\\": \\\"cpu-cluster\\\",\\n  \\\"uniqueChildRunComputeTargets\\\": [],\\n  \\\"tags\\\": {\\n    \\\"_aml_system_ComputeTargetStatus\\\": \\\"{\\\\\\\"AllocationState\\\\\\\":\\\\\\\"steady\\\\\\\",\\\\\\\"PreparingNodeCount\\\\\\\":0,\\\\\\\"RunningNodeCount\\\\\\\":0,\\\\\\\"CurrentNodeCount\\\\\\\":4}\\\"\\n  },\\n  \\\"inputDatasets\\\": [],\\n  \\\"runDefinition\\\": null,\\n  \\\"createdFrom\\\": {\\n    \\\"type\\\": \\\"Notebook\\\",\\n    \\\"locationType\\\": \\\"ArtifactId\\\",\\n    \\\"location\\\": \\\"LocalUpload/tune-lgbm-forecast_1585967817_b60bbc6e/azure_hyperdrive_lightgbm.ipynb\\\"\\n  },\\n  \\\"cancelUri\\\": \\\"https://westcentralus.experiments.azureml.net/execution/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runId/tune-lgbm-forecast_1585967817_b60bbc6e/cancel\\\",\\n  \\\"completeUri\\\": null,\\n  \\\"diagnosticsUri\\\": \\\"https://westcentralus.experiments.azureml.net/execution/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runId/tune-lgbm-forecast_1585967817_b60bbc6e/diagnostics\\\",\\n  \\\"computeRequest\\\": {\\n    \\\"nodeCount\\\": 1\\n  },\\n  \\\"retainForLifetimeOfWorkspace\\\": false,\\n  \\\"queueingInfo\\\": null\\n}\\n2020-04-04 02:38:17,269|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.RunClient.get-async:False|DEBUG|[STOP]\\n2020-04-04 02:38:17,270|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e|DEBUG|Constructing run from dto. type: azureml.scriptrun, source: None, props: {'_azureml.ComputeTargetType': 'amlcompute', 'ContentSnapshotId': 'f53e530a-0dc6-4573-af3b-fa3e6b6f9991', 'azureml.git.repository_uri': 'git@github.com:microsoft/forecasting.git', 'mlflow.source.git.repoURL': 'git@github.com:microsoft/forecasting.git', 'azureml.git.branch': 'chenhui/enhancement', 'mlflow.source.git.branch': 'chenhui/enhancement', 'azureml.git.commit': '93da5e12f190bb96bcb7de6bad0280073e43b984', 'mlflow.source.git.commit': '93da5e12f190bb96bcb7de6bad0280073e43b984', 'azureml.git.dirty': 'True', 'AzureML.DerivedImageName': 'azureml/azureml_7842fd2c5e99a43f1cca1341b66a0ecb', 'ProcessInfoFile': 'azureml-logs/process_info.json', 'ProcessStatusFile': 'azureml-logs/process_status.json'}\\n2020-04-04 02:38:17,270|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunContextManager|DEBUG|Valid logs dir, setting up content loader\\n2020-04-04 02:38:46,240|azureml.core.authentication|DEBUG|Time to expire 1814297.75921 seconds\\n2020-04-04 02:39:02,511|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2020-04-04 02:39:02,514|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2020-04-04 02:39:02,526|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2020-04-04 02:39:02,826|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Calling pyfs\\n2020-04-04 02:39:02,826|azureml.history._tracking.PythonWorkingDirectory|INFO|Current working dir: /mnt/batch/tasks/shared/LS_root/jobs/chhamlws/2de7e9cc412442c590a3a3c62880b1e9/tune-lgbm-forecast_1585967817_b60bbc6e/mounts/workspaceblobstore/azureml/tune-lgbm-forecast_1585967817_b60bbc6e\\n2020-04-04 02:39:02,826|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|Reverting working dir from /mnt/batch/tasks/shared/LS_root/jobs/chhamlws/2de7e9cc412442c590a3a3c62880b1e9/tune-lgbm-forecast_1585967817_b60bbc6e/mounts/workspaceblobstore/azureml/tune-lgbm-forecast_1585967817_b60bbc6e to /mnt/batch/tasks/shared/LS_root/jobs/chhamlws/2de7e9cc412442c590a3a3c62880b1e9/tune-lgbm-forecast_1585967817_b60bbc6e/mounts/workspaceblobstore/azureml/tune-lgbm-forecast_1585967817_b60bbc6e\\n2020-04-04 02:39:02,826|azureml.history._tracking.PythonWorkingDirectory|INFO|Working dir is already updated /mnt/batch/tasks/shared/LS_root/jobs/chhamlws/2de7e9cc412442c590a3a3c62880b1e9/tune-lgbm-forecast_1585967817_b60bbc6e/mounts/workspaceblobstore/azureml/tune-lgbm-forecast_1585967817_b60bbc6e\\n2020-04-04 02:39:02,826|azureml.history._tracking.PythonWorkingDirectory.workingdir|DEBUG|[STOP]\\n2020-04-04 02:39:02,826|azureml.WorkingDirectoryCM|DEBUG|[STOP]\\n2020-04-04 02:39:02,826|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e|INFO|complete is not setting status for submitted runs.\\n2020-04-04 02:39:02,826|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2020-04-04 02:39:02,826|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient|DEBUG|Overrides: Max batch size: 50, batch cushion: 5, Interval: 1.\\n2020-04-04 02:39:02,827|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.PostMetricsBatchDaemon|DEBUG|Starting daemon and triggering first instance\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient|DEBUG|Used <class 'azureml._common.async_utils.batch_task_queue.BatchTaskQueue'> for use_batch=True.\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300 is different from task queue timeout 120, using flush timeout\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300 seconds on tasks: [].\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2020-04-04 02:39:02,829|azureml.RunStatusContext|DEBUG|[STOP]\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [].\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2020-04-04 02:39:02,829|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-04-04 02:39:02,830|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2020-04-04 02:39:02,830|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[START]\\n2020-04-04 02:39:02,830|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Start]\\n2020-04-04 02:39:02,830|azureml.BatchTaskQueueAdd_1_Batches.WorkerPool|DEBUG|submitting future: _handle_batch\\n2020-04-04 02:39:02,831|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Batch size 1.\\n2020-04-04 02:39:02,831|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch|DEBUG|Using basic handler - no exception handling\\n2020-04-04 02:39:02,831|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Adding task 0__handle_batch to queue of approximate size: 0\\n2020-04-04 02:39:02,831|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|[Stop] - waiting default timeout\\n2020-04-04 02:39:02,831|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[START]\\n2020-04-04 02:39:02,831|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Overriding default flush timeout from None to 120\\n2020-04-04 02:39:02,831|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|Waiting 120 seconds on tasks: [AsyncTask(0__handle_batch)].\\n2020-04-04 02:39:02,831|azureml._restclient.clientbase.WorkerPool|DEBUG|submitting future: _log_batch\\n2020-04-04 02:39:02,832|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[START]\\n2020-04-04 02:39:02,832|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch|DEBUG|Using basic handler - no exception handling\\n2020-04-04 02:39:02,833|msrest.service_client|DEBUG|Accept header absent and forced to application/json\\n2020-04-04 02:39:02,833|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|Adding task 0__log_batch to queue of approximate size: 0\\n2020-04-04 02:39:02,833|msrest.universal_http.requests|DEBUG|Configuring retry: max_retries=3, backoff_factor=0.8, max_backoff=90\\n2020-04-04 02:39:02,833|msrest.http_logger|DEBUG|Request URL: 'https://westcentralus.experiments.azureml.net/history/v1.0/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast/runs/tune-lgbm-forecast_1585967817_b60bbc6e/batch/metrics'\\n2020-04-04 02:39:02,833|msrest.http_logger|DEBUG|Request method: 'POST'\\n2020-04-04 02:39:02,833|msrest.http_logger|DEBUG|Request headers:\\n2020-04-04 02:39:02,833|msrest.http_logger|DEBUG|    'Accept': 'application/json'\\n2020-04-04 02:39:02,833|msrest.http_logger|DEBUG|    'Content-Type': 'application/json-patch+json; charset=utf-8'\\n2020-04-04 02:39:02,834|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '856debe6-841e-4e77-ba4a-86273ea493ef'\\n2020-04-04 02:39:02,834|msrest.http_logger|DEBUG|    'request-id': '856debe6-841e-4e77-ba4a-86273ea493ef'\\n2020-04-04 02:39:02,834|msrest.http_logger|DEBUG|    'Content-Length': '341'\\n2020-04-04 02:39:02,834|msrest.http_logger|DEBUG|    'User-Agent': 'python/3.6.2 (Linux-4.15.0-1067-azure-x86_64-with-debian-stretch-sid) msrest/0.6.11 azureml._restclient/core.1.0.85 sdk_run'\\n2020-04-04 02:39:02,834|msrest.http_logger|DEBUG|Request body:\\n2020-04-04 02:39:02,834|msrest.http_logger|DEBUG|{\\\"values\\\": [{\\\"metricId\\\": \\\"380b71e2-fa96-4cc1-a6c4-e78f4cd5d84e\\\", \\\"metricType\\\": \\\"azureml.v1.scalar\\\", \\\"createdUtc\\\": \\\"2020-04-04T02:39:02.511623Z\\\", \\\"name\\\": \\\"MAPE\\\", \\\"description\\\": \\\"\\\", \\\"numCells\\\": 1, \\\"cells\\\": [{\\\"MAPE\\\": 66.59144474679267}], \\\"schema\\\": {\\\"numProperties\\\": 1, \\\"properties\\\": [{\\\"propertyId\\\": \\\"MAPE\\\", \\\"name\\\": \\\"MAPE\\\", \\\"type\\\": \\\"float\\\"}]}}]}\\n2020-04-04 02:39:02,834|msrest.universal_http|DEBUG|Configuring redirects: allow=True, max=30\\n2020-04-04 02:39:02,834|msrest.universal_http|DEBUG|Configuring request: timeout=100, verify=True, cert=None\\n2020-04-04 02:39:02,834|msrest.universal_http|DEBUG|Configuring proxies: ''\\n2020-04-04 02:39:02,834|msrest.universal_http|DEBUG|Evaluate proxies against ENV settings: True\\n2020-04-04 02:39:03,000|msrest.http_logger|DEBUG|Response status: 200\\n2020-04-04 02:39:03,000|msrest.http_logger|DEBUG|Response headers:\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'Date': 'Sat, 04 Apr 2020 02:39:02 GMT'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'Content-Length': '0'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'Connection': 'keep-alive'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'x-ms-client-request-id': '856debe6-841e-4e77-ba4a-86273ea493ef'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'x-ms-client-session-id': ''\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'x-request-time': '0.147'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|    'X-Content-Type-Options': 'nosniff'\\n2020-04-04 02:39:03,001|msrest.http_logger|DEBUG|Response content:\\n2020-04-04 02:39:03,002|msrest.http_logger|DEBUG|\\n2020-04-04 02:39:03,003|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.post_batch-async:False|DEBUG|[STOP]\\n2020-04-04 02:39:03,082|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[START]\\n2020-04-04 02:39:03,082|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|Awaiter is BatchTaskQueueAdd_1_Batches\\n2020-04-04 02:39:03,082|azureml.BatchTaskQueueAdd_1_Batches.0__handle_batch.WaitingTask|DEBUG|[STOP]\\n2020-04-04 02:39:03,083|azureml.BatchTaskQueueAdd_1_Batches|DEBUG|Waiting on task: 0__handle_batch.\\n1 tasks left. Current duration of flush 6.341934204101562e-05 seconds.\\n\\n2020-04-04 02:39:03,083|azureml.BatchTaskQueueAdd_1_Batches.WaitFlushSource:BatchTaskQueueAdd_1_Batches|DEBUG|[STOP]\\n2020-04-04 02:39:03,083|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[START]\\n2020-04-04 02:39:03,083|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|flush timeout 300.0 is different from task queue timeout 120, using flush timeout\\n2020-04-04 02:39:03,083|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|Waiting 300.0 seconds on tasks: [AsyncTask(0__log_batch)].\\n2020-04-04 02:39:03,084|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch.WaitingTask|DEBUG|[START]\\n2020-04-04 02:39:03,084|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch.WaitingTask|DEBUG|Awaiter is PostMetricsBatch\\n2020-04-04 02:39:03,084|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.0__log_batch.WaitingTask|DEBUG|[STOP]\\n2020-04-04 02:39:03,084|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch|DEBUG|\\n2020-04-04 02:39:03,084|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.PostMetricsBatch.WaitFlushSource:MetricsClient|DEBUG|[STOP]\\n2020-04-04 02:39:03,084|azureml._SubmittedRun#tune-lgbm-forecast_1585967817_b60bbc6e.RunHistoryFacade.MetricsClient.FlushingMetricsClient|DEBUG|[STOP]\\n2020-04-04 02:39:03,084|azureml.SendRunKillSignal|DEBUG|[STOP]\\n2020-04-04 02:39:03,085|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[START]\\n2020-04-04 02:39:03,085|azureml.HistoryTrackingWorkerPool.WorkerPoolShutdown|DEBUG|[STOP]\\n2020-04-04 02:39:03,085|azureml.WorkerPool|DEBUG|[STOP]\\n\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": true, \"log_level\": \"INFO\", \"sdk_version\": \"1.0.85\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "RunDetails(run_remote).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the validation metric after the job finishes. The validation metric should be the same as the one we obtained when the script was ran locally. For more details of the job, you can execute `run_remote.get_details()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'MAPE': 66.59144474679267}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get metric value after the job finishes\n",
    "run_remote.wait_for_completion()\n",
    "run_remote.get_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='tune-hyperparameters-using-hyperdrive'></a>\n",
    "## Tune Hyperparameters using HyperDrive\n",
    "\n",
    "Now we are ready to tune hyperparameters of the LightGBM forecast model by launching multiple runs on the cluster. In the following cell, we define the configuration of a HyperDrive job that does a parallel search of the hyperparameter space using a Bayesian sampling method. HyperDrive also supports random sampling of the parameter space.\n",
    "\n",
    "It is recommended that the maximum number of runs should be greater than or equal to 20 times the number of hyperparameters being tuned, for best results with Bayesian sampling. Specifically, it should be no less than 180 in the following case as we have 9 hyperparameters to tune. Nevertheless, we find that even with a very small amount of runs Bayesian search can achieve decent performance. Thus, the maximum number of child runs of HyperDrive `max_total_runs` is set as `20` to reduce the running time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "For best results with Bayesian Sampling we recommend using a maximum number of runs greater than or equal to 20 times the number of hyperparameters being tuned. Current value for max_total_runs:20. Recommendend value:180.\n",
      "WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', OSError(\"(104, 'ECONNRESET')\",))': /azureml/LocalUpload/HD_dcf456bc-203d-42aa-892e-23fe23133bb5/azure_hyperdrive_lightgbm.ipynb?sv=2019-02-02&sr=b&sig=6vSboBoAni5Utk6oKUInYOz1eN258%2BmZiHVvvQQupT4%3D&st=2020-04-04T02%3A29%3A15Z&se=2020-04-05T02%3A39%3A15Z&sp=rcw&comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQXdNREF3TURBd01EQSUzRA%3D%3D&timeout=30\n"
     ]
    }
   ],
   "source": [
    "# Increase this value if you want to achieve better performance\n",
    "max_total_runs = 20\n",
    "script_params = {\"--data-folder\": ds_data.as_mount()}\n",
    "est = Estimator(\n",
    "    source_directory=script_folder,\n",
    "    script_params=script_params,\n",
    "    compute_target=compute_target,\n",
    "    use_docker=True,\n",
    "    entry_script=train_script_name,\n",
    "    environment_definition=env,\n",
    ")\n",
    "\n",
    "# Specify hyperparameter space\n",
    "ps = BayesianParameterSampling(\n",
    "    {\n",
    "        \"--num-leaves\": quniform(8, 128, 1),\n",
    "        \"--min-data-in-leaf\": quniform(20, 500, 10),\n",
    "        \"--learning-rate\": choice(\n",
    "            1e-4, 1e-3, 5e-3, 1e-2, 1.5e-2, 2e-2, 3e-2, 5e-2, 1e-1\n",
    "        ),\n",
    "        \"--feature-fraction\": uniform(0.2, 1),\n",
    "        \"--bagging-fraction\": uniform(0.1, 1),\n",
    "        \"--bagging-freq\": quniform(1, 20, 1),\n",
    "        \"--max-rounds\": quniform(50, 2000, 10),\n",
    "        \"--max-lag\": quniform(3, 40, 1),\n",
    "        \"--window-size\": quniform(3, 40, 1),\n",
    "    }\n",
    ")\n",
    "\n",
    "# HyperDrive job configuration\n",
    "htc = HyperDriveConfig(\n",
    "    estimator=est,\n",
    "    hyperparameter_sampling=ps,\n",
    "    primary_metric_name=\"MAPE\",\n",
    "    primary_metric_goal=PrimaryMetricGoal.MINIMIZE,\n",
    "    max_total_runs=max_total_runs,\n",
    "    max_concurrent_runs=4,\n",
    ")\n",
    "\n",
    "htr = exp.submit(config=htc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the job finishes, you should see outputs from the AzureML widgets similar to the following. Note that you can rerun  `RunDetails(htr).show()` after the job finishes to get the updated results on the dashboard in case it is not automatically refreshed.\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/20047467/76152567-e31a5e00-608e-11ea-90a8-3fdfeafeeb92.png\" width=\"900\" height=\"500\">\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/20047467/76152586-270d6300-608f-11ea-8f83-07fa7a8528f2.png\" width=\"900\" height=\"400\">\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/20047467/76152600-46a48b80-608f-11ea-90d2-297aabd7376a.png\" width=\"900\" height=\"300\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "212902d6c8da419c94ac9df6086678c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_HyperDriveWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': True, 'log_level': 'INFO',…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Running\", \"workbench_run_details_uri\": \"https://ml.azure.com/experiments/tune-lgbm-forecast/runs/HD_dcf456bc-203d-42aa-892e-23fe23133bb5?wsid=/subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourcegroups/chhamlwsrg/workspaces/chhamlws\", \"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5\", \"run_properties\": {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5\", \"created_utc\": \"2020-04-04T02:39:10.910797Z\", \"properties\": {\"primary_metric_config\": \"{\\\"name\\\": \\\"MAPE\\\", \\\"goal\\\": \\\"minimize\\\"}\", \"resume_from\": \"null\", \"runTemplate\": \"HyperDrive\", \"azureml.runsource\": \"hyperdrive\", \"platform\": \"AML\", \"ContentSnapshotId\": \"f53e530a-0dc6-4573-af3b-fa3e6b6f9991\"}, \"tags\": {\"max_concurrent_jobs\": \"4\", \"max_total_jobs\": \"20\", \"max_duration_minutes\": \"10080\", \"policy_config\": \"{\\\"name\\\": \\\"DEFAULT\\\"}\", \"generator_config\": \"{\\\"name\\\": \\\"BAYESIANOPTIMIZATION\\\", \\\"parameter_space\\\": {\\\"--num-leaves\\\": [\\\"quniform\\\", [8, 128, 1]], \\\"--min-data-in-leaf\\\": [\\\"quniform\\\", [20, 500, 10]], \\\"--learning-rate\\\": [\\\"choice\\\", [[0.0001, 0.001, 0.005, 0.01, 0.015, 0.02, 0.03, 0.05, 0.1]]], \\\"--feature-fraction\\\": [\\\"uniform\\\", [0.2, 1]], \\\"--bagging-fraction\\\": [\\\"uniform\\\", [0.1, 1]], \\\"--bagging-freq\\\": [\\\"quniform\\\", [1, 20, 1]], \\\"--max-rounds\\\": [\\\"quniform\\\", [50, 2000, 10]], \\\"--max-lag\\\": [\\\"quniform\\\", [3, 40, 1]], \\\"--window-size\\\": [\\\"quniform\\\", [3, 40, 1]]}}\", \"primary_metric_config\": \"{\\\"name\\\": \\\"MAPE\\\", \\\"goal\\\": \\\"minimize\\\"}\", \"platform_config\": \"{\\\"ServiceAddress\\\": \\\"https://westcentralus.experiments.azureml.net\\\", \\\"ServiceArmScope\\\": \\\"subscriptions/9086b59a-02d7-4687-b3fd-e39fa5e0fd9b/resourceGroups/chhamlwsrg/providers/Microsoft.MachineLearningServices/workspaces/chhamlws/experiments/tune-lgbm-forecast\\\", \\\"SubscriptionId\\\": \\\"9086b59a-02d7-4687-b3fd-e39fa5e0fd9b\\\", \\\"ResourceGroupName\\\": \\\"chhamlwsrg\\\", \\\"WorkspaceName\\\": \\\"chhamlws\\\", \\\"ExperimentName\\\": \\\"tune-lgbm-forecast\\\", \\\"Definition\\\": {\\\"Overrides\\\": {\\\"script\\\": \\\"train_validate.py\\\", \\\"arguments\\\": [\\\"--data-folder\\\", \\\"$AZUREML_DATAREFERENCE_00b162eda80e4ee9a050e69c71311ecc\\\"], \\\"target\\\": \\\"cpu-cluster\\\", \\\"framework\\\": \\\"Python\\\", \\\"communicator\\\": \\\"None\\\", \\\"maxRunDurationSeconds\\\": null, \\\"nodeCount\\\": 1, \\\"environment\\\": {\\\"name\\\": null, \\\"version\\\": null, \\\"environmentVariables\\\": {\\\"EXAMPLE_ENV_VAR\\\": \\\"EXAMPLE_VALUE\\\"}, \\\"python\\\": {\\\"userManagedDependencies\\\": false, \\\"interpreterPath\\\": \\\"python\\\", \\\"condaDependenciesFile\\\": null, \\\"baseCondaEnvironment\\\": null, \\\"condaDependencies\\\": {\\\"name\\\": \\\"project_environment\\\", \\\"dependencies\\\": [\\\"python=3.6.2\\\", {\\\"pip\\\": [\\\"azureml-defaults\\\"]}, \\\"pandas\\\", \\\"numpy\\\", \\\"scipy\\\", \\\"scikit-learn\\\", \\\"lightgbm\\\", \\\"joblib\\\"], \\\"channels\\\": [\\\"conda-forge\\\"]}}, \\\"docker\\\": {\\\"enabled\\\": true, \\\"baseImage\\\": \\\"mcr.microsoft.com/azureml/base:intelmpi2018.3-ubuntu16.04\\\", \\\"baseDockerfile\\\": null, \\\"sharedVolumes\\\": true, \\\"shmSize\\\": \\\"2g\\\", \\\"arguments\\\": [], \\\"baseImageRegistry\\\": {\\\"address\\\": null, \\\"username\\\": null, \\\"password\\\": null}}, \\\"spark\\\": {\\\"repositories\\\": [], \\\"packages\\\": [], \\\"precachePackages\\\": true}, \\\"databricks\\\": {\\\"mavenLibraries\\\": [], \\\"pypiLibraries\\\": [], \\\"rcranLibraries\\\": [], \\\"jarLibraries\\\": [], \\\"eggLibraries\\\": []}, \\\"inferencingStackVersion\\\": null}, \\\"history\\\": {\\\"outputCollection\\\": true, \\\"snapshotProject\\\": true, \\\"directoriesToWatch\\\": [\\\"logs\\\"]}, \\\"spark\\\": {\\\"configuration\\\": {\\\"spark.app.name\\\": \\\"Azure ML Experiment\\\", \\\"spark.yarn.maxAppAttempts\\\": 1}}, \\\"hdi\\\": {\\\"yarnDeployMode\\\": \\\"cluster\\\"}, \\\"tensorflow\\\": {\\\"workerCount\\\": 1, \\\"parameterServerCount\\\": 1}, \\\"mpi\\\": {\\\"processCountPerNode\\\": 1}, \\\"dataReferences\\\": {\\\"00b162eda80e4ee9a050e69c71311ecc\\\": {\\\"dataStoreName\\\": \\\"workspaceblobstore\\\", \\\"pathOnDataStore\\\": \\\"data\\\", \\\"mode\\\": \\\"mount\\\", \\\"overwrite\\\": false, \\\"pathOnCompute\\\": null}}, \\\"data\\\": {}, \\\"sourceDirectoryDataStore\\\": null, \\\"amlcompute\\\": {\\\"vmSize\\\": null, \\\"vmPriority\\\": null, \\\"retainCluster\\\": false, \\\"name\\\": null, \\\"clusterMaxNodeCount\\\": 1}}, \\\"TargetDetails\\\": null, \\\"SnapshotId\\\": \\\"f53e530a-0dc6-4573-af3b-fa3e6b6f9991\\\", \\\"TelemetryValues\\\": {\\\"amlClientType\\\": \\\"azureml-sdk-train\\\", \\\"amlClientModule\\\": \\\"azureml.train.hyperdrive._search\\\", \\\"amlClientFunction\\\": \\\"search\\\", \\\"tenantId\\\": \\\"72f988bf-86f1-41af-91ab-2d7cd011db47\\\", \\\"amlClientRequestId\\\": \\\"d33b8221-6a56-46a2-872d-59fb5518c495\\\", \\\"amlClientSessionId\\\": \\\"ec946709-8415-42aa-8488-c338bfd87e21\\\", \\\"subscriptionId\\\": \\\"9086b59a-02d7-4687-b3fd-e39fa5e0fd9b\\\", \\\"estimator\\\": \\\"Estimator\\\", \\\"samplingMethod\\\": \\\"BayesianOptimization\\\", \\\"terminationPolicy\\\": \\\"Default\\\", \\\"primaryMetricGoal\\\": \\\"minimize\\\", \\\"maxTotalRuns\\\": 20, \\\"maxConcurrentRuns\\\": 4, \\\"maxDurationMinutes\\\": 10080, \\\"computeTarget\\\": \\\"AmlCompute\\\", \\\"vmSize\\\": null}}}\", \"resume_child_runs\": \"null\", \"all_jobs_generated\": \"false\", \"cancellation_requested\": \"false\", \"progress_metadata_evaluation_timestamp\": \"\\\"2020-04-04T02:39:11.746519\\\"\", \"progress_metadata_digest\": \"\\\"b231935578bf29c38a1a8fe1476801df373a5b0316a3e88096fe736ec786869c\\\"\", \"progress_metadata_active_timestamp\": \"\\\"2020-04-04T02:39:11.746519\\\"\", \"environment_preparation_status\": \"PREPARED\", \"prepare_run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_preparation\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_0\": \"{\\\"--num-leaves\\\": 43, \\\"--min-data-in-leaf\\\": 40, \\\"--learning-rate\\\": 0.005, \\\"--feature-fraction\\\": 0.522105759494254, \\\"--bagging-fraction\\\": 0.44597750968785044, \\\"--bagging-freq\\\": 16, \\\"--max-rounds\\\": 1690, \\\"--max-lag\\\": 4, \\\"--window-size\\\": 25}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_1\": \"{\\\"--num-leaves\\\": 73, \\\"--min-data-in-leaf\\\": 60, \\\"--learning-rate\\\": 0.1, \\\"--feature-fraction\\\": 0.4059469819299132, \\\"--bagging-fraction\\\": 0.6418938497070497, \\\"--bagging-freq\\\": 16, \\\"--max-rounds\\\": 1620, \\\"--max-lag\\\": 13, \\\"--window-size\\\": 19}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_2\": \"{\\\"--num-leaves\\\": 108, \\\"--min-data-in-leaf\\\": 120, \\\"--learning-rate\\\": 0.05, \\\"--feature-fraction\\\": 0.21188946746489892, \\\"--bagging-fraction\\\": 0.604654482720839, \\\"--bagging-freq\\\": 4, \\\"--max-rounds\\\": 1240, \\\"--max-lag\\\": 6, \\\"--window-size\\\": 12}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_3\": \"{\\\"--num-leaves\\\": 9, \\\"--min-data-in-leaf\\\": 80, \\\"--learning-rate\\\": 0.015, \\\"--feature-fraction\\\": 0.8524889664424598, \\\"--bagging-fraction\\\": 0.3712070083444936, \\\"--bagging-freq\\\": 9, \\\"--max-rounds\\\": 110, \\\"--max-lag\\\": 15, \\\"--window-size\\\": 40}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_4\": \"{\\\"--num-leaves\\\": 11, \\\"--min-data-in-leaf\\\": 410, \\\"--learning-rate\\\": 0.005, \\\"--feature-fraction\\\": 0.237131500290646, \\\"--bagging-fraction\\\": 0.46710483442198847, \\\"--bagging-freq\\\": 2, \\\"--max-rounds\\\": 1040, \\\"--max-lag\\\": 21, \\\"--window-size\\\": 8}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_5\": \"{\\\"--num-leaves\\\": 46, \\\"--min-data-in-leaf\\\": 190, \\\"--learning-rate\\\": 0.0001, \\\"--feature-fraction\\\": 0.34701496623400496, \\\"--bagging-fraction\\\": 0.9120679296928682, \\\"--bagging-freq\\\": 6, \\\"--max-rounds\\\": 580, \\\"--max-lag\\\": 30, \\\"--window-size\\\": 23}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_6\": \"{\\\"--num-leaves\\\": 107, \\\"--min-data-in-leaf\\\": 250, \\\"--learning-rate\\\": 0.1, \\\"--feature-fraction\\\": 0.9161386628837784, \\\"--bagging-fraction\\\": 0.2417820471108519, \\\"--bagging-freq\\\": 9, \\\"--max-rounds\\\": 120, \\\"--max-lag\\\": 7, \\\"--window-size\\\": 23}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_7\": \"{\\\"--num-leaves\\\": 97, \\\"--min-data-in-leaf\\\": 180, \\\"--learning-rate\\\": 0.05, \\\"--feature-fraction\\\": 0.6031270996331206, \\\"--bagging-fraction\\\": 0.918316833768419, \\\"--bagging-freq\\\": 16, \\\"--max-rounds\\\": 1410, \\\"--max-lag\\\": 25, \\\"--window-size\\\": 29}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_8\": \"{\\\"--num-leaves\\\": 121, \\\"--min-data-in-leaf\\\": 270, \\\"--learning-rate\\\": 0.001, \\\"--feature-fraction\\\": 0.7262272957355567, \\\"--bagging-fraction\\\": 0.49925280404591377, \\\"--bagging-freq\\\": 8, \\\"--max-rounds\\\": 860, \\\"--max-lag\\\": 20, \\\"--window-size\\\": 26}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_9\": \"{\\\"--num-leaves\\\": 121, \\\"--min-data-in-leaf\\\": 310, \\\"--learning-rate\\\": 0.05, \\\"--feature-fraction\\\": 0.6457203702824046, \\\"--bagging-fraction\\\": 0.7834345415012108, \\\"--bagging-freq\\\": 15, \\\"--max-rounds\\\": 1340, \\\"--max-lag\\\": 24, \\\"--window-size\\\": 8}\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_10\": \"{\\\"--num-leaves\\\": 117, \\\"--min-data-in-leaf\\\": 290, \\\"--learning-rate\\\": 0.01, \\\"--feature-fraction\\\": 0.8910572969746846, \\\"--bagging-fraction\\\": 0.7682158168093952, \\\"--bagging-freq\\\": 1, \\\"--max-rounds\\\": 1470, \\\"--max-lag\\\": 33, \\\"--window-size\\\": 7}\"}, \"end_time_utc\": null, \"status\": \"Running\", \"log_files\": {\"azureml-logs/hyperdrive.txt\": \"https://chhamlws4931040064.blob.core.windows.net/azureml/ExperimentRun/dcid.HD_dcf456bc-203d-42aa-892e-23fe23133bb5/azureml-logs/hyperdrive.txt?sv=2019-02-02&sr=b&sig=NG%2FsCUOcXOTez806c1GJln0Ebbb2oTEtNtO9gMsR0vY%3D&st=2020-04-04T02%3A35%3A36Z&se=2020-04-04T10%3A45%3A36Z&sp=r\"}, \"log_groups\": [[\"azureml-logs/hyperdrive.txt\"]], \"run_duration\": \"0:06:25\", \"hyper_parameters\": {\"--num-leaves\": [\"quniform\", [8, 128, 1]], \"--min-data-in-leaf\": [\"quniform\", [20, 500, 10]], \"--learning-rate\": [\"choice\", [[0.0001, 0.001, 0.005, 0.01, 0.015, 0.02, 0.03, 0.05, 0.1]]], \"--feature-fraction\": [\"uniform\", [0.2, 1]], \"--bagging-fraction\": [\"uniform\", [0.1, 1]], \"--bagging-freq\": [\"quniform\", [1, 20, 1]], \"--max-rounds\": [\"quniform\", [50, 2000, 10]], \"--max-lag\": [\"quniform\", [3, 40, 1]], \"--window-size\": [\"quniform\", [3, 40, 1]]}}, \"child_runs\": [{\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_1\", \"run_number\": 305, \"metric\": 35.66304484, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:40:19.067757Z\", \"end_time\": \"2020-04-04T02:42:42.663093Z\", \"created_time\": \"2020-04-04T02:39:43.617669Z\", \"created_time_dt\": \"2020-04-04T02:39:43.617669Z\", \"duration\": \"0:02:59\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 73, \"param_--min-data-in-leaf\": 60, \"param_--learning-rate\": 0.1, \"param_--feature-fraction\": 0.4059469819299132, \"param_--bagging-fraction\": 0.6418938497070497, \"param_--bagging-freq\": 16, \"param_--max-rounds\": 1620, \"param_--max-lag\": 13, \"param_--window-size\": 19, \"best_metric\": 35.66304484}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_2\", \"run_number\": 306, \"metric\": 31.52092181, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:40:17.682969Z\", \"end_time\": \"2020-04-04T02:42:19.718203Z\", \"created_time\": \"2020-04-04T02:39:44.273565Z\", \"created_time_dt\": \"2020-04-04T02:39:44.273565Z\", \"duration\": \"0:02:35\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 108, \"param_--min-data-in-leaf\": 120, \"param_--learning-rate\": 0.05, \"param_--feature-fraction\": 0.21188946746489892, \"param_--bagging-fraction\": 0.604654482720839, \"param_--bagging-freq\": 4, \"param_--max-rounds\": 1240, \"param_--max-lag\": 6, \"param_--window-size\": 12, \"best_metric\": 31.52092181}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_0\", \"run_number\": 307, \"metric\": 40.01020292, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:40:17.827364Z\", \"end_time\": \"2020-04-04T02:43:10.136186Z\", \"created_time\": \"2020-04-04T02:39:44.282105Z\", \"created_time_dt\": \"2020-04-04T02:39:44.282105Z\", \"duration\": \"0:03:25\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 43, \"param_--min-data-in-leaf\": 40, \"param_--learning-rate\": 0.005, \"param_--feature-fraction\": 0.522105759494254, \"param_--bagging-fraction\": 0.44597750968785044, \"param_--bagging-freq\": 16, \"param_--max-rounds\": 1690, \"param_--max-lag\": 4, \"param_--window-size\": 25, \"best_metric\": 31.52092181}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_3\", \"run_number\": 308, \"metric\": 54.46713976, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:40:19.717567Z\", \"end_time\": \"2020-04-04T02:41:16.371071Z\", \"created_time\": \"2020-04-04T02:39:44.984587Z\", \"created_time_dt\": \"2020-04-04T02:39:44.984587Z\", \"duration\": \"0:01:31\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 9, \"param_--min-data-in-leaf\": 80, \"param_--learning-rate\": 0.015, \"param_--feature-fraction\": 0.8524889664424598, \"param_--bagging-fraction\": 0.3712070083444936, \"param_--bagging-freq\": 9, \"param_--max-rounds\": 110, \"param_--max-lag\": 15, \"param_--window-size\": 40, \"best_metric\": 31.52092181}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_4\", \"run_number\": 309, \"metric\": 43.36610118, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:42:20.25632Z\", \"end_time\": \"2020-04-04T02:43:55.750217Z\", \"created_time\": \"2020-04-04T02:41:47.722315Z\", \"created_time_dt\": \"2020-04-04T02:41:47.722315Z\", \"duration\": \"0:02:08\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 11, \"param_--min-data-in-leaf\": 410, \"param_--learning-rate\": 0.005, \"param_--feature-fraction\": 0.237131500290646, \"param_--bagging-fraction\": 0.46710483442198847, \"param_--bagging-freq\": 2, \"param_--max-rounds\": 1040, \"param_--max-lag\": 21, \"param_--window-size\": 8, \"best_metric\": 31.52092181}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_6\", \"run_number\": 310, \"metric\": 36.42988546, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:43:28.431416Z\", \"end_time\": \"2020-04-04T02:44:26.636679Z\", \"created_time\": \"2020-04-04T02:42:49.776024Z\", \"created_time_dt\": \"2020-04-04T02:42:49.776024Z\", \"duration\": \"0:01:36\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 107, \"param_--min-data-in-leaf\": 250, \"param_--learning-rate\": 0.1, \"param_--feature-fraction\": 0.9161386628837784, \"param_--bagging-fraction\": 0.2417820471108519, \"param_--bagging-freq\": 9, \"param_--max-rounds\": 120, \"param_--max-lag\": 7, \"param_--window-size\": 23, \"best_metric\": 31.52092181}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_5\", \"run_number\": 311, \"metric\": 74.39909023, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:43:31.086471Z\", \"end_time\": \"2020-04-04T02:45:18.469123Z\", \"created_time\": \"2020-04-04T02:42:51.128442Z\", \"created_time_dt\": \"2020-04-04T02:42:51.128442Z\", \"duration\": \"0:02:27\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 46, \"param_--min-data-in-leaf\": 190, \"param_--learning-rate\": 0.0001, \"param_--feature-fraction\": 0.34701496623400496, \"param_--bagging-fraction\": 0.9120679296928682, \"param_--bagging-freq\": 6, \"param_--max-rounds\": 580, \"param_--max-lag\": 30, \"param_--window-size\": 23, \"best_metric\": 31.52092181}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_7\", \"run_number\": 312, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:43:59.44461Z\", \"end_time\": \"\", \"created_time\": \"2020-04-04T02:43:24.1603Z\", \"created_time_dt\": \"2020-04-04T02:43:24.1603Z\", \"duration\": \"0:02:12\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 97, \"param_--min-data-in-leaf\": 180, \"param_--learning-rate\": 0.05, \"param_--feature-fraction\": 0.6031270996331206, \"param_--bagging-fraction\": 0.918316833768419, \"param_--bagging-freq\": 16, \"param_--max-rounds\": 1410, \"param_--max-lag\": 25, \"param_--window-size\": 29, \"best_metric\": null}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_8\", \"run_number\": 313, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:45:11.897611Z\", \"end_time\": \"\", \"created_time\": \"2020-04-04T02:44:26.603338Z\", \"created_time_dt\": \"2020-04-04T02:44:26.603338Z\", \"duration\": \"0:01:09\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 121, \"param_--min-data-in-leaf\": 270, \"param_--learning-rate\": 0.001, \"param_--feature-fraction\": 0.7262272957355567, \"param_--bagging-fraction\": 0.49925280404591377, \"param_--bagging-freq\": 8, \"param_--max-rounds\": 860, \"param_--max-lag\": 20, \"param_--window-size\": 26, \"best_metric\": null}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_9\", \"run_number\": 314, \"metric\": null, \"status\": \"Running\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2020-04-04T02:45:35.480471Z\", \"end_time\": \"\", \"created_time\": \"2020-04-04T02:45:02.392458Z\", \"created_time_dt\": \"2020-04-04T02:45:02.392458Z\", \"duration\": \"0:00:34\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 121, \"param_--min-data-in-leaf\": 310, \"param_--learning-rate\": 0.05, \"param_--feature-fraction\": 0.6457203702824046, \"param_--bagging-fraction\": 0.7834345415012108, \"param_--bagging-freq\": 15, \"param_--max-rounds\": 1340, \"param_--max-lag\": 24, \"param_--window-size\": 8, \"best_metric\": null}, {\"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_10\", \"run_number\": 315, \"metric\": null, \"status\": \"Starting\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"\", \"end_time\": \"\", \"created_time\": \"2020-04-04T02:45:33.616722Z\", \"created_time_dt\": \"2020-04-04T02:45:33.616722Z\", \"duration\": \"0:00:02\", \"hyperdrive_id\": \"dcf456bc-203d-42aa-892e-23fe23133bb5\", \"arguments\": null, \"param_--num-leaves\": 117, \"param_--min-data-in-leaf\": 290, \"param_--learning-rate\": 0.01, \"param_--feature-fraction\": 0.8910572969746846, \"param_--bagging-fraction\": 0.7682158168093952, \"param_--bagging-freq\": 1, \"param_--max-rounds\": 1470, \"param_--max-lag\": 33, \"param_--window-size\": 7, \"best_metric\": null}], \"children_metrics\": {\"categories\": [0], \"series\": {\"MAPE\": [{\"categories\": [305, 306, 307, 308, 309, 310, 311], \"mode\": \"markers\", \"name\": \"MAPE\", \"stepped\": false, \"type\": \"scatter\", \"data\": [35.66304484399933, 31.520921807705783, 40.01020292109138, 54.467139761173954, 43.36610118248853, 36.42988545942116, 74.39909022797315]}, {\"categories\": [305, 306, 307, 308, 309, 310, 311], \"mode\": \"lines\", \"name\": \"MAPE_min\", \"stepped\": true, \"type\": \"scatter\", \"data\": [35.66304484399933, 31.520921807705783, 31.520921807705783, 31.520921807705783, 31.520921807705783, 31.520921807705783, 31.520921807705783]}]}, \"metricName\": null, \"primaryMetricName\": \"MAPE\", \"showLegend\": false}, \"run_metrics\": [{\"name\": \"best_child_by_primary_metric\", \"run_id\": \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5\", \"categories\": [0], \"series\": [{\"data\": [{\"metric_name\": [\"MAPE\", \"MAPE\"], \"timestamp\": [\"2020-04-04 02:41:22.286881+00:00\", \"2020-04-04 02:42:27.993375+00:00\"], \"run_id\": [\"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_3\", \"HD_dcf456bc-203d-42aa-892e-23fe23133bb5_2\"], \"metric_value\": [54.467139761173954, 31.520921807705783], \"final\": [false, false]}]}]}], \"run_logs\": \"[2020-04-04T02:39:11.255619][API][INFO]Experiment created\\r\\n[2020-04-04T02:39:12.5213110Z][SCHEDULER][INFO]The execution environment is being prepared. Please be patient as it can take a few minutes.\\r\\n[2020-04-04T02:39:13.556831][GENERATOR][INFO]Trying to sample '4' jobs from the hyperparameter space\\r\\n[2020-04-04T02:39:14.060179][GENERATOR][INFO]Successfully sampled '4' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:39:42.9700580Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_0'\\r\\n[2020-04-04T02:39:42.9722386Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_1'\\r\\n[2020-04-04T02:39:42.9772902Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_2'\\r\\n[2020-04-04T02:39:42.9780717Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_3'\\r\\n[2020-04-04T02:39:42.9693756Z][SCHEDULER][INFO]The execution environment was successfully prepared.\\r\\n[2020-04-04T02:39:43.7589552Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_1'\\r\\n[2020-04-04T02:39:44.3434965Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_2'\\r\\n[2020-04-04T02:39:44.5008781Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_0'\\r\\n[2020-04-04T02:39:45.2071910Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_3'\\r\\n[2020-04-04T02:41:16.872902][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2020-04-04T02:41:17.410855][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:41:46.2064520Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_4'\\r\\n[2020-04-04T02:41:47.8430229Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_4'\\r\\n[2020-04-04T02:42:47.917526][GENERATOR][INFO]Trying to sample '2' jobs from the hyperparameter space\\r\\n[2020-04-04T02:42:48.167748][GENERATOR][INFO]Successfully sampled '2' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:42:48.3844919Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_5'\\r\\n[2020-04-04T02:42:48.3858988Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_6'\\r\\n[2020-04-04T02:42:49.9149211Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_6'\\r\\n[2020-04-04T02:42:51.2342141Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_5'\\r\\n[2020-04-04T02:43:17.808295][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2020-04-04T02:43:18.413897][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:43:21.8429304Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_7'\\r\\n[2020-04-04T02:43:24.2862575Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_7'\\r\\n[2020-04-04T02:44:20.899838][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2020-04-04T02:44:21.230671][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:44:25.1026296Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_8'\\r\\n[2020-04-04T02:44:26.6770139Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_8'\\r\\n[2020-04-04T02:44:51.721028][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2020-04-04T02:44:52.063243][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:44:57.2379682Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_9'\\r\\n[2020-04-04T02:45:02.4538892Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_9'\\r\\n[2020-04-04T02:45:21.786073][GENERATOR][INFO]Trying to sample '1' jobs from the hyperparameter space\\r\\n[2020-04-04T02:45:22.639217][GENERATOR][INFO]Successfully sampled '1' jobs, they will soon be submitted to the execution target.\\r\\n[2020-04-04T02:45:32.8066779Z][SCHEDULER][INFO]Scheduling job, id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_10'\\r\\n[2020-04-04T02:45:33.6834906Z][SCHEDULER][INFO]Successfully scheduled a job. Id='HD_dcf456bc-203d-42aa-892e-23fe23133bb5_10'\\n\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": true, \"log_level\": \"INFO\", \"sdk_version\": \"1.0.85\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "RunDetails(htr).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_0': {'MAPE': 40.01020292109138},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_1': {'MAPE': 35.66304484399933},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_10': {'MAPE': 31.426060595052864},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_11': {'MAPE': 33.91326303044655},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_12': {'MAPE': 51.328982763337386},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_13': {'MAPE': 51.009816065385024},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_14': {'MAPE': 32.77447126411579},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_15': {'MAPE': 39.74068138374286},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_16': {'MAPE': 38.2882828405659},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_17': {'MAPE': 34.40702333027746},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_18': {'MAPE': 74.62836589958232},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_19': {'MAPE': 37.81360598994933},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_2': {'MAPE': 31.520921807705783},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_3': {'MAPE': 54.467139761173954},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_4': {'MAPE': 43.36610118248853},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_5': {'MAPE': 74.39909022797315},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_6': {'MAPE': 36.42988545942116},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_7': {'MAPE': 32.22186405710004},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_8': {'MAPE': 48.3703940647382},\n",
       " 'HD_dcf456bc-203d-42aa-892e-23fe23133bb5_9': {'MAPE': 30.255644695749783}}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "htr.wait_for_completion()\n",
    "htr.get_metrics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best model and its hyperparameter values can be retrieved as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['--data-folder', '$AZUREML_DATAREFERENCE_00b162eda80e4ee9a050e69c71311ecc', '--num-leaves', '121', '--min-data-in-leaf', '310', '--learning-rate', '0.05', '--feature-fraction', '0.645720370282405', '--bagging-fraction', '0.783434541501211', '--bagging-freq', '15', '--max-rounds', '1340', '--max-lag', '24', '--window-size', '8']\n"
     ]
    }
   ],
   "source": [
    "best_run = htr.get_best_run_by_primary_metric()\n",
    "parameter_values = best_run.get_details()[\"runDefinition\"][\"arguments\"]\n",
    "print(parameter_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then register the folder (and all files in it) as a model named `lgbm-oj-forecast` under the workspace for deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = best_run.register_model(\n",
    "    model_name=\"lgbm-oj-forecast\", model_path=\"outputs/model\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the Model in ACI\n",
    "\n",
    "Now we are ready to deploy the model as a web service running in Azure Container Instance [ACI](https://azure.microsoft.com/en-us/services/container-instances/). Azure Machine Learning accomplishes this by constructing a Docker image with the scoring logic and model baked in.\n",
    "\n",
    "### Create score.py\n",
    "\n",
    "First, we will create a scoring script that will be invoked by the web service call.\n",
    "\n",
    "* Note that the scoring script must have two required functions, `init()` and `run(input_data)`.\n",
    "    - In `init()` function, you typically load the model into a global object. This function is executed only once when the Docker container is started.\n",
    "    - In `run(input_data)` function, the model is used to predict a value based on the input data. The input and output to run typically use JSON as serialization and de-serialization format but you are not limited to that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting score.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile score.py\n",
    "import os\n",
    "import json\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import lightgbm as lgb\n",
    "\n",
    "\n",
    "def init():\n",
    "    global bst\n",
    "    model_root = os.getenv(\"AZUREML_MODEL_DIR\")\n",
    "    # The name of the folder in which to look for LightGBM model files\n",
    "    lgbm_model_folder = \"model\"\n",
    "    bst = lgb.Booster(\n",
    "        model_file=os.path.join(model_root, lgbm_model_folder, \"bst-model.txt\")\n",
    "    )\n",
    "\n",
    "\n",
    "def run(raw_data):\n",
    "    columns = bst.feature_name()\n",
    "    data = np.array(json.loads(raw_data)[\"data\"])\n",
    "    test_df = pd.DataFrame(data=data, columns=columns)\n",
    "    # Make prediction\n",
    "    out = bst.predict(test_df)\n",
    "    return out.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create myenv.yml\n",
    "\n",
    "We also need to create an environment file so that Azure Machine Learning can install the necessary packages in the Docker image which are required by your scoring script. In this case, we need to specify packages `numpy`, `pandas`, and `lightgbm`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Conda environment specification. The dependencies defined in this file will\r\n",
      "# be automatically provisioned for runs with userManagedDependencies=False.\r\n",
      "\n",
      "# Details about the Conda environment file format:\r\n",
      "# https://conda.io/docs/user-guide/tasks/manage-environments.html#create-env-file-manually\r\n",
      "\n",
      "name: project_environment\n",
      "dependencies:\n",
      "  # The python interpreter version.\r\n",
      "  # Currently Azure ML only supports 3.5.2 and later.\r\n",
      "- python=3.6.2\n",
      "\n",
      "- pip:\n",
      "  - azureml-defaults\n",
      "- numpy=1.16.2\n",
      "- pandas=0.23.4\n",
      "- lightgbm=2.3.0\n",
      "channels:\n",
      "- conda-forge\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cd = CondaDependencies.create()\n",
    "cd.add_conda_package(\"numpy=1.16.2\")\n",
    "cd.add_conda_package(\"pandas=0.23.4\")\n",
    "cd.add_conda_package(\"lightgbm=2.3.0\")\n",
    "cd.save_to_file(base_directory=\"./\", conda_file_path=\"myenv.yml\")\n",
    "\n",
    "print(cd.serialize_to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy to ACI\n",
    "\n",
    "We are almost ready to deploy. In the next cell, we first create the inference configuration and deployment configuration. Then, we deploy the model to ACI. This cell will run for several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running........................\n",
      "Succeeded\n",
      "ACI service creation operation finished, operation \"Succeeded\"\n",
      "Healthy\n",
      "CPU times: user 399 ms, sys: 181 ms, total: 580 ms\n",
      "Wall time: 2min 33s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "inference_config = InferenceConfig(runtime=\"python\", entry_script=\"score.py\", conda_file=\"myenv.yml\")\n",
    "\n",
    "aciconfig = AciWebservice.deploy_configuration(\n",
    "    cpu_cores=1,\n",
    "    memory_gb=1,\n",
    "    tags={\"name\": \"ojdata\", \"framework\": \"LightGBM\"},\n",
    "    description=\"LightGBM model on Orange Juice data\",\n",
    ")\n",
    "\n",
    "service = Model.deploy(\n",
    "    workspace=ws, name=\"lgbm-oj-svc\", models=[model], inference_config=inference_config, deployment_config=aciconfig\n",
    ")\n",
    "\n",
    "service.wait_for_deployment(True)\n",
    "print(service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Tip: If something goes wrong with the deployment, you could look at the logs from the service by running this command `print(service.get_logs())`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the scoring web service endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://33dfd29b-476b-4af9-9343-a96dff2bf80b.westus.azurecontainer.io/score\n"
     ]
    }
   ],
   "source": [
    "print(service.scoring_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the web service is successfully deployed, you will see a deployment in the Azure Machine Learning workspace on Azure portal\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/20047467/76572336-2100f300-6490-11ea-9467-83cd693d23c1.png\" width=\"900\" height=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the deployed model\n",
    "\n",
    "Let's test the deployed model. We create a few test data points and send them to the web service hosted in ACI. Note here we are using the run API in the SDK to invoke the service. You can also make raw HTTP calls using any HTTP tool such as curl.\n",
    "\n",
    "After the invocation, we print the returned predictions each of which represents the forecasted sales of a target store, brand in a given week as specified by `store, brand, week` in `used_columns`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prediction: [10657.690610049329, 17938.679182737964, 5974.42429028091]\n"
     ]
    }
   ],
   "source": [
    "# Prepare features according to the input schema of the best model\n",
    "train_dir = os.path.join(DATA_DIR, \"train\")\n",
    "max_lag = int(parameter_values[parameter_values.index(\"--max-lag\") + 1])\n",
    "lags = np.arange(2, max_lag + 1)\n",
    "window_size = int(parameter_values[parameter_values.index(\"--window-size\") + 1])\n",
    "used_columns = [\n",
    "    \"store\",\n",
    "    \"brand\",\n",
    "    \"week\",\n",
    "    \"week_of_month\",\n",
    "    \"month\",\n",
    "    \"deal\",\n",
    "    \"feat\",\n",
    "    \"move\",\n",
    "    \"price\",\n",
    "    \"price_ratio\",\n",
    "]\n",
    "GAP = 2\n",
    "features, train_end_week = create_features(\n",
    "    1, train_dir, lags, window_size, used_columns\n",
    ")\n",
    "test_fea = features[features.week >= train_end_week + GAP].reset_index(drop=True)\n",
    "test_fea.drop(\"move\", axis=1, inplace=True)\n",
    "\n",
    "# Pick a few test data points\n",
    "test_samples = json.dumps({\"data\": np.array(test_fea.iloc[:3]).tolist()})\n",
    "test_samples = bytes(test_samples, encoding=\"utf8\")\n",
    "\n",
    "# Predict using the deployed model\n",
    "result = service.run(input_data=test_samples)\n",
    "print(\"prediction:\", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also send raw HTTP request to the service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "POST to url http://33dfd29b-476b-4af9-9343-a96dff2bf80b.westus.azurecontainer.io/score\n",
      "\n",
      "input data: b'{\"data\": [[2.0, 1.0, 137.0, 4.0, 4.0, 0.0, 0.0, 0.0416446872, 1.1124927835293534, 12416.0, 28096.0, 15168.0, 20736.0, 31808.0, 25728.0, 43584.0, 5056.0, 20224.0, 6720.0, 5504.0, 3520.0, 9792.0, 13120.0, 13120.0, 17728.0, 8320.0, 5120.0, 6080.0, 7168.0, 34240.0, 7296.0, 9216.0, 22824.0], [2.0, 1.0, 138.0, 5.0, 4.0, 1.0, 1.0, 0.03734375, 0.9420125411290402, 12416.0, 12416.0, 28096.0, 15168.0, 20736.0, 31808.0, 25728.0, 43584.0, 5056.0, 20224.0, 6720.0, 5504.0, 3520.0, 9792.0, 13120.0, 13120.0, 17728.0, 8320.0, 5120.0, 6080.0, 7168.0, 34240.0, 7296.0, 23744.0], [2.0, 2.0, 137.0, 4.0, 4.0, 0.0, 0.0, 0.0519791667, 1.388567227553081, 11424.0, 4992.0, 7008.0, 6816.0, 5280.0, 7296.0, 5664.0, 17184.0, 6048.0, 8544.0, 12864.0, 18240.0, 6816.0, 8448.0, 9024.0, 9504.0, 9120.0, 10944.0, 16032.0, 7968.0, 8064.0, 8352.0, 7488.0, 8208.0]]}'\n",
      "\n",
      "prediction: [10657.690610049329, 17938.679182737964, 5974.42429028091]\n"
     ]
    }
   ],
   "source": [
    "headers = {\"Content-Type\": \"application/json\"}\n",
    "\n",
    "resp = requests.post(service.scoring_uri, test_samples, headers=headers)\n",
    "\n",
    "print(\"POST to url\", service.scoring_uri)\n",
    "print(\"\")\n",
    "print(\"input data:\", test_samples)\n",
    "print(\"\")\n",
    "print(\"prediction:\", resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up\n",
    "\n",
    "After finishing the tests, you can delete the ACI deployment with a simple delete API call as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "service.delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Reading:\n",
    "\n",
    "\\[1\\] Training, hyperparameter tune, and deploy with TensorFlow: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/ml-frameworks/tensorflow/deployment/train-hyperparameter-tune-deploy-with-tensorflow/train-hyperparameter-tune-deploy-with-tensorflow.ipynb <br>\n",
    "\n",
    "\\[2\\] AzureML HyperDrive package: https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.hyperdrive?view=azure-ml-py"
   ]
  }
 ],
 "metadata": {
  "author_info": {
   "affiliation": "Microsoft",
   "created_by": "Chenhui Hu"
  },
  "kernelspec": {
   "display_name": "forecasting_env",
   "language": "python",
   "name": "forecasting_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}