sdk/python/jobs/single-step/lightgbm/iris/lightgbm-iris-sweep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run hyperparameter sweep on a Command or CommandComponent\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace with computer cluster - [Configure workspace](../../../configuration.ipynb) \n",
    "\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Create and run a `Command` from Python SDK.\n",
    "- Use the `SweepJob ` to run a hyperparameter sweep on this Command\n",
    "- Create a `CommandComponent` from Python SDK.\n",
    "- Use the `SweepJob ` to run a hyperparameter sweep on the CommandComponent.\n",
    "\n",
    "**Motivations** - This example explains how Azure Machine Learning lets you automate hyperparameter tuning using the SweepJob to efficiently optimize hyperparameters. Hyperparameters are adjustable parameters that let you control the model training process. Hyperparameter tuning, also called hyperparameter optimization, is the process of finding the configuration of hyperparameters that results in the best performance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "## 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml import command, Input\n",
    "from azure.ai.ml.sweep import Choice, Uniform, MedianStoppingPolicy\n",
    "from azure.identity import DefaultAzureCredential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../../configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "connect-workspace"
   },
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "# Enter details of your AML workspace\n",
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\"\n",
    "\n",
    "# connect to the workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Configure and run the Command\n",
    "In this section we will configure and run a standalone job using the `command` class. The `command` class can be used to run standalone jobs and can also be used as a function inside pipelines.\n",
    "\n",
    "## 2.1 Configure the Command\n",
    "The `command` allows user to configure the following key aspects.\n",
    "- `code` - This is the path where the code to run the command is located\n",
    "- `command` - This is the command that needs to be run\n",
    "- `inputs` - This is the dictionary of inputs using name value pairs to the command. The key is a name for the input within the context of the job and the value is the input value. Inputs can be referenced in the `command` using the `${{inputs.<input_name>}}` expression. To use files or folders as inputs, we can use the `Input` class. The `Input` class supports three parameters:\n",
    "    - `type` - The type of input. This can be a `uri_file` or `uri_folder`. The default is `uri_folder`.         \n",
    "    - `path` - The path to the file or folder. These can be local or remote files or folders. For remote files - http/https, wasb are supported. \n",
    "        - Azure ML `data`/`dataset` or `datastore` are of type `uri_folder`. To use `data`/`dataset` as input, you can use registered dataset in the workspace using the format '<data_name>:<version>'. For e.g Input(type='uri_folder', path='my_dataset:1')\n",
    "    - `mode` - \tMode of how the data should be delivered to the compute target. Allowed values are `ro_mount`, `rw_mount` and `download`. Default is `ro_mount`\n",
    "- `environment` - This is the environment needed for the command to run. Curated or custom environments from the workspace can be used. Or a custom environment can be created and used as well. Check out the [environment](../../../../assets/environment/environment.ipynb) notebook for more examples.\n",
    "- `compute` - The compute on which the command will run. In this example we are using a compute called `cpu-cluster` present in the workspace. You can replace it any other compute in the workspace. You can run it on the local machine by using `local` for the compute. This will run the command on the local machine and all the run details and output of the job will be uploaded to the Azure ML workspace.\n",
    "- `distribution` - Distribution configuration for distributed training scenarios. Azure Machine Learning supports PyTorch, TensorFlow, and MPI-based distributed training. The allowed values are `PyTorch`, `TensorFlow` or `Mpi`.\n",
    "- `display_name` - The display name of the Job\n",
    "- `description` - The description of the experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create-command"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml import command, Input\n",
    "\n",
    "# define the command\n",
    "command_job = command(\n",
    "    code=\"./src\",\n",
    "    command=\"python main.py --iris-csv ${{inputs.iris_csv}} --learning-rate ${{inputs.learning_rate}} --boosting ${{inputs.boosting}}\",\n",
    "    environment=\"AzureML-lightgbm-3.2-ubuntu18.04-py37-cpu@latest\",\n",
    "    inputs={\n",
    "        \"iris_csv\": Input(\n",
    "            type=\"uri_file\",\n",
    "            path=\"https://azuremlexamples.blob.core.windows.net/datasets/iris.csv\",\n",
    "        ),\n",
    "        \"learning_rate\": 0.9,\n",
    "        \"boosting\": \"gbdt\",\n",
    "    },\n",
    "    compute=\"cpu-cluster\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 Run the Command\n",
    "Using the `MLClient` created earlier, we will now run this command in the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "run-command"
   },
   "outputs": [],
   "source": [
    "# submit the command\n",
    "returned_job = ml_client.jobs.create_or_update(command_job)\n",
    "# get a URL for the status of the job\n",
    "returned_job.studio_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.  Run a sweep on this command\n",
    "Hyperparameters are adjustable parameters that let you control the model training process. Hyperparameter tuning, also called hyperparameter optimization, is the process of finding the configuration of hyperparameters that results in the best performance. Azure Machine Learning provides a `sweep` function on the `command` to do hyperparameter tuning. \n",
    "\n",
    "The `sweep` function allows user to configure the following key aspects:\n",
    "- `sampling_algorithm`- The hyperparameter sampling algorithm to use over the search_space. Allowed values are `random`, `grid` and `bayesian`.\n",
    "- `objective` - the objective of the sweep\n",
    "  - `primary_metric` - The name of the primary metric reported by each trial job. The metric must be logged in the user's training script using `mlflow.log_metric()` with the same corresponding metric name.\n",
    "  - `goal` - The optimization goal of the objective.primary_metric. The allowed values are `maximize` and `minimize`.\n",
    "- `compute` - Name of the compute target to execute the job on.\n",
    "- `inputs` - This is the dictionary of inputs using name value pairs to the SweepJob. The key is a name for the input within the context of the job and the value is the input value. Inputs can be referenced in the `command` using the `${{inputs.<input_name>}}` expression. \n",
    "- `limits` - Limits for the sweep job\n",
    "\n",
    "## 3.1 Modify or define inputs to sweep\n",
    "To perform a sweep, there needs to be input(s) against which the sweep needs to be performed. These inputs can have a discrete or continuos value. The `sweep` function will run the `command` multiple times using different combination of input values specified. Each input is a dictionary of name value pairs. The key is the name of the hyperparameter and the value is the parameter expression. \n",
    "\n",
    "Let us improve our model by sweeping on `learning_rate` and `boosting` inputs to the script. In the previous step, we used a specific value for these, but now we will use a range or  choice of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "search-space"
   },
   "outputs": [],
   "source": [
    "# we will reuse the command_job created before. we call it as a function so that we can apply inputs\n",
    "# we do not apply the 'iris_csv' input again -- we will just use what was already defined earlier\n",
    "command_job_for_sweep = command_job(\n",
    "    learning_rate=Uniform(min_value=0.01, max_value=0.9),\n",
    "    boosting=Choice(values=[\"gbdt\", \"dart\"]),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Define the sweep function \n",
    "We will now define the parameters of the sweep function. In this example we configure the sweep function with `trial`, `search_space`, `sampling_algorithm`, `objective`, `limits` and `compute`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "configure-sweep"
   },
   "outputs": [],
   "source": [
    "# apply the sweep parameter to obtain the sweep_job\n",
    "sweep_job = command_job_for_sweep.sweep(\n",
    "    compute=\"cpu-cluster\",\n",
    "    sampling_algorithm=\"random\",\n",
    "    primary_metric=\"test-multi_logloss\",\n",
    "    goal=\"Minimize\",\n",
    ")\n",
    "\n",
    "# define the limits for this sweep\n",
    "sweep_job.set_limits(max_total_trials=20, max_concurrent_trials=10, timeout=7200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Run the SweepJob\n",
    "Using the `MLClient` created earlier, we will now run this SweepJob in the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "run-sweep"
   },
   "outputs": [],
   "source": [
    "# submit the sweep\n",
    "returned_sweep_job = ml_client.create_or_update(sweep_job)\n",
    "# get a URL for the status of the job\n",
    "returned_sweep_job.studio_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Run a sweep on a Command Component\n",
    "An Azure Machine Learning [component](https://docs.microsoft.com/en-us/azure/machine-learning/concept-component) is a self-contained piece of code that performs a step in a machine learning job or pipeline. Components can do tasks such as data processing, model training, model scoring, and so on. A component is analogous to a function - it has a name, parameters, expects input, and returns output. This makes them very useful to reuse code/functionality.\n",
    "\n",
    "# 4.1 Create reusable component\n",
    "Let us create a component to perform the same functionality of the `command_job` we created earlier. A component allows user to define the following key parameters:\n",
    "- `command` - The command to execute\n",
    "- `code` - Local path to the source code directory to be uploaded and used for the component.\n",
    "- `environment` - The environment to use for the component. This can be either a reference to an existing versioned environment in the workspace or an inline environment specification.\n",
    "- `inputs` - This is the dictionary of inputs using name value pairs to the CommandComponent. Inputs can be referenced in the `command` using the `${{inputs.<input_name>}}` expression.\n",
    "- `instance_count` - The number of nodes to use for the component\n",
    "- `name` - Name of the component\n",
    "- `version` - Version of the component\n",
    "- `distribution` - The distribution configuration for distributed training scenarios. Allowed values are `Pytorch`, `Mpi`, or `TensorFlow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the command component -- this is the same as the command above, only no compute is specified and no values but type infomation is provided as inputs\n",
    "command_component = command(\n",
    "    code=\"./src\",\n",
    "    command=\"python main.py --iris-csv ${{inputs.iris_csv}} --learning-rate ${{inputs.learning_rate}} --boosting ${{inputs.boosting}}\",\n",
    "    environment=\"AzureML-lightgbm-3.2-ubuntu18.04-py37-cpu@latest\",\n",
    "    inputs={\n",
    "        \"iris_csv\": Input(type=\"uri_file\"),\n",
    "        \"learning_rate\": 0.9,\n",
    "        \"boosting\": \"gbdt\",\n",
    "    },\n",
    "    display_name=\"lightgbm-iris-example\",\n",
    "    experiment_name=\"lightgbm-iris-example\",\n",
    "    description=\"Train a LightGBM model on the Iris dataset.\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Define the SweepJob \n",
    "We will now define the sweep job. In this example we configure the sweep job with `trial`, `search_space`, `sampling_algorithm`, `objective`, `limits` and `compute`.\n",
    "\n",
    "We will use the `trial_component` created earlier as the `trial` for this `SweepJob`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run sweep using this component\n",
    "# this time we will provide in input for iris_csv since no default was provide at compnent definition\n",
    "command_component_job_for_sweep = command_component(\n",
    "    learning_rate=Uniform(min_value=0.01, max_value=0.9),\n",
    "    boosting=Choice(values=[\"gbdt\", \"dart\"]),\n",
    "    iris_csv=Input(\n",
    "        type=\"uri_file\",\n",
    "        path=\"wasbs://datasets@azuremlexamples.blob.core.windows.net/iris.csv\",\n",
    "    ),\n",
    ")\n",
    "\n",
    "# this is the same as above\n",
    "cmd_component_sweep_job = command_component_job_for_sweep.sweep(\n",
    "    compute=\"cpu-cluster\",\n",
    "    sampling_algorithm=\"random\",\n",
    "    primary_metric=\"test-multi_logloss\",\n",
    "    goal=\"Minimize\",\n",
    ")\n",
    "\n",
    "cmd_component_sweep_job.display_name = \"lightgbm-iris-sweep-example\"\n",
    "cmd_component_sweep_job.experiment_name = \"lightgbm-iris-sweep-example\"\n",
    "cmd_component_sweep_job.description = (\n",
    "    \"Run a hyperparameter sweep job for LightGBM on Iris dataset.\"\n",
    ")\n",
    "\n",
    "# define the limits for this sweep\n",
    "cmd_component_sweep_job.set_limits(\n",
    "    max_total_trials=20, max_concurrent_trials=10, timeout=7200\n",
    ")\n",
    "\n",
    "# set early stopping on this one\n",
    "cmd_component_sweep_job.early_termination = MedianStoppingPolicy(\n",
    "    delay_evaluation=5, evaluation_interval=2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 Run the SweepJob\n",
    "Using the `MLClient` created earlier, we will now run this SweepJob in the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit the sweep job\n",
    "returned_sweep_job_cmd = ml_client.jobs.create_or_update(cmd_component_sweep_job)\n",
    "\n",
    "# get a URL for the status of the job\n",
    "returned_sweep_job_cmd.studio_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "You can see further examples of running a job [here](../../../single-step/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "description": {
   "description": "Run **hyperparameter sweep** on a Command or CommandComponent"
  },
  "interpreter": {
   "hash": "66962d4c952b5ba37638a017d6cc83bab37d76f69b13c17d86b9f71233a0aa71"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}