sdk/python/jobs/pipelines/2b_train_cifar_10_with_pytorch/train_cifar_10_with_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cifar-10 pipeline\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace with computer cluster - [Configure workspace](../../configuration.ipynb)\n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../../../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Connect to your AML workspace from the Python SDK\n",
    "- Define `CommandComponent` using YAML\n",
    "- Create basic `Pipeline` using component from local YAML file\n",
    "\n",
    "**Motivations** - This notebook explains how to define `commandComponent` via YAML then use command component to build pipeline. The command compoonent is a fundamental construct of Azure Machine Learning pipeline. It can be used to run a task on a specified compute (either local or on the cloud). The command compoonent accepts `Environment` to setup required infrastructure. You can define a `command` to run on this infrastructure with `inputs`. You can reuse the same `Component` with different pipeline.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "## 1.1 Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
    "\n",
    "from azure.ai.ml import MLClient, Input\n",
    "from azure.ai.ml.dsl import pipeline\n",
    "from azure.ai.ml import load_component"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Configure credential\n",
    "\n",
    "We are using `DefaultAzureCredential` to get access to workspace. \n",
    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
    "\n",
    "Reference for more available credentials if it does not work for you: [configure credential example](../../configuration.ipynb), [azure-identity reference doc](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    # Check if given credential can get token successfully.\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
    "    credential = InteractiveBrowserCredential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Get a handle to the workspace\n",
    "\n",
    "We use config file to connect to a workspace. The Azure ML workspace should be configured with computer cluster. [Check this notebook for configure a workspace](../../configuration.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a handle to workspace\n",
    "ml_client = MLClient.from_config(credential=credential)\n",
    "\n",
    "# Retrieve an already attached Azure Machine Learning Compute.\n",
    "cluster_name = \"cpu-cluster\"\n",
    "print(ml_client.compute.get(cluster_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Define command component via YAML\n",
    "Define command component using YAML and load as function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parent_dir = \".\"\n",
    "get_data_func = load_component(source=parent_dir + \"/get-data.yml\")\n",
    "train_model_func = load_component(source=parent_dir + \"/train-model.yml\")\n",
    "eval_model_func = load_component(source=parent_dir + \"/eval-model.yml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Basic pipeline job\n",
    "\n",
    "## 3.1 Build pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note that the used docker image doesn't suit for all size of gpu compute. Please use the following command to create gpu compute if experiment failed\n",
    "# !az ml compute create -n gpu-cluster --type amlcompute --min-instances 0 --max-instances 4 --size Standard_NC12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define pipeline\n",
    "@pipeline()\n",
    "def train_cifar_10_with_pytorch():\n",
    "    \"\"\"CIFAR-10 Pipeline Example.\"\"\"\n",
    "    # define the job to get data\n",
    "    get_data = get_data_func(\n",
    "        cifar_zip=Input(\n",
    "            path=\"wasbs://datasets@azuremlexamples.blob.core.windows.net/cifar-10-python.tar.gz\",\n",
    "            type=\"uri_file\",\n",
    "        )\n",
    "    )\n",
    "    get_data.outputs.cifar.mode = \"upload\"\n",
    "\n",
    "    # define the job to train the model\n",
    "    train_model = train_model_func(epochs=1, cifar=get_data.outputs.cifar)\n",
    "    train_model.compute = \"gpu-cluster\"\n",
    "    train_model.outputs.model_dir.mode = \"upload\"\n",
    "\n",
    "    # define the job to evaluate the model\n",
    "    eval_model = eval_model_func(\n",
    "        cifar=get_data.outputs.cifar, model_dir=train_model.outputs.model_dir\n",
    "    )\n",
    "    eval_model.compute = \"gpu-cluster\"\n",
    "\n",
    "\n",
    "pipeline_job = train_cifar_10_with_pytorch()\n",
    "\n",
    "# set pipeline level compute\n",
    "pipeline_job.settings.default_compute = \"cpu-cluster\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.2 Submit pipeline job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit the pipeline job\n",
    "pipeline_job = ml_client.jobs.create_or_update(\n",
    "    pipeline_job, experiment_name=\"pipeline_samples\"\n",
    ")\n",
    "pipeline_job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait until the job completes\n",
    "ml_client.jobs.stream(pipeline_job.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "You can see further examples of running a pipeline job [here](../)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Get data, train and evaluate a model in pipeline with Components"
  },
  "interpreter": {
   "hash": "45ee23ad53d8447c1a4a7f9f605254595f8ee53c2e1723e7948bbd485e96ca91"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}