scenarios/classification/20_azure_workspace_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>\n",
    "\n",
    "# Setup of an Azure workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/ComputerVision/classification/notebooks/20_azure_workspace_setup.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction <a id=\"intro\"></a>\n",
    "\n",
    "This notebook is the first of a series (starting with \"2x_\") that leverage the [Azure Machine Learning Service](https://docs.microsoft.com/en-us/azure/machine-learning/service/overview-what-is-azure-ml). Azure ML, as we also call it, is a service that allows us to train, deploy, automate, and manage machine learning models, at scale, in the cloud.\n",
    "\n",
    "In this tutorial, we will set up an [Azure ML workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-azure-machine-learning-architecture#workspace). Such resource organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inferencing, and the monitoring of deployed models.\n",
    "\n",
    "## 2. Pre-requisites\n",
    "<a id=\"pre-reqs\"></a>\n",
    "\n",
    "For this and all the other \"2x_\" notebooks to run properly on our machine, we need access to the Azure platform.\n",
    "\n",
    "Unless we already have one, we should first:\n",
    "- [Create an account](https://azure.microsoft.com/en-us/free/services/machine-learning/)\n",
    "- [Create a resource group and a workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/setup-create-workspace#portal)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Azure workspace <a id=\"workspace\"></a>\n",
    "\n",
    "In the deployment tutorials present in this repository (numbered \"21_\" to \"23_\"), we use the Azure ML SDK. It allows us to access our Azure resources programmatically. As we are running our notebooks in the \"cvbp\" conda environment, the SDK should already be installed on our machine. Let's check which version of the Azure SDK we are working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Azure ML SDK Version: 1.0.48\n"
     ]
    }
   ],
   "source": [
    "# For automatic reloading of modified libraries\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import sys\n",
    "sys.path.extend([\"..\", \"../..\"]) # to access the utils_cv library\n",
    "# Azure\n",
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Check core SDK version number\n",
    "print(f\"Azure ML SDK Version: {azureml.core.VERSION}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create or access an Azure ML Workspace, you will need the following information:\n",
    "\n",
    "- <b>subscription ID:</b> the ID of the Azure subscription we are using\n",
    "- <b>resource group:</b> the name of the resource group in which our workspace resides\n",
    "- <b>workspace region:</b> the geographical area in which our workspace resides (e.g. \"eastus2\" -- other examples are available [here](https://azure.microsoft.com/en-us/global-infrastructure/geographies/) <i>-- note the lack of spaces</i>)\n",
    "- <b>workspace name:</b> the name of the workspace we want to create or retrieve.\n",
    "\n",
    "Note: As with other Azure services, there are limits on certain resources like cluster size associated with the Azure Machine Learning service. Please read this article on the default limits and how to request more quota.\n",
    "\n",
    "If you have a workspace created already, you need to get your subscription and workspace information. You can find the values for those by visiting your workspace in the Azure portal. If you don't have a workspace, the create workspace command in the next section will create a resource group and a workspace using the names you provide.\n",
    "\n",
    "Replace the values in the following cell with your information. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "subscription_id = \"YOUR_SUBSCRIPTION_ID\"\n",
    "resource_group = \"YOUR_RESOURCE_GROUP_NAME\"  \n",
    "workspace_name = \"YOUR_WORKSPACE_NAME\"  \n",
    "workspace_region = \"YOUR_WORKSPACE_REGION\" #Possible values eastus, eastus2 and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get or Create the workspace\n",
    "This cell will create an AML workspace for you in a subscription, if one does not exist already, provided you have the correct permissions.\n",
    "\n",
    "This will fail when:\n",
    "\n",
    "You do not have permission to create a workspace in the resource group\n",
    "You do not have permission to create a resource group if it's non-existing.\n",
    "You are not a subscription owner or contributor and no Azure ML workspaces have ever been created in this subscription\n",
    "If workspace creation fails, please work with your IT admin to provide you with the appropriate permissions or to provision the required resources. If this cell succeeds, you're done configuring AML!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING - Warning: Falling back to use azure cli login credentials.\n",
      "If you run your code in unattended mode, i.e., where you can't give a user input, then we recommend to use ServicePrincipalAuthentication or MsiAuthentication.\n",
      "Please refer to aka.ms/aml-notebook-auth for different authentication mechanisms in azureml-sdk.\n",
      "ERROR - get_workspace error using subscription_id=2ad17db4-e26d-4c9e-999e-adae9182530c, resource_group_name=amlnotebookrg, workspace_name=amlnotebookws\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating new workspace\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UserWarning: The resource group doesn't exist or was not provided. AzureML SDK is creating a resource group=amlnotebookrg in location=eastus using subscription=2ad17db4-e26d-4c9e-999e-adae9182530c.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deploying AppInsights with name amlnotebinsights0ca90703.\n",
      "Deployed AppInsights with name amlnotebinsights0ca90703. Took  seconds.\n",
      "Deploying KeyVault with name amlnotebkeyvaultb03f393a.\n",
      "Deploying StorageAccount with name amlnotebstorage914c3d2fd.\n",
      "Deployed StorageAccount with name amlnotebstorage914c3d2fd. Took  seconds.\n",
      "Deployed KeyVault with name amlnotebkeyvaultb03f393a. Took  seconds.\n",
      "Deploying Workspace with name amlnotebookws.\n",
      "Deployed Workspace with name amlnotebookws. Took  seconds.\n"
     ]
    }
   ],
   "source": [
    "# A util method that creates a workspace or retrieves one if it exists, also takes care of Azure Authentication\n",
    "from utils_cv.common.azureml import get_or_create_workspace\n",
    "\n",
    "ws = get_or_create_workspace(\n",
    "        subscription_id,\n",
    "        resource_group,\n",
    "        workspace_name,\n",
    "        workspace_region)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check that the workspace is properly loaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workspace name: amlnotebookws\n",
      "       Workspace region: eastus\n",
      "       Resource group: amlnotebookrg\n"
     ]
    }
   ],
   "source": [
    "# Print the workspace attributes\n",
    "print(f'Workspace name: {ws.name}\\n \\\n",
    "      Workspace region: {ws.location}\\n \\\n",
    "      Subscription id: {ws.subscription_id}\\n \\\n",
    "      Resource group: {ws.resource_group}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see this workspace on the [Azure portal](http://portal.azure.com) by sequentially clicking on:\n",
    "- Resource groups, and clicking the one we referenced above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/resource_group.jpg\" width=\"800\" alt=\"Azure portal view of resource group\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Workspace_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"media/workspace.jpg\" width=\"800\" alt=\"Azure portal view of workspace\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more details on the setup of a workspace and other Azure resources, we can refer to this [configuration](https://github.com/Azure/MachineLearningNotebooks/blob/dcce6f227f9ca62e4c201fb48ae9dc8739eaedf7/configuration.ipynb) notebook.\n",
    "\n",
    "Creating a workspace automatically adds associated resources:\n",
    "- A container registry, which can host Docker images, and gets lazily created\n",
    "- A storage account, in which output files get stored\n",
    "- Application Insights, which allows us to monitor the health of and traffic to a web service, for instance\n",
    "- A key vault, which stores our credentials.\n",
    "\n",
    "Such resources, when first created, cost less than a penny per day. To get a better sense of pricing, we can refer to this [calculator](https://azure.microsoft.com/en-us/pricing/calculator/). We can also navigate to the [Cost Management + Billing](https://ms.portal.azure.com/#blade/Microsoft_Azure_Billing/ModernBillingMenuBlade/Overview) pane on the portal, click on our subscription ID, and click on the Cost Analysis tab to check our credit usage.\n",
    "\n",
    "We will continue using our workspace in the next notebooks, so will keep it available. However, if we needed to delete it, we would run the command below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write the workspace configuration for the rest of the notebooks to connect to the workspace.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws.write_config()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ws.delete(delete_dependent_resources=True)\n",
    "# This deletes our workspace, the container registry, the account storage, Application Insights and the key vault"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Next steps\n",
    "\n",
    "In this notebook, we created a new workspace, and stored configuration information in a `./aml_config/config.json` file. Using this file we can load as workspace using from_config() method `ws = Workspace.from_config()`.\n",
    "\n",
    "In the next notebook, we will learn how to deploy a trained model as a web service on Azure."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "cv",
   "language": "python",
   "name": "cv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}