sdk/python/jobs/configuration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Configuration\n",
    "\n",
    "_**Setting up your Azure Machine Learning services workspace and configuring needed resources**_\n",
    "\n",
    "---\n",
    "---\n",
    "\n",
    "**Requirements** - In order to benefit from this tutorial, you will need:\n",
    "- A basic understanding of Machine Learning\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace - [Create workspace](../resources/workspace/workspace.ipynb) \n",
    "- A python environment\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](../README.md) - check the getting started section\n",
    "\n",
    "**Learning Objectives** - By the end of this tutorial, you should be able to:\n",
    "- Connect to your AML workspace from the Python SDK using different auth credentials\n",
    "- Create workspace config file\n",
    "- Create Compute clusters which required by jobs notebooks. [Check this notebook to create a compute cluster](../resources/compute/compute.ipynb)\n",
    "\n",
    "**Motivations** - This notebook covers the scenario that user define components using yaml then use these components to build pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# 1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import required libraries\n",
    "from azure.ai.ml import MLClient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Configure credential\n",
    "\n",
    "We are using `DefaultAzureCredential` to get access to workspace. When an access token is needed, it requests one using multiple identities(`EnvironmentCredential, ManagedIdentityCredential, SharedTokenCacheCredential, VisualStudioCodeCredential, AzureCliCredential, AzurePowerShellCredential`) in turn, stopping when one provides a token.\n",
    "Reference [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for more information.\n",
    "\n",
    "`DefaultAzureCredential` should be capable of handling most Azure SDK authentication scenarios. \n",
    "Reference [here](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity?view=azure-python) for all available credentials if it does not work for you.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    # Check if given credential can get token successfully.\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work\n",
    "    # This will open a browser page for\n",
    "    credential = InteractiveBrowserCredential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run. \n",
    "\n",
    "Check this notebook for creating a [workspace](../resources/workspace/workspace.ipynb).\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. \n",
    "The config details of a workspace can be saved to a file from the Azure Machine Learning [portal](https://ml.azure.com/). Click on the name of the portal on the top right corner to see the link to save the config file.\n",
    "This config file can be used to load a workspace using `MLClient`. If no path is mentioned, path is defaulted to current folder. If no file name is mentioned, file name will be defaulted to `config.json`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    ml_client = MLClient.from_config(credential=credential)\n",
    "except Exception as ex:\n",
    "    # NOTE: Update following workspace information if not correctly configure before\n",
    "    client_config = {\n",
    "        \"subscription_id\": \"<SUBSCRIPTION_ID>\",\n",
    "        \"resource_group\": \"<RESOURCE_GROUP>\",\n",
    "        \"workspace_name\": \"<AML_WORKSPACE_NAME>\",\n",
    "    }\n",
    "\n",
    "    if client_config[\"subscription_id\"].startswith(\"<\"):\n",
    "        print(\n",
    "            \"please update your <SUBSCRIPTION_ID> <RESOURCE_GROUP> <AML_WORKSPACE_NAME> in notebook cell\"\n",
    "        )\n",
    "        raise ex\n",
    "    else:  # write and reload from config file\n",
    "        import json, os\n",
    "\n",
    "        config_path = \"../.azureml/config.json\"\n",
    "        os.makedirs(os.path.dirname(config_path), exist_ok=True)\n",
    "        with open(config_path, \"w\") as fo:\n",
    "            fo.write(json.dumps(client_config))\n",
    "        ml_client = MLClient.from_config(credential=credential, path=config_path)\n",
    "print(ml_client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Retrieve or create an Azure Machine Learning compute target\n",
    "\n",
    "To create a Azure Machine Learning job, you need a compute cluster as prerequisite. Below code ensures computes named `cpu-cluster` and `gpu-cluster` exists in your workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create-cpu-compute"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import AmlCompute\n",
    "\n",
    "# specify aml compute name.\n",
    "cpu_compute_target = \"cpu-cluster\"\n",
    "\n",
    "try:\n",
    "    ml_client.compute.get(cpu_compute_target)\n",
    "except Exception:\n",
    "    print(\"Creating a new cpu compute target...\")\n",
    "    compute = AmlCompute(\n",
    "        name=cpu_compute_target, size=\"STANDARD_D2_V2\", min_instances=0, max_instances=4\n",
    "    )\n",
    "    ml_client.compute.begin_create_or_update(compute).result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "create-gpu-compute"
   },
   "outputs": [],
   "source": [
    "from azure.ai.ml.entities import AmlCompute\n",
    "\n",
    "# specify aml compute name.\n",
    "gpu_compute_target = \"gpu-cluster\"\n",
    "\n",
    "try:\n",
    "    ml_client.compute.get(gpu_compute_target)\n",
    "except Exception:\n",
    "    print(\"Creating a new gpu compute target...\")\n",
    "    compute = AmlCompute(\n",
    "        name=gpu_compute_target, size=\"STANDARD_NC6\", min_instances=0, max_instances=4\n",
    "    )\n",
    "    ml_client.compute.begin_create_or_update(compute).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next Steps\n",
    "You can see further examples of running a pipeline job [here](../jobs/pipelines/README.md)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Setting up your Azure Machine Learning services workspace and configuring needed resources"
  },
  "interpreter": {
   "hash": "3e9e0e270b75c5e6da2e22113ba4f77b864d68f95da6601809c29e46c73ae6bb"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}