sdk/python/endpoints/batch/deploy-models/custom-outputs-parquet/custom-output-batch.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "# Customize outputs in batch deployments\n",
    "\n",
    "Sometimes you need to execute inference having a higher control of what is being written as output of the batch job. Batch Deployments allow you to take control of the output of the jobs by allowing you to write directly to the output of the batch deployment job. In this tutorial, we'll see how to deploy a model to perform batch inference and writes the outputs in parquet format by appending the predictions to the original input data."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook requires:\n",
    "\n",
    "* `azure-ai-ml`\n",
    "* `mlflow`\n",
    "* `azureml-mlflow`\n",
    "* `lightgbm==1.5.2`\n",
    "* `numpy`\n",
    "* `pandas`\n",
    "* `pyarrow`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Connect to Azure Machine Learning Workspace\n",
    "\n",
    "The [workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run.\n",
    "\n",
    "### 1.1. Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "from azure.ai.ml import MLClient, Input\n",
    "from azure.ai.ml.entities import (\n",
    "    BatchEndpoint,\n",
    "    ModelBatchDeployment,\n",
    "    ModelBatchDeploymentSettings,\n",
    "    Model,\n",
    "    AmlCompute,\n",
    "    BatchRetrySettings,\n",
    "    CodeConfiguration,\n",
    "    Environment,\n",
    ")\n",
    "from azure.ai.ml.constants import AssetTypes, BatchDeploymentOutputAction\n",
    "from azure.identity import DefaultAzureCredential"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [default azure authentication](https://docs.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) for this tutorial. Check the [configuration notebook](../../jobs/configuration.ipynb) for more details on how to configure credentials and connect to a workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are working in a Azure Machine Learning compute, you can simply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client = MLClient.from_config(DefaultAzureCredential())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 2. Registering the model\n",
    "\n",
    "### 2.1 About the model\n",
    "\n",
    "This example shows how you can deploy a model to a batch endpoint to perform batch predictions. This example uses a model based on the UCI Heart Disease Data Set. The database contains 76 attributes, but we are using a subset of 14 of them. The model tries to predict the presence of heart disease in a patient. It is integer valued from 0 (no presence) to 1 (presence).\n",
    "\n",
    "The model has been trained using an `XGBBoost` classifier and all the required preprocessing has been packaged as a `scikit-learn` pipeline, making this model an end-to-end pipeline that goes from raw data to predictions.\n",
    "\n",
    "### 2.2 Registering the model in the workspace\n",
    "\n",
    "Let's verify if the model we want to deploy, `heart-classifier`, is registered in the model registry. If not, we will register it from a local version we have in the repository:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "register_model",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "model_name = \"heart-classifier-sklpipe\"\n",
    "model_description = \"A heart condition classifier trained with XGBoosts and Scikit-Learn for feature processing.\"\n",
    "model_local_path = \"model\"\n",
    "\n",
    "model = ml_client.models.create_or_update(\n",
    "    Model(\n",
    "        name=model_name,\n",
    "        path=model_local_path,\n",
    "        type=AssetTypes.CUSTOM_MODEL,\n",
    "        tags={\"framework\": \"scikit-learn\", \"estimator\": \"xgboost\"},\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "Let's get the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "get_model",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "model = ml_client.models.get(name=model_name, label=\"latest\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## 3 Create Batch Endpoint\n",
    "\n",
    "Batch endpoints are endpoints that are used batch inferencing on large volumes of data over a period of time. Batch endpoints receive pointers to data and run jobs asynchronously to process the data in parallel on compute clusters. Batch endpoints store outputs to a data store for further analysis.\n",
    "\n",
    "To create an online endpoint we will use `BatchEndpoint`. This class allows user to configure the following key aspects:\n",
    "- `name` - Name of the endpoint. Needs to be unique at the Azure region level\n",
    "- `auth_mode` - The authentication method for the endpoint. Currently only Azure Active Directory (Azure AD) token-based (`aad_token`) authentication is supported. \n",
    "- `defaults` - Default settings for the endpoint.\n",
    "   - `deployment_name` - Name of the deployment that will serve as the default deployment for the endpoint.\n",
    "- `description`- Description of the endpoint.\n",
    "\n",
    "### 3.1 Configure the endpoint\n",
    "\n",
    "First, let's create the endpoint that is going to host the batch deployments. To ensure that our endpoint name is unique, let's create a random suffix to append to it. \n",
    "\n",
    "> In general, you won't need to use this technique but you will use more meaningful names. Please skip the following cell if your case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "name_endpoint"
   },
   "outputs": [],
   "source": [
    "endpoint_name = \"heart-classifier-custom\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "# Creating a unique endpoint name by including a random suffix\n",
    "allowed_chars = string.ascii_lowercase + string.digits\n",
    "endpoint_suffix = \"\".join(random.choice(allowed_chars) for x in range(5))\n",
    "endpoint_name = f\"{endpoint_name}-{endpoint_suffix}\"\n",
    "\n",
    "print(f\"Endpoint name: {endpoint_name}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's configure the endpoint:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "configure_endpoint",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "endpoint = BatchEndpoint(\n",
    "    name=endpoint_name,\n",
    "    description=\"A heart condition classifier for batch inference\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Create the endpoint\n",
    "Using the `MLClient` created earlier, we will now create the Endpoint in the workspace. This command will start the endpoint creation and return a confirmation response while the endpoint creation continues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "create_endpoint",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "ml_client.batch_endpoints.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create a batch deployment\n",
    "\n",
    "A deployment is a set of resources required for hosting the model that does the actual inferencing. We will create a deployment for our endpoint using the `BatchDeployment` class.\n",
    "\n",
    "### 4.1 Creating an scoring script to work with the model\n",
    "\n",
    "The following scoring script uses the environment variable `AZURML_BI_OUTPUT` to generate predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "%%writefile code/batch_driver.py\n",
    "\n",
    "import os\n",
    "import pickle\n",
    "import glob\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "from typing import List\n",
    "\n",
    "\n",
    "def init():\n",
    "    global model\n",
    "    global output_path\n",
    "\n",
    "    # AZUREML_MODEL_DIR is an environment variable created during deployment\n",
    "    # It is the path to the model folder\n",
    "    # Please provide your model's folder name if there's one:\n",
    "    output_path = os.environ[\"AZUREML_BI_OUTPUT_PATH\"]\n",
    "    model_path = os.environ[\"AZUREML_MODEL_DIR\"]\n",
    "    model_file = glob.glob(f\"{model_path}/*/*.pkl\")[-1]\n",
    "\n",
    "    with open(model_file, \"rb\") as file:\n",
    "        model = pickle.load(file)\n",
    "\n",
    "\n",
    "def run(mini_batch: List[str]):\n",
    "    for file_path in mini_batch:\n",
    "        data = pd.read_csv(file_path)\n",
    "        pred = model.predict(data)\n",
    "\n",
    "        data[\"prediction\"] = pred\n",
    "\n",
    "        output_file_name = Path(file_path).stem\n",
    "        output_file_path = os.path.join(output_path, output_file_name + \".parquet\")\n",
    "        data.to_parquet(output_file_path)\n",
    "\n",
    "    return mini_batch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.2 Creating the compute\n",
    "\n",
    "Batch deployments can run on any Azure ML compute that already exists in the workspace. That means that multiple batch deployments can share the same compute infrastructure. In this example, we are going to work on an AzureML compute cluster called `cpu-cluster`. Let's verify the compute exists on the workspace or create it otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "configure_compute",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "compute_name = \"batch-cluster\"\n",
    "if not any(filter(lambda m: m.name == compute_name, ml_client.compute.list())):\n",
    "    compute_cluster = AmlCompute(\n",
    "        name=compute_name, description=\"amlcompute\", min_instances=0, max_instances=5\n",
    "    )\n",
    "    ml_client.begin_create_or_update(compute_cluster).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.3 Creating the environment\n",
    "\n",
    "Let's create the environment. In our case, our model uses `XGBoost`. We have a conda file already available with all the required dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "configure_environment",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "environment = Environment(\n",
    "    name=\"batch-mlflow-xgboost\",\n",
    "    conda_file=\"environment/conda.yaml\",\n",
    "    image=\"mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Configuring the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "configure_deployment",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "deployment = ModelBatchDeployment(\n",
    "    name=\"classifier-xgboost-custom\",\n",
    "    description=\"A heart condition classifier based on XGBoost which writes outputs in parquet format.\",\n",
    "    endpoint_name=endpoint.name,\n",
    "    model=model,\n",
    "    environment=environment,\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"code\",\n",
    "        scoring_script=\"batch_driver.py\",\n",
    "    ),\n",
    "    compute=compute_name,\n",
    "    settings=ModelBatchDeploymentSettings(\n",
    "        mini_batch_size=2,\n",
    "        instance_count=2,\n",
    "        max_concurrency_per_instance=2,\n",
    "        output_action=BatchDeploymentOutputAction.SUMMARY_ONLY,\n",
    "        retry_settings=BatchRetrySettings(max_retries=3, timeout=300),\n",
    "        logging_level=\"info\",\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5 Create the deployment\n",
    "Using the `MLClient` created earlier, we will now create the deployment in the workspace. This command will start the deployment creation and return a confirmation response while the deployment creation continues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "create_deployment",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "ml_client.batch_deployments.begin_create_or_update(deployment).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once created, let's configure this new deployment as the default one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "set_default_deployment"
   },
   "outputs": [],
   "source": [
    "endpoint = ml_client.batch_endpoints.get(endpoint_name)\n",
    "endpoint.defaults.deployment_name = deployment.name\n",
    "ml_client.batch_endpoints.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The default deployment is {endpoint.defaults.deployment_name}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "### 4.6 Testing the deployment\n",
    "\n",
    "Once the deployment is created, it is ready to recieve jobs."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### 4.6.1 Creating an input for the deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "configure_inputs",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "input = Input(\n",
    "    type=AssetTypes.URI_FOLDER,\n",
    "    path=\"https://azuremlexampledata.blob.core.windows.net/data/heart-disease-uci/data\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.6.2 Invoke the deployment\n",
    "\n",
    "Using the `MLClient` created earlier, we will get a handle to the endpoint. The endpoint can be invoked using the `invoke` command with the following parameters:\n",
    "- `name` - Name of the endpoint\n",
    "- `input_path` - Path where input data is present\n",
    "- `deployment_name` - Name of the specific deployment to test in an endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "start_batch_scoring_job",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "job = ml_client.batch_endpoints.invoke(\n",
    "    endpoint_name=endpoint.name, deployment_name=deployment.name, input=input\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### 4.6.3 Get the details of the invoked job\n",
    "\n",
    "Let us get details and logs of the invoked job:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "get_job",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "ml_client.jobs.get(job.name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can wait for the job to finish using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "stream_job_logs"
   },
   "outputs": [],
   "source": [
    "ml_client.jobs.stream(job.name)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.7 Exploring the results\n",
    "\n",
    "The deployment creates a child job that executes the scoring. We can get the details of it using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_job = list(ml_client.jobs.list(parent_job_name=job.name))[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Job name:\", scoring_job.name)\n",
    "print(\"Job status:\", scoring_job.status)\n",
    "print(\n",
    "    \"Job duration:\",\n",
    "    scoring_job.creation_context.last_modified_at\n",
    "    - scoring_job.creation_context.created_at,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "#### 4.7.1 Download the results\n",
    "\n",
    "The outputs generated by the deployment job will be placed in an output named `score`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "download_outputs",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "ml_client.jobs.download(name=scoring_job.name, download_path=\".\", output_name=\"score\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can read this data using pandas library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "name": "read_outputs",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import glob\n",
    "\n",
    "output_files = glob.glob(\"named-outputs/score/*.parquet\")\n",
    "score = pd.concat((pd.read_parquet(f) for f in output_files))\n",
    "score"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Clean up resources\n",
    "\n",
    "Clean-up the resources created. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "name": "delete_endpoint"
   },
   "outputs": [],
   "source": [
    "ml_client.batch_endpoints.begin_delete(endpoint_name).result()"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "amlv2"
  },
  "kernelspec": {
   "display_name": "previews",
   "language": "python",
   "name": "previews"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}