experimental/deploy-triton/1.densenet-local.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy to Triton Inference Server locally\n",
    "\n",
    "description: (preview) deploy an image classification model trained on densenet locally via Triton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that this Public Preview release is subject to the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nvidia-pyindex\n",
    "!pip install --upgrade tritonclient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "ws"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download model\n",
    "\n",
    "It's important that your model have this directory structure for Triton Inference Server to be able to load it. [Read more about the directory structure that Triton expects](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_repository.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from pathlib import Path\n",
    "from src.model_utils import download_triton_models, delete_triton_models\n",
    "\n",
    "prefix = Path(\".\")\n",
    "download_triton_models(prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register model\n",
    "\n",
    "A registered model is a logical container stored in the cloud, containing all files located at `model_path`, which is associated with a version number and other metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "model_path = \"models\"\n",
    "\n",
    "model = Model.register(\n",
    "    model_path=model_path,\n",
    "    model_name=\"densenet-onnx-example\",\n",
    "    tags={\"area\": \"Image classification\", \"type\": \"classification\"},\n",
    "    description=\"Image classification trained on Imagenet Dataset\",\n",
    "    workspace=ws,\n",
    ")\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy webservice\n",
    "\n",
    "In this case we deploy to the local compute, but for other options, see [our documentation](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-and-where?tabs=azcli).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.webservice import LocalWebservice\n",
    "from azureml.core import Environment\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core.model import InferenceConfig\n",
    "from random import randint\n",
    "\n",
    "service_name = \"triton-densenet-onnx-local\" + str(randint(10000, 99999))\n",
    "my_env = Environment.get(ws, \"AzureML-Triton\").clone(\"my-triton\")\n",
    "\n",
    "for pip_package in [\"pillow\", \"numpy\"]:\n",
    "    my_env.python.conda_dependencies.add_pip_package(pip_package)\n",
    "\n",
    "my_env.environment_variables[\"WORKER_COUNT\"] = \"1\"\n",
    "\n",
    "inference_config = InferenceConfig(\n",
    "    # this entry script is where we dispatch a call to the Triton server\n",
    "    source_directory=\"src\",\n",
    "    entry_script=\"score_densenet.py\",\n",
    "    environment=my_env,\n",
    ")\n",
    "\n",
    "config = LocalWebservice.deploy_configuration(port=6789)\n",
    "\n",
    "service = Model.deploy(\n",
    "    workspace=ws,\n",
    "    name=service_name,\n",
    "    models=[model],\n",
    "    inference_config=inference_config,\n",
    "    deployment_config=config,\n",
    "    overwrite=True,\n",
    ")\n",
    "\n",
    "service.wait_for_deployment(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(service.get_logs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the webservice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "headers = {\"Content-Type\": \"application/octet-stream\"}\n",
    "\n",
    "test_sample = requests.get(\"https://aka.ms/peacock-pic\", allow_redirects=True).content\n",
    "resp = requests.post(service.scoring_uri, data=test_sample, headers=headers)\n",
    "print(resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the webservice and the downloaded model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "service.delete()\n",
    "delete_triton_models(prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "Try changing the deployment configuration to [deploy to Azure Kubernetes Service](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-azure-kubernetes-service?tabs=python) for higher availability and better scalability."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8",
   "language": "python",
   "name": "python3.8"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  },
  "name": "deploy-densenet-local",
  "task": "Use the high-performance Triton Inference Server with Azure Machine Learning"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}