tutorials/deploy-triton/2.bidaf-aks-v100.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy to Triton Inference Server on AKS\n",
    "\n",
    "description: (preview) deploy a bi-directional attention flow (bidaf) Q&A model to V100s on AKS via Triton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that this Public Preview release is subject to the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade https://aka.ms/triton/packages/tritonclientutils-2.2.0-py3-none-any.whl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "ws"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download model\n",
    "\n",
    "It's important that your model have this directory structure for Triton Inference Server to be able to load it. [Read more about the directory structure that Triton expects](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_repository.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import git\n",
    "import sys\n",
    "from pathlib import Path\n",
    "\n",
    "# get the root of the repo\n",
    "prefix = Path(git.Repo(\".\", search_parent_directories=True).working_tree_dir)\n",
    "\n",
    "# Enables us to import helper functions as Python modules\n",
    "path_to_insert = prefix.joinpath(\"code\", \"deploy\", \"triton\").__str__()\n",
    "if path_to_insert not in sys.path:\n",
    "    sys.path.insert(1, path_to_insert)\n",
    "\n",
    "from model_utils import download_triton_models\n",
    "\n",
    "download_triton_models(prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "model_path = prefix.joinpath(\"models\", \"triton\")\n",
    "\n",
    "model = Model.register(\n",
    "    model_path=model_path,\n",
    "    model_name=\"bidaf-9-tutorial\",\n",
    "    tags={\"area\": \"Natural language processing\", \"type\": \"Question-answering\"},\n",
    "    description=\"Question answering from ONNX model zoo\",\n",
    "    workspace=ws,\n",
    "    model_framework=Model.Framework.MULTI,\n",
    ")\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy webservice\n",
    "\n",
    "Deploy to a pre-created [AksCompute](https://docs.microsoft.com/python/api/azureml-core/azureml.core.compute.aks.akscompute?view=azure-ml-py#provisioning-configuration-agent-count-none--vm-size-none--ssl-cname-none--ssl-cert-pem-file-none--ssl-key-pem-file-none--location-none--vnet-resourcegroup-name-none--vnet-name-none--subnet-name-none--service-cidr-none--dns-service-ip-none--docker-bridge-cidr-none--cluster-purpose-none--load-balancer-type-none-) named `aks-gpu-deploy`. For other options, see [our documentation](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-and-where?tabs=azcli).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AksWebservice\n",
    "from azureml.core.model import InferenceConfig\n",
    "from random import randint\n",
    "\n",
    "service_name = \"triton-bidaf-9\" + str(randint(10000, 99999))\n",
    "\n",
    "config = AksWebservice.deploy_configuration(\n",
    "    compute_target_name=\"aks-gpu-deploy\",\n",
    "    gpu_cores=1,\n",
    "    cpu_cores=1,\n",
    "    memory_gb=4,\n",
    "    auth_enabled=True,\n",
    ")\n",
    "\n",
    "service = Model.deploy(\n",
    "    workspace=ws,\n",
    "    name=service_name,\n",
    "    models=[model],\n",
    "    deployment_config=config,\n",
    "    overwrite=True,\n",
    ")\n",
    "\n",
    "service.wait_for_deployment(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the webservice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade nltk geventhttpclient python-rapidjson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service.get_keys()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!curl -v $service.scoring_uri/v2/health/ready #-H \"Authorization: Bearer 1Y3cvc6wHsXI5wKCWduRmmptRNadQeSR\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Using a modified version of tritonhttpclient for Preview, PR is out for review\n",
    "# https://github.com/triton-inference-server/server/pull/2047\n",
    "import tritonhttpclient\n",
    "from tritonclientutils import triton_to_np_dtype\n",
    "\n",
    "from bidaf_utils import preprocess, postprocess\n",
    "\n",
    "key = service.get_keys()[0]\n",
    "headers = {}\n",
    "headers[\"Authorization\"] = f\"Bearer {key}\"\n",
    "\n",
    "triton_client = tritonhttpclient.InferenceServerClient(service.scoring_uri)\n",
    "\n",
    "context = \"A quick brown fox jumped over the lazy dog.\"\n",
    "query = \"Which animal was lower?\"\n",
    "\n",
    "model_name = \"bidaf-9\"\n",
    "\n",
    "model_metadata = triton_client.get_model_metadata(\n",
    "    model_name=model_name, headers=headers\n",
    ")\n",
    "\n",
    "input_meta = model_metadata[\"inputs\"]\n",
    "output_meta = model_metadata[\"outputs\"]\n",
    "\n",
    "# We use the np.object data type for string data\n",
    "np_dtype = triton_to_np_dtype(input_meta[0][\"datatype\"])\n",
    "cw, cc = preprocess(context, np_dtype)\n",
    "qw, qc = preprocess(query, np_dtype)\n",
    "\n",
    "input_mapping = {\n",
    "    \"query_word\": qw,\n",
    "    \"query_char\": qc,\n",
    "    \"context_word\": cw,\n",
    "    \"context_char\": cc,\n",
    "}\n",
    "\n",
    "inputs = []\n",
    "outputs = []\n",
    "\n",
    "# Populate the inputs array\n",
    "for in_meta in input_meta:\n",
    "    input_name = in_meta[\"name\"]\n",
    "    data = input_mapping[input_name]\n",
    "\n",
    "    input = tritonhttpclient.InferInput(\n",
    "        input_name, data.shape, in_meta[\"datatype\"]\n",
    "    )\n",
    "\n",
    "    input.set_data_from_numpy(data, binary_data=False)\n",
    "    inputs.append(input)\n",
    "\n",
    "# Populate the outputs array\n",
    "for out_meta in output_meta:\n",
    "    output_name = out_meta[\"name\"]\n",
    "    output = tritonhttpclient.InferRequestedOutput(\n",
    "        output_name, binary_data=False\n",
    "    )\n",
    "    outputs.append(output)\n",
    "\n",
    "# Run inference\n",
    "res = triton_client.infer(\n",
    "    model_name,\n",
    "    inputs,\n",
    "    request_id=\"0\",\n",
    "    outputs=outputs,\n",
    "    model_version=\"1\",\n",
    "    headers=headers,\n",
    ")\n",
    "\n",
    "result = postprocess(context_words=cw, answer=res)\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the webservice and the downloaded model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from model_utils import delete_triton_models\n",
    "\n",
    "service.delete()\n",
    "delete_triton_models(prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "Try reading [our documentation](https://aka.ms/triton-aml-docs) to use Triton with your own models or check out the other notebooks in this folder for ways to do pre- and post-processing on the server. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8",
   "language": "python",
   "name": "python3.8"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  },
  "name": "deploy-bidaf-aks",
  "task": "Use the high-performance Triton Inference Server with Azure Machine Learning"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}