experimental/deploy-triton/2.bidaf-aks-v100.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy to Triton Inference Server on AKS\n",
    "\n",
    "description: (preview) deploy a bi-directional attention flow (bidaf) Q&A model to V100s on AKS via Triton"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please note that this Public Preview release is subject to the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nvidia-pyindex\n",
    "!pip install --upgrade tritonclient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "ws"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download model\n",
    "\n",
    "It's important that your model have this directory structure for Triton Inference Server to be able to load it. [Read more about the directory structure that Triton expects](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/model_repository.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from src.model_utils import download_triton_models, delete_triton_models\n",
    "from pathlib import Path\n",
    "\n",
    "prefix = Path(\".\")\n",
    "download_triton_models(prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "model_path = prefix.joinpath(\"models\")\n",
    "\n",
    "model = Model.register(\n",
    "    model_path=model_path,\n",
    "    model_name=\"bidaf-9-tutorial\",\n",
    "    tags={\"area\": \"Natural language processing\", \"type\": \"Question-answering\"},\n",
    "    description=\"Question answering from ONNX model zoo\",\n",
    "    workspace=ws,\n",
    "    model_framework=Model.Framework.MULTI,\n",
    ")\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy webservice\n",
    "\n",
    "Deploy to a pre-created [AksCompute](https://docs.microsoft.com/python/api/azureml-core/azureml.core.compute.aks.akscompute?view=azure-ml-py#provisioning-configuration-agent-count-none--vm-size-none--ssl-cname-none--ssl-cert-pem-file-none--ssl-key-pem-file-none--location-none--vnet-resourcegroup-name-none--vnet-name-none--subnet-name-none--service-cidr-none--dns-service-ip-none--docker-bridge-cidr-none--cluster-purpose-none--load-balancer-type-none-) named `aks-gpu-deploy`. For other options, see [our documentation](https://docs.microsoft.com/azure/machine-learning/how-to-deploy-and-where?tabs=azcli).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.core.webservice import AksWebservice\n",
    "from azureml.core.model import InferenceConfig\n",
    "from random import randint\n",
    "\n",
    "service_name = \"triton-bidaf-9\" + str(randint(10000, 99999))\n",
    "\n",
    "config = AksWebservice.deploy_configuration(\n",
    "    compute_target_name=\"aks-gpu-deploy\",\n",
    "    gpu_cores=1,\n",
    "    cpu_cores=1,\n",
    "    memory_gb=4,\n",
    "    auth_enabled=True,\n",
    ")\n",
    "\n",
    "service = Model.deploy(\n",
    "    workspace=ws,\n",
    "    name=service_name,\n",
    "    models=[model],\n",
    "    deployment_config=config,\n",
    "    overwrite=True,\n",
    ")\n",
    "\n",
    "service.wait_for_deployment(show_output=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(service.get_logs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the webservice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade nltk geventhttpclient python-rapidjson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "service_key = service.get_keys()[0]\n",
    "scoring_uri = service.scoring_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!curl -v $scoring_uri/v2/health/ready -H 'Authorization: Bearer '\"$service_key\"''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Using a modified version of tritonhttpclient for Preview, PR is out for review\n",
    "# https://github.com/triton-inference-server/server/pull/2047\n",
    "import tritonclient.http as tritonhttpclient\n",
    "from tritonclientutils import triton_to_np_dtype\n",
    "\n",
    "from src.bidaf_utils import preprocess, postprocess\n",
    "\n",
    "key = service.get_keys()[0]\n",
    "headers = {}\n",
    "headers[\"Authorization\"] = f\"Bearer {key}\"\n",
    "\n",
    "triton_client = tritonhttpclient.InferenceServerClient(service.scoring_uri[7:])\n",
    "\n",
    "context = \"A quick brown fox jumped over the lazy dog.\"\n",
    "query = \"Which animal was lower?\"\n",
    "\n",
    "model_name = \"bidaf-9\"\n",
    "\n",
    "model_metadata = triton_client.get_model_metadata(\n",
    "    model_name=model_name, headers=headers\n",
    ")\n",
    "\n",
    "input_meta = model_metadata[\"inputs\"]\n",
    "output_meta = model_metadata[\"outputs\"]\n",
    "\n",
    "# We use the np.object data type for string data\n",
    "np_dtype = triton_to_np_dtype(input_meta[0][\"datatype\"])\n",
    "cw, cc = preprocess(context, np_dtype)\n",
    "qw, qc = preprocess(query, np_dtype)\n",
    "\n",
    "input_mapping = {\n",
    "    \"query_word\": qw,\n",
    "    \"query_char\": qc,\n",
    "    \"context_word\": cw,\n",
    "    \"context_char\": cc,\n",
    "}\n",
    "\n",
    "inputs = []\n",
    "outputs = []\n",
    "\n",
    "# Populate the inputs array\n",
    "for in_meta in input_meta:\n",
    "    input_name = in_meta[\"name\"]\n",
    "    data = input_mapping[input_name]\n",
    "\n",
    "    input = tritonhttpclient.InferInput(input_name, data.shape, in_meta[\"datatype\"])\n",
    "\n",
    "    input.set_data_from_numpy(data, binary_data=False)\n",
    "    inputs.append(input)\n",
    "\n",
    "# Populate the outputs array\n",
    "for out_meta in output_meta:\n",
    "    output_name = out_meta[\"name\"]\n",
    "    output = tritonhttpclient.InferRequestedOutput(output_name, binary_data=False)\n",
    "    outputs.append(output)\n",
    "\n",
    "# Run inference\n",
    "res = triton_client.infer(\n",
    "    model_name,\n",
    "    inputs,\n",
    "    request_id=\"0\",\n",
    "    outputs=outputs,\n",
    "    model_version=\"1\",\n",
    "    headers=headers,\n",
    ")\n",
    "\n",
    "result = postprocess(context_words=cw, answer=res)\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the webservice and the downloaded model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "service.delete()\n",
    "delete_triton_models(prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next steps\n",
    "\n",
    "Try reading [our documentation](https://aka.ms/triton-aml-docs) to use Triton with your own models or check out the other notebooks in this folder for ways to do pre- and post-processing on the server. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8",
   "language": "python",
   "name": "python3.8"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  },
  "name": "deploy-bidaf-aks",
  "task": "Use the high-performance Triton Inference Server with Azure Machine Learning"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}