example/notebook/object_detection_model_deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy Object Detection Model Use ModelCI\n",
    "\n",
    "MMDetction is a well-known open source object detection toolbox based on PyTorch. You can refer to <https://arxiv.org/abs/1906.07155> for more details.\n",
    "\n",
    "By walking through this tutorial, you will be able to:\n",
    "\n",
    "- Load pretained MMDetction model\n",
    "- Convert MMDetction model into ONNX format \n",
    "- Register and retrieve models by ModelHub\n",
    "\n",
    "## 1. Prequisities\n",
    "\n",
    "In order to run this notebook, you have to upgrade your PyTorch version from 1.5.0 to 1.8.0.\n",
    " \n",
    "### 1.1 Installation of MMDetction\n",
    " \n",
    " Firstly you have to install MMDetction according to official instructions : <https://mmdetection.readthedocs.io/en/latest/get_started.html#installation> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Looking in links: https://download.openmmlab.com/mmcv/dist/cu102/torch1.8.0/index.html\n",
      "Collecting mmcv-full\n",
      "  Downloading https://download.openmmlab.com/mmcv/dist/cu102/torch1.8.0/mmcv_full-1.3.2-cp37-cp37m-manylinux1_x86_64.whl (16.6 MB)\n",
      "\u001b[K     |████████████████████████████████| 16.6 MB 231 kB/s \n",
      "\u001b[?25hRequirement already satisfied: yapf in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmcv-full) (0.31.0)\n",
      "Requirement already satisfied: Pillow in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmcv-full) (8.2.0)\n",
      "Requirement already satisfied: opencv-python>=3 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmcv-full) (4.5.1.48)\n",
      "Requirement already satisfied: addict in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmcv-full) (2.4.0)\n",
      "Requirement already satisfied: numpy in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmcv-full) (1.20.1)\n",
      "Requirement already satisfied: pyyaml in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmcv-full) (5.3.1)\n",
      "Installing collected packages: mmcv-full\n",
      "Successfully installed mmcv-full-1.3.2\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Obtaining file:///home/modelci/NTU/ML-Model-CI/example/notebook/mmdetection\n",
      "Requirement already satisfied: matplotlib in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmdet==2.11.0) (3.4.1)\n",
      "Requirement already satisfied: numpy in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmdet==2.11.0) (1.20.1)\n",
      "Requirement already satisfied: six in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmdet==2.11.0) (1.15.0)\n",
      "Requirement already satisfied: terminaltables in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmdet==2.11.0) (3.1.0)\n",
      "Requirement already satisfied: pycocotools in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from mmdet==2.11.0) (2.0.2)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from matplotlib->mmdet==2.11.0) (1.3.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from matplotlib->mmdet==2.11.0) (0.10.0)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from matplotlib->mmdet==2.11.0) (8.2.0)\n",
      "Requirement already satisfied: pyparsing>=2.2.1 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from matplotlib->mmdet==2.11.0) (2.4.7)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from matplotlib->mmdet==2.11.0) (2.8.1)\n",
      "Requirement already satisfied: cython>=0.27.3 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from pycocotools->mmdet==2.11.0) (0.29.23)\n",
      "Requirement already satisfied: setuptools>=18.0 in /home/modelci/miniconda3/envs/test/lib/python3.7/site-packages (from pycocotools->mmdet==2.11.0) (52.0.0.post20210125)\n",
      "Installing collected packages: mmdet\n",
      "  Attempting uninstall: mmdet\n",
      "    Found existing installation: mmdet 2.11.0\n",
      "    Uninstalling mmdet-2.11.0:\n",
      "      Successfully uninstalled mmdet-2.11.0\n",
      "  Running setup.py develop for mmdet\n",
      "Successfully installed mmdet\n"
     ]
    }
   ],
   "source": [
    "!pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.8.0/index.html\n",
    "!git clone https://github.com/open-mmlab/mmdetection.git\n",
    "!cd mmdetection && pip install -q -r requirements/build.txt && pip install -q -v -e ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Start ModelCI Service\n",
    "Then we can start our ModelCI service, you should at least set env variables once before starting. You can refer to [last notebook](https://github.com/cap-ntu/ML-Model-CI/blob/master/example/notebook/image_classification_model_deployment.ipynb) for more details.\n",
    "\n",
    "You should run the `modelci service init` command in your own terminal, so that the uvicorn server can be running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-29 10:28:02.981282: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "2021-05-29 10:28:32,723 - ml-modelci Docker Container Manager - INFO - Container name=mongo-43297 stared\n",
      "2021-05-29 10:28:41,534 - ml-modelci Docker Container Manager - INFO - Container name=cadvisor-61767 started.\n",
      "2021-05-29 10:28:50,751 - ml-modelci Docker Container Manager - INFO - Container name=dcgm-exporter-29046 started.\n",
      "2021-05-29 10:29:02,601 - ml-modelci Docker Container Manager - INFO - gpu-metrics-exporter-65952 stared\n",
      "2021-05-29 10:29:03,620 - modelci backend - INFO - Uvicorn server listening on http://localhost:8000, check full log at /home/modelci/tmp/modelci.log\n"
     ]
    }
   ],
   "source": [
    "!modelci service init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a minute, all the services will be started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build MMdetection Model\n",
    "### 2.1 Imports\n",
    "We should import the following functions:\n",
    "- preprocess_example_input: for generating tensor and meta info from example image file\n",
    "- build_model_from_cfg: for building model form config file and checkpoint file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mmdet.core.export import preprocess_example_input, build_model_from_cfg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Model Config\n",
    "\n",
    "We should either use a dict or config file for configuration of MMDetection model, to make things simple, we use a config file provided by MMDetection.\n",
    "\n",
    "Notice: \n",
    "\n",
    "- You may need to manually download pretrained model checkpoints from [MMDetection models zoo](https://github.com/open-mmlab/mmdetection/blob/master/docs/model_zoo.md).\n",
    "- Only a few MMdet models are able to converted into ONNX format, you can refer to [documentation](https://mmdetection.readthedocs.io/en/latest/tutorials/pytorch2onnx.html#list-of-supported-models-exportable-to-onnx) for more detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_file = 'mmdetection/configs/retinanet/retinanet_r50_fpn_1x_coco.py'\n",
    "checkpoint_file = 'retinanet_r50_fpn_1x_coco_20200130-c2398f9e.pth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Build Model\n",
    "Then we can build our MMdetection model based on the configuration above and the checkpoint file we already download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use load_from_local loader\n"
     ]
    }
   ],
   "source": [
    "model = build_model_from_cfg(config_file, checkpoint_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before conversion, we need to modify forward function to provide the necessary **kwargs parameters such as img_metas.\n",
    "\n",
    "In order to obtain valid bbox data during the onnx tracing process, we also need to use a tensor generated from image file as model input instead of random tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_config = {\n",
    "    'input_shape': (1,3,224,224),\n",
    "    'input_path': 'mmdetection/demo/demo.jpg',\n",
    "    'normalize_cfg': {\n",
    "        'mean': (123.675, 116.28, 103.53),\n",
    "        'std': (58.395, 57.12, 57.375)\n",
    "        }\n",
    "}\n",
    "one_img, one_meta = preprocess_example_input(input_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 224, 224])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "model.forward = partial(model.forward, img_metas=[[one_meta]], return_loss=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Save Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "torch_model_path = Path.home()/'.modelci/RetinaNet/PyTorch-PYTORCH/Object_Detection/1.pth'\n",
    "if not Path.is_dir(torch_model_path.parent):\n",
    "    os.makedirs(torch_model_path.parent, exist_ok=True)\n",
    "torch.save(model, torch_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Register the Model\n",
    "We can convert the pytorch model above into optimized formats, such as ONNX through modelci\n",
    "\n",
    "### 3.1 Construct MLModel Instance\n",
    "\n",
    "Here are some parameters need to be specified before model conversion.\n",
    "- inputs: The model inputs info\n",
    "- outputs: The model outputs info\n",
    "- metric: The evaludation metric data\n",
    "\n",
    "We can use YAML file to construct the MLModel Instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "weight: \"~/.modelci/RetinaNet/PyTorch-PYTORCH/Object_Detection/1.pth\"\n",
    "architecture: RetinaNet\n",
    "framework: PyTorch\n",
    "engine: PYTORCH\n",
    "version: 1\n",
    "dataset: COCO\n",
    "task: Object_Detection\n",
    "metric:\n",
    "  mAP: 0.365\n",
    "inputs:\n",
    "  - name: \"input\"\n",
    "    shape: [ -1, 3, 224, 224 ]\n",
    "    dtype: TYPE_FP32\n",
    "outputs:\n",
    "  - name: \"BBOX\"\n",
    "    shape: [ -1, 100, 5 ]\n",
    "    dtype: TYPE_FP32\n",
    "  - name: \"SCORE\"\n",
    "    shape: [ -1, 100 ]\n",
    "    dtype: TYPE_FP32\n",
    "convert: true\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Register\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-29 19:03:58.430792: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "{'data': {'id': ['60b21fac55849db050b511ba']}, 'status': True}\n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub publish -f ../retinanet.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-05-29 19:04:31.137077: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1\n",
      "\u001b[1mID       \u001b[0m  \u001b[1mArchitec…\u001b[0m  \u001b[1mFramework\u001b[0m  \u001b[1mVersion\u001b[0m  \u001b[1mPretrained\u001b[0m  \u001b[1mMetric\u001b[0m  \u001b[1mScore\u001b[0m  \u001b[1mTask       \u001b[0m\n",
      "                                          \u001b[1mDataset   \u001b[0m                            \n",
      "60b21fac…  RetinaNet  PyTorch    1        COCO        mAP     0.365  Object     \n",
      "                                                                     Detection  \n",
      "\u001b[1;38;5;43mConve…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mModel \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[1;38;5;43mInfo  \u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247mServi…\u001b[0m\u001b[38;5;247m  \u001b[0mPYTOR…  \u001b[38;5;247mStatus\u001b[0m\u001b[38;5;247m  \u001b[0m💔      \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0mshans…  \u001b[38;5;247mCreat…\u001b[0m\u001b[38;5;247m  \u001b[0m25      \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[38;5;247mEngine\u001b[0m\u001b[38;5;247m  \u001b[0m                Unkno…                          secon…                  \n",
      "                                                        ago                     \n",
      "\u001b[1;38;5;247mInputs\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0minput   \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        3,      \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "                                        224,                                    \n",
      "                                        224]                                    \n",
      "\u001b[1;38;5;247mOutpu…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0mBBOX    \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        100,    \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "                                        5]                                      \n",
      "\u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m  Name\u001b[0m\u001b[38;5;247m  \u001b[0mSCORE   \u001b[38;5;247m Shape\u001b[0m\u001b[38;5;247m  \u001b[0m[-1,    \u001b[38;5;247m  Data\u001b[0m\u001b[38;5;247m  \u001b[0mFP32    \u001b[38;5;247mFormat\u001b[0m\u001b[38;5;247m  \u001b[0mFORMAT_…\n",
      "                                        100]    \u001b[38;5;247m  Type\u001b[0m\u001b[38;5;247m  \u001b[0m                        \n",
      "\u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mProfi…\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n",
      "\u001b[1;38;5;43mResul…\u001b[0m\u001b[38;5;247m  \u001b[0m                                                                        \n",
      "\u001b[38;5;247m  N.A.\u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \u001b[38;5;247m      \u001b[0m\u001b[38;5;247m  \u001b[0m        \n"
     ]
    }
   ],
   "source": [
    "!modelci modelhub detail 60b21fac55849db050b511ba"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we could see, MLModelCI support auto conversion of PyTorch models into both torchscript and ONNX format, as a result.\n",
    "\n",
    "However, this model cannot be transformed into torchscript format, but supportive of ONNX format conversion, there could be serveral factors contributing to model conversion failture such as the model structure and code format.\n",
    "\n",
    "In this case, we failed to convert this model into ONNX format automatically, the error log is  `imgs must be a list, but got <class 'torch.Tensor'>`, then we need to specify the input image and manually convert this model in the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Convert the Model\n",
    "\n",
    "The following steps will convert the model we just registered into ONNX format. One thing to notice is we need to upgrade PyTorch version to 1.8.0 to support this conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelci.types.bo import IOShape\n",
    "from modelci.types.trtis_objects import ModelInputFormat\n",
    "from pathlib import Path\n",
    "# get ONNX model saved path\n",
    "onnx_model_path = Path.home() / '.modelci/RetinaNet/PyTorch-ONNX/Object_Detection/1.onnx'\n",
    "# specify inputs and outputs shape\n",
    "inputs = [IOShape([-1, 3, 204, 204], dtype=float, name='IMAGE', format=ModelInputFormat.FORMAT_NCHW)]\n",
    "outputs = [\n",
    "    IOShape([-1, 100, 5], dtype=float, name='BBOX'),\n",
    "    IOShape([-1, 100], dtype=float, name='SCORE')\n",
    "    ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/models/dense_heads/anchor_head.py:640: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  dtype=torch.long)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/models/dense_heads/anchor_head.py:647: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert cls_score.size()[-2:] == bbox_pred.size()[-2:]\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:62: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if k <= 0 or size <= 0:\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/models/dense_heads/anchor_head.py:661: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if nms_pre > 0:\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/models/dense_heads/anchor_head.py:671: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  _, topk_inds = max_scores.topk(nms_pre)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/bbox/coder/delta_xywh_bbox_coder.py:78: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert pred_bboxes.size(0) == bboxes.size(0)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/bbox/coder/delta_xywh_bbox_coder.py:80: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  assert pred_bboxes.size(1) == bboxes.size(1)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:118: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  iou_threshold = torch.tensor([iou_threshold], dtype=torch.float32)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:119: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  score_threshold = torch.tensor([score_threshold], dtype=torch.float32)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:123: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  nms_pre = torch.tensor(pre_top_k, device=scores.device, dtype=torch.long)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:147: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  batch_inds = torch.randint(batch_size, (num_fake_det, 1))\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:148: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  cls_inds = torch.randint(num_class, (num_fake_det, 1))\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:149: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  box_inds = torch.randint(num_box, (num_fake_det, 1))\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:180: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.\n",
      "  after_top_k, device=scores.device, dtype=torch.long)\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:183: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  if nms_after > 0:\n",
      "~/ML-Model-CI/example/notebook/mmdetection/mmdet/core/export/onnx_helper.py:184: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
      "  _, topk_inds = scores.topk(nms_after)\n",
      "/home/modelci/miniconda3/envs/test/lib/python3.7/site-packages/torch/onnx/symbolic_opset9.py:2607: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results.\n",
      "  \"If indices include negative values, the exported graph will produce incorrect results.\")\n",
      "2021-05-29 19:05:50,486 - converter - INFO - ONNX format converted successfully\n",
      "2021-05-29 19:05:50,486 - converter - INFO - ONNX format converted successfully\n",
      "2021-05-29 19:05:50,486 - converter - INFO - ONNX format converted successfully\n",
      "2021-05-29 19:05:50,486 - converter - INFO - ONNX format converted successfully\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from modelci.hub.converter import convert\n",
    "import torch\n",
    "convert(\n",
    "    model=model,\n",
    "    src_framework='pytorch',\n",
    "    dst_framework='onnx',\n",
    "    save_path=onnx_model_path,\n",
    "    inputs=inputs,\n",
    "    outputs=outputs,\n",
    "    model_input=[one_img],\n",
    "    opset=11,\n",
    "    optimize=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  5. Deploy the Model As a Service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Container: 3276321e72>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from modelci.hub.deployer.dispatcher import serve\n",
    "\n",
    "batch_size =1\n",
    "server_name = 'RetinaNet'\n",
    "\n",
    "serve(save_path=onnx_model_path, device='cuda:0', name=server_name, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3276321e7280   mlmodelci/onnx-serving:latest-gpu   \"/bin/sh -c 'python …\"   11 seconds ago   Up 2 seconds    0.0.0.0:8001->8000/tcp, 0.0.0.0:8002->8001/tcp   \u001b[01;31m\u001b[KRetinaNet\u001b[m\u001b[K\r\n"
     ]
    }
   ],
   "source": [
    "!docker ps | grep RetinaNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Profile the Model\n",
    "\n",
    "Firstly, we retrieve the model and build a client for ONNX serving platform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelci.persistence.service import ModelService\n",
    "model = ModelService.get_model_by_id(\"60b21fac55849db050b511ba\")\n",
    "model.inputs[0].dtype = 11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from modelci.hub.client.onnx_client import CVONNXClient\n",
    "test_img_bytes = torch.rand(3, 224, 224)\n",
    "onnx_client = CVONNXClient(test_img_bytes, model, batch_num=20, batch_size=1, asynchronous=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The we can init a profiler and start the profiling process, one thing to notice is that the batch size can only be 1 in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " latency: 0.2068 sec throughput: 4.8361 req/sec\n",
      " latency: 0.2623 sec throughput: 3.8130 req/sec\n",
      " latency: 0.2128 sec throughput: 4.6993 req/sec\n",
      " latency: 0.1491 sec throughput: 6.7060 req/sec\n",
      " latency: 0.2125 sec throughput: 4.7050 req/sec\n",
      " latency: 0.2243 sec throughput: 4.4590 req/sec\n",
      " latency: 0.1809 sec throughput: 5.5279 req/sec\n",
      " latency: 0.2041 sec throughput: 4.9000 req/sec\n",
      " latency: 0.2129 sec throughput: 4.6978 req/sec\n",
      " latency: 0.2573 sec throughput: 3.8865 req/sec\n",
      " latency: 0.1944 sec throughput: 5.1448 req/sec\n",
      " latency: 0.2100 sec throughput: 4.7616 req/sec\n",
      " latency: 0.2141 sec throughput: 4.6718 req/sec\n",
      " latency: 0.2286 sec throughput: 4.3745 req/sec\n",
      " latency: 0.2290 sec throughput: 4.3660 req/sec\n",
      " latency: 0.2660 sec throughput: 3.7590 req/sec\n",
      " latency: 0.2972 sec throughput: 3.3651 req/sec\n",
      " latency: 0.2097 sec throughput: 4.7695 req/sec\n",
      " latency: 0.2065 sec throughput: 4.8430 req/sec\n",
      " latency: 0.1932 sec throughput: 5.1764 req/sec\n",
      "\n",
      "\n",
      "testing device: GeForce MX110\n",
      "total batches: 20, batch_size: 1\n",
      "total latency: 4.3843653202056885 s\n",
      "total throughput: 4.561663670640867 req/sec\n",
      "50th-percentile latency: 0.21266722679138184 s\n",
      "95th-percentile latency: 0.26758338212966926 s\n",
      "99th-percentile latency: 0.2912516617774963 s\n",
      "total GPU memory: 2101870592.0 bytes\n",
      "average GPU memory usage percentage: 0.2575\n",
      "average GPU memory used: 541196288.0 bytes\n",
      "average GPU utilization: 21.9231%\n",
      "completed at 2021-05-29 19:08:34.261123\n"
     ]
    }
   ],
   "source": [
    "from modelci.hub.profiler import Profiler\n",
    "profiler = Profiler(model_info=model, server_name='RetinaNet', inspector=onnx_client)\n",
    "dps = profiler.diagnose(device='cuda:0')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}