examples/notebooks/AdvancedTopicsInInferenceAPIs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Topics in Inference APIs\n",
    "\n",
    "This tutorial explains a little bit more advanced topics about Inference APIs. The followings are the main topics:\n",
    "* How to specify a NPU device including *NPU core fusion*.\n",
    "* Asynchronous and non-blocking inference API\n",
    "\n",
    "## Prerequisites\n",
    "To follow this tutorial, please install the following requisites.\n",
    "\n",
    "First, you must install NPU driver, firmware, and runtime by following the instruction at [FuriosaAI Driver, Firmware, Runtime Installation Guide](https://furiosa-ai.github.io/docs/latest/ko/software/installation.html).\n",
    "\n",
    "Then, please install the following python packages:\n",
    "```sh\n",
    "pip install furiosa-sdk matplotlib mnist\n",
    "```\n",
    "Or, you can run the following command to install all dependent packages for all notebook examples at once:\n",
    "```sh\n",
    "pip install -r requirements.txt\n",
    "```\n",
    "\n",
    "And then, let's check if your NPU device is ready as following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m+\u001b[0m\u001b[0m------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m----------------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m-------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------------\u001b[0m\u001b[0m+\n",
      "\u001b[0m\u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[1mNPU \u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[1mName  \u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[1mFirmware      \u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[1mTemp.\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[1mPower \u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m\u001b[1mPCI-BDF     \u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m\n",
      "\u001b[0m\u001b[0m\u001b[0m+\u001b[0m\u001b[0m------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m----------------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m-------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------------\u001b[0m\u001b[0m+\n",
      "\u001b[0m\u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0mnpu0\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0mwarboy\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m1.7.0, 0a4411e\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m 38°C\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m2.22 W\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m \u001b[0m\u001b[0m\u001b[0m0000:49:00.0\u001b[0m \u001b[0m\u001b[0m|\u001b[0m\u001b[0m\n",
      "\u001b[0m\u001b[0m\u001b[0m+\u001b[0m\u001b[0m------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m----------------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m-------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------\u001b[0m\u001b[0m+\u001b[0m\u001b[0m--------------\u001b[0m\u001b[0m+\n",
      "\u001b[0m\u001b[0m"
     ]
    }
   ],
   "source": [
    "!furiosactl info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's make sure that your SDK is ready to run immediately by running the following command. If you see any error here, please follow the instructions at\n",
    "* [FuriosaAI Driver, Firmware, Runtime Installation Guide](https://furiosa-ai.github.io/docs/v0.5.0/ko/software/installation.html)\n",
    "* [Setting up a Python Environment](https://furiosa-ai.github.io/docs/v0.5.0/ko/software/python-sdk.html#python)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "libfuriosa_hal.so --- v0.11.0, built @ 43c901f\n",
      "Furiosa SDK Runtime 0.10.0-dev (rev: e80482f4) (libnux 0.9.0 062c7dd1f 2023-04-12T20:55:14Z)\n"
     ]
    }
   ],
   "source": [
    "!python -c \"from furiosa import runtime;print(runtime.__full_version__)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Specify a NPU device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may need to specify a NPU device for your applications in the following cases:\n",
    "* Case A: when you have more than one NPU devices\n",
    "* Case B: if you want to use individual PEs separately for smaller DNN applications or a single fusioned PE\n",
    "\n",
    "FuriosaAI SDK provides a couple of ways to specify a NPU device that your application uses. In this section, we are going to explain this feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding NPU IDs\n",
    "\n",
    "NPU IDs are used across all of furioaAI SDK components. So, you need to understand how a NPU device is represented as a single NPU ID string.\n",
    "\n",
    "`npu0`, `npu1`, `npuN` represents a single NPU device. The last digit number starts from 0, and can be increased sequentially as you add more NPUS to your machine. There are individual 2 PEs in a single NPU device. They are individually represented as `pe0` and `pe1`.\n",
    "\n",
    "Usually, a NPU ID can represent both a certain NPU device and certain PE(s). For example, if you have 2 NPU devices and want to list all available individual PEs, they are represented by:\n",
    "* `npu0pe0`\n",
    "* `npu0pe1`\n",
    "* `npu1pe0`\n",
    "* `npu1pe1`\n",
    "\n",
    "In Warboy, you are able to fuse 2 PEs belonging to the same NPU. 2 fused NPUs are represented by:\n",
    "* `npu0pe0-1`\n",
    "* `npu1pe0-1`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Shell Environment Variable to Specify a NPU device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of FuriosaAI SDKs recognize the shell environment variable `NPU_DEVNAME`. If you specify `NPU_DEVNAME` in your shell, your application will use the NPU device specified in `NPU_DEVNAME`. For example, you can specify a NPU device in your shell as following:\n",
    "\n",
    "```sh\n",
    "export NPU_DEVNAME=\"npu0pe0\"\n",
    "```\n",
    "\n",
    "Please note that a single NPU device is occupied while another application is using the device. So, you cannot run multiple applications with the same `NPU_DEVNAME` setting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Session Option"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python SDK, `Session` is the core class to run inferences, and it allows various options. One of the options is `device`, allowing a user to specific a NPU device for the session. If you are not familar with `Session`, you can learn from [Getting Started With Python SDK](GettingStartedWithPythonSDK.ipynb).\n",
    "\n",
    "For example, you can specify a NPU device when you create a `Session` object, as following:\n",
    "```python\n",
    "from furiosa.runtime import session\n",
    "sess = session.create('mnist-8.onnx', device=\"npu0pe0\")\n",
    "```\n",
    "\n",
    "Please note that a specific NPU device in Session option overrides the shell environment variable `NPU_DEVNAME`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Asynchronous Inference APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Asynchronous Inference API allows an user application to handle multiple inference requests through a single thread.\n",
    "\n",
    "To use asynchronous inference APIs, please call `session.create_async()` that create both `submitter` and `queue` instances as following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "libfuriosa_hal.so --- v0.11.0, built @ 43c901f\n",
      "Saving the compilation log into /home/hyunsik/.local/state/furiosa/logs/compile-20230413184105-357mhm.log\n",
      "Using furiosa-compiler 0.9.0 (rev: 062c7dd1f built at 2023-04-12T20:55:14Z)\n",
      "\u001b[1m\u001b[2m[1/6]\u001b[0m 🔍   Compiling from tflite to dfg\n",
      "Done in 0.014001096s\n",
      "\u001b[1m\u001b[2m[2/6]\u001b[0m 🔍   Compiling from dfg to ldfg\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m2023-04-13T23:41:05.755788Z\u001b[0m \u001b[32m INFO\u001b[0m \u001b[2mnux::npu\u001b[0m\u001b[2m:\u001b[0m Npu (npu0pe0-1) is being initialized\n",
      "\u001b[2m2023-04-13T23:41:05.759270Z\u001b[0m \u001b[32m INFO\u001b[0m \u001b[2mnux\u001b[0m\u001b[2m:\u001b[0m NuxInner create with pes: [PeId(0)]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Done in 19.000982s\n",
      "\u001b[1m\u001b[2m[3/6]\u001b[0m 🔍   Compiling from ldfg to cdfg\n",
      "Done in 0.000484564s\n",
      "\u001b[1m\u001b[2m[4/6]\u001b[0m 🔍   Compiling from cdfg to gir\n",
      "Done in 0.003669258s\n",
      "\u001b[1m\u001b[2m[5/6]\u001b[0m 🔍   Compiling from gir to lir\n",
      "Done in 0.001117423s\n",
      "\u001b[1m\u001b[2m[6/6]\u001b[0m 🔍   Compiling from lir to enf\n",
      "Done in 0.008298314s\n",
      "✨  Finished in 19.029139s\n"
     ]
    }
   ],
   "source": [
    "from furiosa.runtime import session\n",
    "\n",
    "model_path = \"models/MNIST_MobileNet_v2_uint8_quant_without_avgpool_softmax.tflite\"\n",
    "\n",
    "submitter, queue = session.create_async(model_path, \n",
    "                                        worker_num=1, \n",
    "                                        # Determine how many asynchronous requests you can submit\n",
    "                                        # without blocking.\n",
    "                                        input_queue_size=100,\n",
    "                                        output_queue_size=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `submitter` provides APIs to submit inference requests, and a `queue` provides APIs to receive the completed inference requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[TensorDesc(shape=(1, 28, 28, 1), dtype=UINT8, format=NHWC, size=784, len=784)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submitter.inputs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[TensorDesc(shape=(1, 10), dtype=UINT8, format=??, size=10, len=10)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submitter.outputs()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from furiosa.runtime import tensor\n",
    "import numpy as np\n",
    "import mnist\n",
    "import random\n",
    "\n",
    "train_images = mnist.train_images()\n",
    "\n",
    "# Submit the inference requests asynchronously\n",
    "for i in range(0, 5):\n",
    "    idx = random.randint(0, 59999)\n",
    "    input = np.array(train_images[idx:idx+1].reshape(1, 28, 28, 1), np.uint8)\n",
    "    submitter.submit(input, context=idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context: 49148, Predict: 1\n",
      "Context: 856, Predict: 1\n",
      "Context: 17638, Predict: 1\n",
      "Context: 32495, Predict: 1\n",
      "Context: 4727, Predict: 2\n"
     ]
    }
   ],
   "source": [
    "# Receive the results asynchronously\n",
    "for i in range(0, 5):\n",
    "    context, outputs = queue.recv(100) # 100 is timeout. If None, queue.recv() will be blocking.\n",
    "    print(f\"Context: {context}, Predict: {np.argmax(outputs[0].numpy())}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to close `queue` and `submitter` after you use the asynchronous session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2m2023-04-13T23:41:25.541872Z\u001b[0m \u001b[32m INFO\u001b[0m \u001b[2mnux::npu\u001b[0m\u001b[2m:\u001b[0m NPU (npu0pe0-1) has been destroyed\n"
     ]
    }
   ],
   "source": [
    "if queue:\n",
    "    queue.close()\n",
    "if submitter:\n",
    "    submitter.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}