notebooks/3_translation_ONNX_GraphPipe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Model Translation and Deployment: Serve an ONNX Model via GraphPipe\n",
    "\n",
    "Also see the [GraphPipe User Guide](https://oracle.github.io/graphpipe/#/guide/user-guide/overview)\n",
    "\n",
    "In this section, I will show how to import and export from an onnx model. I will also show how to set up a model server with GraphPipe and test it using the GraphPipe client.\n",
    "\n",
    "Make sure to have the following technologies available on your machine:\n",
    "* [Docker](https://www.docker.com/)\n",
    "* [GraphPipe Docker images for TensorFlow and ONNX](https://oracle.github.io/graphpipe/#/guide/servers/installation)\n",
    "* [GraphPipe Client](https://oracle.github.io/graphpipe/#/guide/clients/overview)\n",
    "* [ONNX](https://github.com/onnx/onnx)\n",
    "* [ONNX TensorFlow connector](https://github.com/onnx/tensorflow-onnx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "import numpy as np\n",
    "import onnx\n",
    "from onnx_tf.backend import prepare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path = '../models/dnn_model_pt.onnx'\n",
    "dnn_model_onnx = onnx.load(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a .json-file describing the network inputs as required by `graphpipe-onnx`\n",
    "\n",
    "Unfortunately, there is insufficient documentation on how to set up the value_inputs.json, but we just follow the structure for the exemplary [Squeezenet input](https://oracle.github.io/graphpipe/models/squeezenet.value_inputs.json) assuming that the outer list annotates the no. of examples per request and the inner list describes the dimensions of the input:\n",
    "\n",
    "`{\"data_0\": [1, [1, 3, 227, 227]]}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"flattened_rescaled_img_28x28\": [1, [1, 784]]}\n"
     ]
    }
   ],
   "source": [
    "input_name = dnn_model_onnx.graph.node[0].input[0]\n",
    "graphpipe_value_inputs = {input_name: [1, [1, 28*28]]}\n",
    "json.dump(graphpipe_value_inputs,\n",
    "          open('../models/dnn_model_pt.value_inputs.json', 'w'))\n",
    "print(json.dumps(graphpipe_value_inputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphPipe with ONNX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to [specify the volume correctly](https://docs.docker.com/storage/volumes/)!\n",
    "\n",
    "I execute the docker commands below from the root of the repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "docker run -it --rm \\\n",
    "    -v \"$PWD/models:/models/\" \\\n",
    "    -p 9000:9000 \\\n",
    "    sleepsonthefloor/graphpipe-onnx:cpu \\\n",
    "    --value-inputs=/models/dnn_model_pt.value_inputs.json \\\n",
    "    --model=../models/dnn_model_pt.onnx \\\n",
    "    --listen=0.0.0.0:9000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Unfortunately, this failes with the following log:**\n",
    "\n",
    "```\n",
    "INFO[0000] Setting MKL_NUM_THREADS=4.  You can override this variable in your environment. \n",
    "INFO[0000] Starting graphpipe-caffe2 version 1.0.0.4.0a1675f.dev (built from sha 0a1675f) \n",
    "WARNING: Logging before InitGoogleLogging() is written to STDERR\n",
    "E0908 12:45:33.273241     1 c2_api.cc:309] Binary compiled without cuda support.  Using cpu backend.\n",
    "INFO[0000] Loading file %!(EXTRA string=../models/dnn_model_pt.value_inputs.json) \n",
    "INFO[0000] Loading file %!(EXTRA string=../models/dnn_model_pt.onnx) \n",
    "E0908 12:45:33.287909     1 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.\n",
    "E0908 12:45:33.288249     1 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.\n",
    "E0908 12:45:33.288272     1 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.\n",
    "terminate called after throwing an instance of 'caffe2::EnforceNotMet'\n",
    "  what():  [enforce fail at tensor.h:147] values.size() == size_. 784 vs 1229312 \n",
    "*** Aborted at 1536410733 (unix time) try \"date -d @1536410733\" if you are using GNU date ***\n",
    "PC: @     0x7ff4d3c2b428 gsignal\n",
    "*** SIGABRT (@0x1) received by PID 1 (TID 0x7ff4d5c08b40) from PID 1; stack trace: ***\n",
    "    @     0x7ff4d4569390 (unknown)\n",
    "    @     0x7ff4d3c2b428 gsignal\n",
    "    @     0x7ff4d3c2d02a abort\n",
    "    @     0x7ff4d426584d __gnu_cxx::__verbose_terminate_handler()\n",
    "    @     0x7ff4d42636b6 (unknown)\n",
    "    @     0x7ff4d4263701 std::terminate()\n",
    "    @     0x7ff4d4263919 __cxa_throw\n",
    "    @           0x73e86e caffe2::Tensor<>::Tensor<>()\n",
    "    @           0x737ada _initialize()\n",
    "    @           0x738b9f c2_engine_initialize_onnx\n",
    "    @           0x733a8f _cgo_e12a854003a1_Cfunc_c2_engine_initialize_onnx\n",
    "    @           0x45f340 runtime.asmcgocall\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Trying to use online resources, we gave it a second try:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "docker run -it --rm \\\n",
    "      -e https_proxy=${https_proxy} \\\n",
    "      -p 9000:9000 \\\n",
    "      sleepsonthefloor/graphpipe-onnx:cpu \\\n",
    "      --value-inputs=https://raw.githubusercontent.com/squall-1002/test_graphpipe/master/dnn_model_pt.value_inputs.json \\\n",
    "      --model=https://github.com/squall-1002/test_graphpipe/blob/master/dnn_model_pt.onnx \\\n",
    "      --listen=0.0.0.0:9000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "INFO[0000] Setting MKL_NUM_THREADS=4.  You can override this variable in your environment. \n",
    "INFO[0000] Starting graphpipe-caffe2 version 1.0.0.4.0a1675f.dev (built from sha 0a1675f) \n",
    "WARNING: Logging before InitGoogleLogging() is written to STDERR\n",
    "E0908 17:39:21.048094     1 c2_api.cc:309] Binary compiled without cuda support.  Using cpu backend.\n",
    "INFO[0000] Loading file %!(EXTRA string=../models/dnn_model_pt.value_inputs.json) \n",
    "FATA[0000] Could not load value_input: open ../models/dnn_model_pt.value_inputs.json: no such file or directory \n",
    "(base) Marcels-MBP:notebooks mkurovski$ docker run -it --rm \\\n",
    ">       -e https_proxy=${https_proxy} \\\n",
    ">       -p 9000:9000 \\\n",
    ">       sleepsonthefloor/graphpipe-onnx:cpu \\\n",
    ">       --value-inputs=https://raw.githubusercontent.com/squall-1002/test_graphpipe/master/dnn_model_pt.value_inputs.json \\\n",
    ">       --model=https://github.com/squall-1002/test_graphpipe/blob/master/dnn_model_pt.onnx \\\n",
    ">       --listen=0.0.0.0:9000\n",
    "INFO[0000] Setting MKL_NUM_THREADS=4.  You can override this variable in your environment. \n",
    "INFO[0000] Starting graphpipe-caffe2 version 1.0.0.4.0a1675f.dev (built from sha 0a1675f) \n",
    "WARNING: Logging before InitGoogleLogging() is written to STDERR\n",
    "E0908 17:42:27.937944     1 c2_api.cc:309] Binary compiled without cuda support.  Using cpu backend.\n",
    "INFO[0000] Loading file %!(EXTRA string=https://raw.githubusercontent.com/squall-1002/test_graphpipe/master/dnn_model_pt.value_inputs.json) \n",
    "INFO[0000] Loading file %!(EXTRA string=https://github.com/squall-1002/test_graphpipe/blob/master/dnn_model_pt.onnx) \n",
    "E0908 17:42:28.951375    14 init_intrinsics_check.cc:43] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.\n",
    "E0908 17:42:28.954814    14 init_intrinsics_check.cc:43] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.\n",
    "E0908 17:42:28.954874    14 init_intrinsics_check.cc:43] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.\n",
    "terminate called after throwing an instance of 'onnx_c2::checker::ValidationError'\n",
    "  what():  The model does not have an ir_version set properly.\n",
    "*** Aborted at 1536428548 (unix time) try \"date -d @1536428548\" if you are using GNU date ***\n",
    "PC: @     0x7f5a50bc6428 gsignal\n",
    "*** SIGABRT (@0x1) received by PID 1 (TID 0x7f5a2cb92700) from PID 1; stack trace: ***\n",
    "    @     0x7f5a51504390 (unknown)\n",
    "    @     0x7f5a50bc6428 gsignal\n",
    "    @     0x7f5a50bc802a abort\n",
    "    @     0x7f5a5120084d __gnu_cxx::__verbose_terminate_handler()\n",
    "    @     0x7f5a511fe6b6 (unknown)\n",
    "    @     0x7f5a511fe701 std::terminate()\n",
    "    @     0x7f5a511fe919 __cxa_throw\n",
    "    @     0x7f5a5236945a onnx_c2::checker::check_model()\n",
    "    @     0x7f5a51d90f7f caffe2::onnx::Caffe2Backend::Prepare()\n",
    "    @           0x738b4a c2_engine_initialize_onnx\n",
    "    @           0x733a8f _cgo_e12a854003a1_Cfunc_c2_engine_initialize_onnx\n",
    "    @           0x45f340 runtime.asmcgocall\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GraphPipe with TensorFlow Serving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from onnx_tf.backend import prepare\n",
    "dnn_model_tf = prepare(dnn_model_onnx, device='cpu')\n",
    "dnn_model_tf.export_graph('../models/dnn_model_tf.pb')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "docker run -it --rm \\\n",
    "      -v \"$PWD/models:/models/\" \\\n",
    "      -p 9000:9000 \\\n",
    "      sleepsonthefloor/graphpipe-tf:cpu \\\n",
    "      --model=/models/dnn_model_tf.pb \\\n",
    "      --listen=0.0.0.0:9000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "INFO[0000] Starting graphpipe-tf version 1.0.0.10.f235920 (built from sha f235920) \n",
    "2018-09-09 13:15:38.235084: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
    "2018-09-09 13:15:38.236119: I tensorflow/core/common_runtime/process_util.cc:63] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
    "INFO[0000] Model hash is 'e3ee2541642a8ef855d49ba387cee37d5678901f95e8aa0d3ed9a355cf464fb2' \n",
    "INFO[0000] Using default inputs [flattened_rescaled_img_28x28:0] \n",
    "INFO[0000] Using default outputs [Softmax:0]            \n",
    "INFO[0000] Listening on '0.0.0.0:9000' \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the Prediction Accuracy by sending some test queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from emnist_dl2prod.utils import load_emnist, get_emnist_mapping\n",
    "from graphpipe import remote"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2018-09-09 15:16:17] INFO:emnist_dl2prod.utils:Loading train and test data from emnist_data/emnist-byclass.mat\n"
     ]
    }
   ],
   "source": [
    "x_train, y_train, x_test, y_test, _ = load_emnist('emnist_data/')\n",
    "mapping = get_emnist_mapping()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Label / True Label: 2 == z ? - False !\n",
      "Predicted Label / True Label: r == r ? - True !\n",
      "Predicted Label / True Label: 3 == 3 ? - True !\n",
      "Predicted Label / True Label: h == h ? - True !\n",
      "Predicted Label / True Label: 2 == 2 ? - True !\n",
      "Predicted Label / True Label: j == j ? - True !\n",
      "Predicted Label / True Label: 5 == 5 ? - True !\n",
      "Predicted Label / True Label: 2 == 2 ? - True !\n",
      "Predicted Label / True Label: 7 == 7 ? - True !\n",
      "Predicted Label / True Label: 8 == 8 ? - True !\n"
     ]
    }
   ],
   "source": [
    "n_test_instances = 10\n",
    "n_test = x_test.shape[0]\n",
    "for _ in range(n_test_instances):\n",
    "    idx = np.random.randint(n_test)\n",
    "    # flatten and normalize test image\n",
    "    x = x_test[idx].reshape(1, -1)/255\n",
    "    y = y_test[idx][0]\n",
    "    softmax_pred = remote.execute(\"http://127.0.0.1:9000\", x)\n",
    "    pred_class = mapping[np.argmax(softmax_pred)]\n",
    "    true_class = mapping[y_test[idx][0]]\n",
    "    print(\"Predicted Label / True Label: {} == {} ? - {} !\".format(\n",
    "        pred_class, true_class, (pred_class==true_class)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The backend tell us the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "INFO[0042] Request for / took 191.793305ms              \n",
    "INFO[0042] Request for / took 3.571981ms                \n",
    "INFO[0042] Request for / took 1.388616ms                \n",
    "INFO[0042] Request for / took 1.93661ms                 \n",
    "INFO[0042] Request for / took 2.060367ms                \n",
    "INFO[0042] Request for / took 2.004791ms                \n",
    "INFO[0042] Request for / took 4.566884ms                \n",
    "INFO[0042] Request for / took 2.421152ms                \n",
    "INFO[0042] Request for / took 713.579µs                 \n",
    "INFO[0042] Request for / took 810.051µs \n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}