testing/benchmarking/python/python_serialization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Wrapper Benchmarking\n",
    "\n",
    "## Prequisites\n",
    "\n",
    " * An authenticated K8S cluster with istio and Seldon Core installed\n",
    "   * You can use the ansible seldon-core playbook at https://github.com/SeldonIO/ansible-k8s-collection\n",
    " * vegeta and ghz benchmarking tools\n",
    " \n",
    " Port forward to istio\n",
    " \n",
    " ```\n",
    " kubectl port-forward $(kubectl get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}') -n istio-system 8003:8080\n",
    " ```\n",
    " \n",
    " Tests\n",
    " \n",
    "   * **Large Batch Size**\n",
    "      * `predict` method with:\n",
    "         * REST\n",
    "            * ndarray\n",
    "            * tensor\n",
    "            * tftensor\n",
    "         * gRPC\n",
    "            * ndarray\n",
    "            * tensor\n",
    "            * tftensor\n",
    "      * `predict_raw` method with:\n",
    "          * REST\n",
    "            * ndarray\n",
    "            * tensor\n",
    "            * tftensor\n",
    "          * gRPC\n",
    "            * ndarray\n",
    "            * tensor\n",
    "            * tftensor\n",
    "   * **Small Batch Size**\n",
    "       * `predict` method with:\n",
    "         * REST\n",
    "            * ndarray\n",
    "            * tensor\n",
    "            * tftensor\n",
    "         * gRPC\n",
    "            * ndarray\n",
    "            * tensor\n",
    "            * tftensor  \n",
    "            \n",
    "            \n",
    "## TLDR\n",
    "\n",
    "  * gRPC is faster than REST\n",
    "  * tftensor is best for large batch size\n",
    "  * ndarray with gRPC is bad for large batch size\n",
    "  * simpler tensor/ndarray is better for small batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.magic import register_line_cell_magic\n",
    "\n",
    "\n",
    "@register_line_cell_magic\n",
    "def writetemplate(line, cell):\n",
    "    with open(line, \"w\") as f:\n",
    "        f.write(cell.format(**globals()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.10.0-dev'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "VERSION = !cat ../../../version.txt\n",
    "VERSION = VERSION[0]\n",
    "VERSION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error from server (AlreadyExists): namespaces \"seldon\" already exists\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Release \"seldon-core\" has been upgraded. Happy Helming!\r\n",
      "NAME: seldon-core\r\n",
      "LAST DEPLOYED: Thu Jul  1 14:03:55 2021\r\n",
      "NAMESPACE: seldon-system\r\n",
      "STATUS: deployed\r\n",
      "REVISION: 2\r\n",
      "TEST SUITE: None\r\n"
     ]
    }
   ],
   "source": [
    "!helm upgrade --install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --version 1.9.0 --namespace seldon-system --set istio.enabled=\"true\" --set istio.gateway=\"seldon-gateway.istio-system.svc.cluster.local\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test with Predict method on Large Batch Size\n",
    "\n",
    "The `seldontest_predict` has simply a `predict` method that does a loop with a configurable number of iterations (default 1) to simulate work. The iterations can be set as a Seldon parameter but in this case we are looking to benchmark the serialization/deserialization cost so want a minimal amount of work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writetemplate model.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: seldon-model\n",
    "  namespace: seldon\n",
    "spec:\n",
    "  predictors:\n",
    "  - annotations:\n",
    "      seldon.io/no-engine: \"true\"\n",
    "    componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - image: seldonio/seldontest_predict:{VERSION}\n",
    "          imagePullPolicy: IfNotPresent\n",
    "          name: classifier\n",
    "          resources:\n",
    "            requests:\n",
    "              cpu: 1\n",
    "            limits:\n",
    "              cpu: 1\n",
    "          env:\n",
    "          - name: GUNICORN_WORKERS\n",
    "            value: \"1\"\n",
    "          - name: GUNICORN_THREADS\n",
    "            value: \"1\"\n",
    "        tolerations:\n",
    "        - key: model\n",
    "          operator: Exists\n",
    "          effect: NoSchedule\n",
    "    graph:\n",
    "      children: []\n",
    "      name: classifier\n",
    "      type: MODEL\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-model created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f model.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/seldon-model-default-0-classifier-5445bd4ccf-c2vdr condition met\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl wait --for condition=ready --timeout=600s pods --all -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create payloads and associated vegeta configurations for\n",
    "\n",
    "  1. ndarray\n",
    "  1. tensor\n",
    "  1. tftensor\n",
    "  \n",
    "  We will create an array of 100,000 consecutive integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "sz = 100000\n",
    "vals = list(range(sz))\n",
    "valStr = f\"{vals}\"\n",
    "payload = '{\"data\": {\"ndarray\": [' + valStr + \"]}}\"\n",
    "with open(\"data_ndarray.json\", \"w\") as f:\n",
    "    f.write(payload)\n",
    "payload_tensor = (\n",
    "    '{\"data\":{\"tensor\":{\"shape\":[1,' + str(sz) + '],\"values\":' + valStr + \"}}}\"\n",
    ")\n",
    "with open(\"data_tensor.json\", \"w\") as f:\n",
    "    f.write(payload_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from google.protobuf import json_format\n",
    "\n",
    "array = np.array(vals)\n",
    "tftensor = tf.make_tensor_proto(array)\n",
    "jStrTensor = json_format.MessageToJson(tftensor)\n",
    "jTensor = json.loads(jStrTensor)\n",
    "payload_tftensor = (\n",
    "    '{\"data\":{\"tftensor\":' + json.dumps(jTensor, separators=(\",\", \":\")) + \"}}\"\n",
    ")\n",
    "with open(\"data_tftensor.json\", \"w\") as f:\n",
    "    f.write(payload_tftensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import json\n",
    "\n",
    "sample_string_bytes = payload_tensor.encode(\"ascii\")\n",
    "base64_bytes = base64.b64encode(sample_string_bytes)\n",
    "base64_string = base64_bytes.decode(\"ascii\")\n",
    "jqPayload = {\n",
    "    \"method\": \"POST\",\n",
    "    \"url\": \"http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions\",\n",
    "    \"body\": base64_string,\n",
    "    \"header\": {\"Content-Type\": [\"application/json\"]},\n",
    "}\n",
    "with open(\"vegeta_tensor.json\", \"w\") as f:\n",
    "    f.write(json.dumps(jqPayload, separators=(\",\", \":\")))\n",
    "    f.write(\"\\n\")\n",
    "\n",
    "sample_string_bytes = payload.encode(\"ascii\")\n",
    "base64_bytes = base64.b64encode(sample_string_bytes)\n",
    "base64_string = base64_bytes.decode(\"ascii\")\n",
    "jqPayload = {\n",
    "    \"method\": \"POST\",\n",
    "    \"url\": \"http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions\",\n",
    "    \"body\": base64_string,\n",
    "    \"header\": {\"Content-Type\": [\"application/json\"]},\n",
    "}\n",
    "with open(\"vegeta_ndarray.json\", \"w\") as f:\n",
    "    f.write(json.dumps(jqPayload, separators=(\",\", \":\")))\n",
    "    f.write(\"\\n\")\n",
    "\n",
    "\n",
    "sample_string_bytes = payload_tftensor.encode(\"ascii\")\n",
    "base64_bytes = base64.b64encode(sample_string_bytes)\n",
    "base64_string = base64_bytes.decode(\"ascii\")\n",
    "jqPayload = {\n",
    "    \"method\": \"POST\",\n",
    "    \"url\": \"http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions\",\n",
    "    \"body\": base64_string,\n",
    "    \"header\": {\"Content-Type\": [\"application/json\"]},\n",
    "}\n",
    "with open(\"vegeta_tftensor.json\", \"w\") as f:\n",
    "    f.write(json.dumps(jqPayload, separators=(\",\", \":\")))\n",
    "    f.write(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Smoke test port-forward to check everything is working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"data\":{\"names\":[],\"ndarray\":[1]},\"meta\":{\"requestPath\":{\"classifier\":\"seldonio/seldontest_predict:1.10.0-dev\"}}}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "   -d '@./data_ndarray.json' \\\n",
    "    http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"data\":{\"names\":[],\"tensor\":{\"shape\":[1],\"values\":[1]}},\"meta\":{\"requestPath\":{\"classifier\":\"seldonio/seldontest_predict:1.10.0-dev\"}}}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "   -d '@./data_tensor.json' \\\n",
    "    http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"data\":{\"names\":[],\"tftensor\":{\"dtype\":\"DT_INT64\",\"int64Val\":[\"1\"],\"tensorShape\":{\"dim\":[{\"size\":\"1\"}]}}},\"meta\":{\"requestPath\":{\"classifier\":\"seldonio/seldontest_predict:1.10.0-dev\"}}}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "   -d '@./data_tftensor.json' \\\n",
    "    http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test REST\n",
    "\n",
    " 1. ndarray\n",
    " 1. tensor\n",
    " 1. tftensor\n",
    " \n",
    " This can be done locally as the results should be indicative of the relative differences rather than very accurate timings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         518, 51.76, 51.66\n",
      "Duration      [total, attack, wait]             10.027s, 10.008s, 19.333ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  17.337ms, 19.355ms, 19.136ms, 20.336ms, 21.214ms, 24.886ms, 27.831ms\n",
      "Bytes In      [total, mean]                     59570, 115.00\n",
      "Bytes Out     [total, mean]                     356857970, 688915.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:518  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_ndarray.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         504, 50.35, 50.25\n",
      "Duration      [total, attack, wait]             10.03s, 10.01s, 19.353ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  17.885ms, 19.897ms, 19.616ms, 21.1ms, 22.205ms, 25.498ms, 34.99ms\n",
      "Bytes In      [total, mean]                     69048, 137.00\n",
      "Bytes Out     [total, mean]                     347225760, 688940.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:504  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_tensor.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         636, 63.55, 63.45\n",
      "Duration      [total, attack, wait]             10.023s, 10.008s, 14.782ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  13.646ms, 15.756ms, 15.461ms, 17.41ms, 18.729ms, 20.628ms, 23.465ms\n",
      "Bytes In      [total, mean]                     118932, 187.00\n",
      "Bytes Out     [total, mean]                     678466356, 1066771.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:636  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_tftensor.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example results\n",
    "\n",
    "| ndarray | tensor | tftensor |\n",
    "| ------- | ------ | -------- |\n",
    "| 19.8ms | 19.7ms | 16.2ms |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Test gRPC\n",
    " \n",
    "  1. ndarray\n",
    "  1. tensor\n",
    "  1. tftensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t24\n",
      "  Total:\t10.13 s\n",
      "  Slowest:\t278.81 ms\n",
      "  Fastest:\t242.25 ms\n",
      "  Average:\t244.06 ms\n",
      "  Requests/sec:\t2.37\n",
      "\n",
      "Response time histogram:\n",
      "  242.253 [1]\t|∎∎∎∎∎∎∎\n",
      "  245.909 [2]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  249.564 [4]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  253.219 [6]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  256.874 [4]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  260.530 [1]\t|∎∎∎∎∎∎∎\n",
      "  264.185 [1]\t|∎∎∎∎∎∎∎\n",
      "  267.840 [2]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  271.496 [0]\t|\n",
      "  275.151 [1]\t|∎∎∎∎∎∎∎\n",
      "  278.806 [1]\t|∎∎∎∎∎∎∎\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 247.44 ms \n",
      "  25 % in 249.47 ms \n",
      "  50 % in 252.85 ms \n",
      "  75 % in 260.70 ms \n",
      "  90 % in 272.55 ms \n",
      "  95 % in 278.81 ms \n",
      "  0 % in 0 ns \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]         23 responses   \n",
      "  [Canceled]   1 responses    \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Canceled desc = grpc: the client connection is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_ndarray.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t92\n",
      "  Total:\t10.10 s\n",
      "  Slowest:\t21.23 ms\n",
      "  Fastest:\t4.91 ms\n",
      "  Average:\t7.58 ms\n",
      "  Requests/sec:\t9.11\n",
      "\n",
      "Response time histogram:\n",
      "  4.906 [1]\t|∎\n",
      "  6.539 [55]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  8.171 [17]\t|∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  9.804 [4]\t|∎∎∎\n",
      "  11.436 [0]\t|\n",
      "  13.069 [4]\t|∎∎∎\n",
      "  14.701 [3]\t|∎∎\n",
      "  16.334 [3]\t|∎∎\n",
      "  17.966 [0]\t|\n",
      "  19.599 [2]\t|∎\n",
      "  21.232 [2]\t|∎\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 5.51 ms \n",
      "  25 % in 5.70 ms \n",
      "  50 % in 6.14 ms \n",
      "  75 % in 7.09 ms \n",
      "  90 % in 14.14 ms \n",
      "  95 % in 18.77 ms \n",
      "  0 % in 0 ns \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]         91 responses   \n",
      "  [Canceled]   1 responses    \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Canceled desc = grpc: the client connection is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_tensor.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t425\n",
      "  Total:\t10.04 s\n",
      "  Slowest:\t16.38 ms\n",
      "  Fastest:\t3.97 ms\n",
      "  Average:\t5.33 ms\n",
      "  Requests/sec:\t42.31\n",
      "\n",
      "Response time histogram:\n",
      "  3.970 [1]\t|\n",
      "  5.211 [281]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  6.452 [91]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  7.692 [25]\t|∎∎∎∎\n",
      "  8.933 [8]\t|∎\n",
      "  10.174 [6]\t|∎\n",
      "  11.415 [7]\t|∎\n",
      "  12.656 [2]\t|\n",
      "  13.896 [1]\t|\n",
      "  15.137 [1]\t|\n",
      "  16.378 [1]\t|\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 4.34 ms \n",
      "  25 % in 4.54 ms \n",
      "  50 % in 4.89 ms \n",
      "  75 % in 5.52 ms \n",
      "  90 % in 6.79 ms \n",
      "  95 % in 8.30 ms \n",
      "  99 % in 11.71 ms \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]         424 responses   \n",
      "  [Canceled]   1 responses     \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Canceled desc = grpc: the client connection is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_tftensor.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example results\n",
    "\n",
    "| ndarray | tensor | tftensor |\n",
    "| ------- | ------ | -------- |\n",
    "| 253ms | 8.4ms | 5.5ms |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    " * gRPC is generally faster than REST except for ndarray which is much worse and should not be used with gRPC\n",
    " * tftensor is fastest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"seldon-model\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f model.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test Predct Raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writetemplate model.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: seldon-model\n",
    "  namespace: seldon\n",
    "spec:\n",
    "  predictors:\n",
    "  - annotations:\n",
    "      seldon.io/no-engine: \"true\"\n",
    "    componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - image: seldonio/seldontest_predict_raw:{VERSION}\n",
    "          imagePullPolicy: IfNotPresent\n",
    "          name: classifier\n",
    "          resources:\n",
    "            requests:\n",
    "              cpu: 1\n",
    "            limits:\n",
    "              cpu: 1\n",
    "          env:\n",
    "          - name: GUNICORN_WORKERS\n",
    "            value: \"1\"\n",
    "          - name: GUNICORN_THREADS\n",
    "            value: \"1\"\n",
    "        tolerations:\n",
    "        - key: model\n",
    "          operator: Exists\n",
    "          effect: NoSchedule\n",
    "    graph:\n",
    "      children: []\n",
    "      name: classifier\n",
    "      type: MODEL\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-model created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f model.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/seldon-model-default-0-classifier-5dc8fbd597-kk7td condition met\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl wait --for condition=ready --timeout=600s pods --all -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Smoke test port-forward to check everything is working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\r\n"
     ]
    }
   ],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "   -d '@./data_tftensor.json' \\\n",
    "    http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test REST\n",
    "\n",
    " 1. ndarray\n",
    " 1. tensor\n",
    " 1. tftensor\n",
    " \n",
    " This can be done locally as the results should be indicative of the relative differences rather than very accurate timings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         724, 72.35, 72.25\n",
      "Duration      [total, attack, wait]             10.021s, 10.007s, 14.458ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  12.228ms, 13.838ms, 13.683ms, 14.641ms, 15.489ms, 17.888ms, 22.263ms\n",
      "Bytes In      [total, mean]                     2896, 4.00\n",
      "Bytes Out     [total, mean]                     498774460, 688915.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:724  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_ndarray.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         724, 72.32, 72.22\n",
      "Duration      [total, attack, wait]             10.025s, 10.011s, 14.307ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  12.362ms, 13.844ms, 13.701ms, 14.655ms, 15.493ms, 17.976ms, 18.802ms\n",
      "Bytes In      [total, mean]                     2896, 4.00\n",
      "Bytes Out     [total, mean]                     498792560, 688940.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:724  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_tensor.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         901, 90.04, 89.93\n",
      "Duration      [total, attack, wait]             10.018s, 10.007s, 11.64ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  8.955ms, 11.116ms, 10.994ms, 12.099ms, 12.721ms, 15.208ms, 19.918ms\n",
      "Bytes In      [total, mean]                     3604, 4.00\n",
      "Bytes Out     [total, mean]                     961160671, 1066771.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:901  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_tftensor.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example results\n",
    "\n",
    "| ndarray | tensor | tftensor |\n",
    "| ------- | ------ | -------- |\n",
    "| 13.3ms | 13.3ms | 11.1ms |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Test gRPC\n",
    " \n",
    "  1. ndarray\n",
    "  1. tensor\n",
    "  1. tftensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t44\n",
      "  Total:\t10.04 s\n",
      "  Slowest:\t69.07 ms\n",
      "  Fastest:\t44.44 ms\n",
      "  Average:\t46.03 ms\n",
      "  Requests/sec:\t4.38\n",
      "\n",
      "Response time histogram:\n",
      "  44.440 [1]\t|∎\n",
      "  46.904 [31]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  49.367 [6]\t|∎∎∎∎∎∎∎∎\n",
      "  51.831 [2]\t|∎∎∎\n",
      "  54.294 [2]\t|∎∎∎\n",
      "  56.758 [0]\t|\n",
      "  59.221 [0]\t|\n",
      "  61.684 [0]\t|\n",
      "  64.148 [0]\t|\n",
      "  66.611 [0]\t|\n",
      "  69.075 [1]\t|∎\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 45.05 ms \n",
      "  25 % in 45.40 ms \n",
      "  50 % in 46.30 ms \n",
      "  75 % in 47.34 ms \n",
      "  90 % in 50.16 ms \n",
      "  95 % in 53.38 ms \n",
      "  0 % in 0 ns \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]         43 responses   \n",
      "  [Canceled]   1 responses    \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Canceled desc = grpc: the client connection is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_ndarray.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t92\n",
      "  Total:\t10.10 s\n",
      "  Slowest:\t19.81 ms\n",
      "  Fastest:\t4.93 ms\n",
      "  Average:\t7.91 ms\n",
      "  Requests/sec:\t9.11\n",
      "\n",
      "Response time histogram:\n",
      "  4.932 [1]\t|∎\n",
      "  6.419 [53]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  7.907 [12]\t|∎∎∎∎∎∎∎∎∎\n",
      "  9.395 [5]\t|∎∎∎∎\n",
      "  10.882 [4]\t|∎∎∎\n",
      "  12.370 [1]\t|∎\n",
      "  13.858 [3]\t|∎∎\n",
      "  15.346 [3]\t|∎∎\n",
      "  16.833 [2]\t|∎∎\n",
      "  18.321 [3]\t|∎∎\n",
      "  19.809 [4]\t|∎∎∎\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 5.21 ms \n",
      "  25 % in 5.68 ms \n",
      "  50 % in 6.04 ms \n",
      "  75 % in 8.27 ms \n",
      "  90 % in 15.77 ms \n",
      "  95 % in 19.04 ms \n",
      "  0 % in 0 ns \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]         91 responses   \n",
      "  [Canceled]   1 responses    \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Canceled desc = grpc: the client connection is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_tensor.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t426\n",
      "  Total:\t10.03 s\n",
      "  Slowest:\t11.74 ms\n",
      "  Fastest:\t3.67 ms\n",
      "  Average:\t5.02 ms\n",
      "  Requests/sec:\t42.48\n",
      "\n",
      "Response time histogram:\n",
      "  3.668 [1]\t|\n",
      "  4.475 [174]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  5.282 [141]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  6.089 [43]\t|∎∎∎∎∎∎∎∎∎∎\n",
      "  6.897 [30]\t|∎∎∎∎∎∎∎\n",
      "  7.704 [16]\t|∎∎∎∎\n",
      "  8.511 [6]\t|∎\n",
      "  9.318 [8]\t|∎∎\n",
      "  10.126 [2]\t|\n",
      "  10.933 [1]\t|\n",
      "  11.740 [3]\t|∎\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 4.08 ms \n",
      "  25 % in 4.27 ms \n",
      "  50 % in 4.61 ms \n",
      "  75 % in 5.30 ms \n",
      "  90 % in 6.62 ms \n",
      "  95 % in 7.66 ms \n",
      "  99 % in 10.26 ms \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]         425 responses   \n",
      "  [Canceled]   1 responses     \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Canceled desc = grpc: the client connection is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_tftensor.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example results\n",
    "\n",
    "| ndarray | tensor | tftensor |\n",
    "| ------- | ------ | -------- |\n",
    "| 46ms | 7.9ms | 5.0ms |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    " * `predict_raw` is faster than `predict` but you will need to handle the serialization/deserializtion yourself which maybe will make them equivalent unless specific techniques can be applied for your use case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test with Predict method on Small Batch Size\n",
    "\n",
    "The `seldontest_predict` has simply a `predict` method that does a loop with a configurable number of iterations (default 1) to simulate work. The iterations can be set as a Seldon parameter but in this case we are looking to benchmark the serialization/deserialization cost so want a minimal amount of work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writetemplate model.yaml\n",
    "apiVersion: machinelearning.seldon.io/v1\n",
    "kind: SeldonDeployment\n",
    "metadata:\n",
    "  name: seldon-model\n",
    "  namespace: seldon\n",
    "spec:\n",
    "  predictors:\n",
    "  - annotations:\n",
    "      seldon.io/no-engine: \"true\"\n",
    "    componentSpecs:\n",
    "    - spec:\n",
    "        containers:\n",
    "        - image: seldonio/seldontest_predict:{VERSION}\n",
    "          imagePullPolicy: IfNotPresent\n",
    "          name: classifier\n",
    "          resources:\n",
    "            requests:\n",
    "              cpu: 1\n",
    "            limits:\n",
    "              cpu: 1\n",
    "          env:\n",
    "          - name: GUNICORN_WORKERS\n",
    "            value: \"1\"\n",
    "          - name: GUNICORN_THREADS\n",
    "            value: \"1\"\n",
    "        tolerations:\n",
    "        - key: model\n",
    "          operator: Exists\n",
    "          effect: NoSchedule\n",
    "    graph:\n",
    "      children: []\n",
    "      name: classifier\n",
    "      type: MODEL\n",
    "    name: default\n",
    "    replicas: 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-model configured\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f model.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pod/seldon-model-default-0-classifier-5445bd4ccf-bgkcm condition met\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl wait --for condition=ready --timeout=600s pods --all -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create payloads and associated vegeta configurations for\n",
    "\n",
    "  1. ndarray\n",
    "  1. tensor\n",
    "  1. tftensor\n",
    "  \n",
    "  We will create an array of 100,000 consecutive integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "sz = 1\n",
    "vals = list(range(sz))\n",
    "valStr = f\"{vals}\"\n",
    "payload = '{\"data\": {\"ndarray\": [' + valStr + \"]}}\"\n",
    "with open(\"data_ndarray.json\", \"w\") as f:\n",
    "    f.write(payload)\n",
    "payload_tensor = (\n",
    "    '{\"data\":{\"tensor\":{\"shape\":[1,' + str(sz) + '],\"values\":' + valStr + \"}}}\"\n",
    ")\n",
    "with open(\"data_tensor.json\", \"w\") as f:\n",
    "    f.write(payload_tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from google.protobuf import json_format\n",
    "\n",
    "array = np.array(vals)\n",
    "tftensor = tf.make_tensor_proto(array)\n",
    "jStrTensor = json_format.MessageToJson(tftensor)\n",
    "jTensor = json.loads(jStrTensor)\n",
    "payload_tftensor = (\n",
    "    '{\"data\":{\"tftensor\":' + json.dumps(jTensor, separators=(\",\", \":\")) + \"}}\"\n",
    ")\n",
    "with open(\"data_tftensor.json\", \"w\") as f:\n",
    "    f.write(payload_tftensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import json\n",
    "\n",
    "sample_string_bytes = payload_tensor.encode(\"ascii\")\n",
    "base64_bytes = base64.b64encode(sample_string_bytes)\n",
    "base64_string = base64_bytes.decode(\"ascii\")\n",
    "jqPayload = {\n",
    "    \"method\": \"POST\",\n",
    "    \"url\": \"http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions\",\n",
    "    \"body\": base64_string,\n",
    "    \"header\": {\"Content-Type\": [\"application/json\"]},\n",
    "}\n",
    "with open(\"vegeta_tensor.json\", \"w\") as f:\n",
    "    f.write(json.dumps(jqPayload, separators=(\",\", \":\")))\n",
    "    f.write(\"\\n\")\n",
    "\n",
    "sample_string_bytes = payload.encode(\"ascii\")\n",
    "base64_bytes = base64.b64encode(sample_string_bytes)\n",
    "base64_string = base64_bytes.decode(\"ascii\")\n",
    "jqPayload = {\n",
    "    \"method\": \"POST\",\n",
    "    \"url\": \"http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions\",\n",
    "    \"body\": base64_string,\n",
    "    \"header\": {\"Content-Type\": [\"application/json\"]},\n",
    "}\n",
    "with open(\"vegeta_ndarray.json\", \"w\") as f:\n",
    "    f.write(json.dumps(jqPayload, separators=(\",\", \":\")))\n",
    "    f.write(\"\\n\")\n",
    "\n",
    "\n",
    "sample_string_bytes = payload_tftensor.encode(\"ascii\")\n",
    "base64_bytes = base64.b64encode(sample_string_bytes)\n",
    "base64_string = base64_bytes.decode(\"ascii\")\n",
    "jqPayload = {\n",
    "    \"method\": \"POST\",\n",
    "    \"url\": \"http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions\",\n",
    "    \"body\": base64_string,\n",
    "    \"header\": {\"Content-Type\": [\"application/json\"]},\n",
    "}\n",
    "with open(\"vegeta_tftensor.json\", \"w\") as f:\n",
    "    f.write(json.dumps(jqPayload, separators=(\",\", \":\")))\n",
    "    f.write(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Smoke test port-forward to check everything is working"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"data\":{\"names\":[],\"tensor\":{\"shape\":[1],\"values\":[1]}},\"meta\":{\"requestPath\":{\"classifier\":\"seldonio/seldontest_predict:1.10.0-dev\"}}}\r\n"
     ]
    }
   ],
   "source": [
    "!curl -X POST -H 'Content-Type: application/json' \\\n",
    "   -d '@./data_tensor.json' \\\n",
    "    http://localhost:8003/seldon/seldon/seldon-model/api/v1.0/predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test REST\n",
    "\n",
    " 1. ndarray\n",
    " 1. tensor\n",
    " 1. tftensor\n",
    " \n",
    " This can be done locally as the results should be indicative of the relative differences rather than very accurate timings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         5538, 553.80, 553.67\n",
      "Duration      [total, attack, wait]             10.002s, 10s, 2.364ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  1.569ms, 1.804ms, 1.739ms, 1.984ms, 2.198ms, 2.861ms, 6.62ms\n",
      "Bytes In      [total, mean]                     636870, 115.00\n",
      "Bytes Out     [total, mean]                     155064, 28.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:5538  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_ndarray.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         5557, 555.65, 555.55\n",
      "Duration      [total, attack, wait]             10.003s, 10.001s, 1.753ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  1.578ms, 1.798ms, 1.74ms, 1.925ms, 2.119ms, 2.981ms, 5.968ms\n",
      "Bytes In      [total, mean]                     761309, 137.00\n",
      "Bytes Out     [total, mean]                     266736, 48.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:5557  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_tensor.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requests      [total, rate, throughput]         4548, 454.75, 454.65\n",
      "Duration      [total, attack, wait]             10.003s, 10.001s, 2.141ms\n",
      "Latencies     [min, mean, 50, 90, 95, 99, max]  1.937ms, 2.197ms, 2.138ms, 2.351ms, 2.482ms, 3.215ms, 9.424ms\n",
      "Bytes In      [total, mean]                     850476, 187.00\n",
      "Bytes Out     [total, mean]                     436608, 96.00\n",
      "Success       [ratio]                           100.00%\n",
      "Status Codes  [code:count]                      200:4548  \n",
      "Error Set:\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "vegeta attack -format=json -duration=10s -rate=0 -max-workers=1 -targets=vegeta_tftensor.json | \n",
    "  vegeta report -type=text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example results\n",
    "\n",
    "| ndarray | tensor | tftensor |\n",
    "| ------- | ------ | -------- |\n",
    "| 1.8ms | 1.8ms | 2.1ms |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Test gRPC\n",
    " \n",
    "  1. ndarray\n",
    "  1. tensor\n",
    "  1. tftensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t6506\n",
      "  Total:\t10.01 s\n",
      "  Slowest:\t18.58 ms\n",
      "  Fastest:\t1.26 ms\n",
      "  Average:\t1.46 ms\n",
      "  Requests/sec:\t650.23\n",
      "\n",
      "Response time histogram:\n",
      "  1.260 [1]\t|\n",
      "  2.992 [6465]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  4.724 [30]\t|\n",
      "  6.456 [5]\t|\n",
      "  8.187 [2]\t|\n",
      "  9.919 [1]\t|\n",
      "  11.651 [0]\t|\n",
      "  13.382 [0]\t|\n",
      "  15.114 [0]\t|\n",
      "  16.846 [0]\t|\n",
      "  18.578 [1]\t|\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 1.33 ms \n",
      "  25 % in 1.36 ms \n",
      "  50 % in 1.39 ms \n",
      "  75 % in 1.45 ms \n",
      "  90 % in 1.58 ms \n",
      "  95 % in 1.79 ms \n",
      "  99 % in 2.50 ms \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]            6505 responses   \n",
      "  [Unavailable]   1 responses      \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Unavailable desc = transport is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_ndarray.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t6429\n",
      "  Total:\t10.01 s\n",
      "  Slowest:\t16.30 ms\n",
      "  Fastest:\t1.29 ms\n",
      "  Average:\t1.49 ms\n",
      "  Requests/sec:\t642.56\n",
      "\n",
      "Response time histogram:\n",
      "  1.287 [1]\t|\n",
      "  2.789 [6375]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  4.290 [36]\t|\n",
      "  5.792 [11]\t|\n",
      "  7.293 [2]\t|\n",
      "  8.795 [1]\t|\n",
      "  10.296 [0]\t|\n",
      "  11.798 [1]\t|\n",
      "  13.299 [0]\t|\n",
      "  14.801 [0]\t|\n",
      "  16.303 [1]\t|\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 1.36 ms \n",
      "  25 % in 1.38 ms \n",
      "  50 % in 1.42 ms \n",
      "  75 % in 1.48 ms \n",
      "  90 % in 1.60 ms \n",
      "  95 % in 1.80 ms \n",
      "  99 % in 2.67 ms \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]            6428 responses   \n",
      "  [Unavailable]   1 responses      \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Unavailable desc = transport is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_tensor.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Summary:\n",
      "  Count:\t6066\n",
      "  Total:\t10.01 s\n",
      "  Slowest:\t9.38 ms\n",
      "  Fastest:\t1.39 ms\n",
      "  Average:\t1.57 ms\n",
      "  Requests/sec:\t606.20\n",
      "\n",
      "Response time histogram:\n",
      "  1.387 [1]\t|\n",
      "  2.187 [5945]\t|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎\n",
      "  2.986 [84]\t|∎\n",
      "  3.785 [20]\t|\n",
      "  4.585 [7]\t|\n",
      "  5.384 [2]\t|\n",
      "  6.183 [4]\t|\n",
      "  6.983 [0]\t|\n",
      "  7.782 [0]\t|\n",
      "  8.582 [1]\t|\n",
      "  9.381 [1]\t|\n",
      "\n",
      "Latency distribution:\n",
      "  10 % in 1.46 ms \n",
      "  25 % in 1.48 ms \n",
      "  50 % in 1.52 ms \n",
      "  75 % in 1.57 ms \n",
      "  90 % in 1.66 ms \n",
      "  95 % in 1.81 ms \n",
      "  99 % in 2.61 ms \n",
      "\n",
      "Status code distribution:\n",
      "  [OK]            6065 responses   \n",
      "  [Unavailable]   1 responses      \n",
      "\n",
      "Error distribution:\n",
      "  [1]   rpc error: code = Unavailable desc = transport is closing   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "ghz \\\n",
    "    --insecure \\\n",
    "    --proto ../../../proto/prediction.proto \\\n",
    "    --call seldon.protos.Seldon/Predict \\\n",
    "    --data-file=./data_tftensor.json \\\n",
    "    --qps=0 \\\n",
    "    --cpus=1 \\\n",
    "    --concurrency=1 \\\n",
    "    --duration=\"10s\" \\\n",
    "    --format summary \\\n",
    "    --metadata='{\"seldon\": \"seldon-model\", \"namespace\": \"seldon\"}' \\\n",
    "    localhost:8003"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example results\n",
    "\n",
    "| ndarray | tensor | tftensor |\n",
    "| ------- | ------ | -------- |\n",
    "| 1.46ms | 1.49ms | 1.57ms |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    " * gRPC is generally faster than REST\n",
    " * There is very little difference between payload types with simpler tensor/ndarray probably being slightly faster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"seldon-model\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f model.yaml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}