examples/batch/benchmarking-argo-workflows/README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarking with Argo Worfklows & Vegeta\n",
    "\n",
    "In this notebook we will dive into how you can run bench marking with batch processing with Argo Workflows, Seldon Core and Vegeta.\n",
    "\n",
    "Dependencies:\n",
    "\n",
    "* Seldon core installed as per the docs with Istio as an ingress \n",
    "* Argo Workfklows installed in cluster (and argo CLI for commands)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Install Seldon Core\n",
    "Use the notebook to [set-up Seldon Core with Ambassador or Istio Ingress](https://docs.seldon.io/projects/seldon-core/en/latest/examples/seldon_core_setup.html).\n",
    "\n",
    "Note: If running with KIND you need to make sure do follow [these steps](https://github.com/argoproj/argo/issues/2376#issuecomment-595593237) as workaround to the `/.../docker.sock` known issue.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Argo Workflows\n",
    "You can follow the instructions from the official [Argo Workflows Documentation](https://github.com/argoproj/argo#quickstart).\n",
    "\n",
    "You also need to make sure that argo has permissions to create seldon deployments - for this you can just create a default-admin rolebinding as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rolebinding.rbac.authorization.k8s.io/default-admin created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_results(results, print_results=True):\n",
    "    final = {}\n",
    "    if \"average\" in results:\n",
    "        final[\"mean\"] = results[\"average\"] / 1e6\n",
    "        if results.get(\"latencyDistribution\", False):\n",
    "            final[\"50th\"] = results[\"latencyDistribution\"][-5][\"latency\"] / 1e6\n",
    "            final[\"90th\"] = results[\"latencyDistribution\"][-3][\"latency\"] / 1e6\n",
    "            final[\"95th\"] = results[\"latencyDistribution\"][-2][\"latency\"] / 1e6\n",
    "            final[\"99th\"] = results[\"latencyDistribution\"][-1][\"latency\"] / 1e6\n",
    "        final[\"rate\"] = results[\"rps\"]\n",
    "        final[\"errors\"] = results[\"statusCodeDistribution\"]\n",
    "    else:\n",
    "        final[\"mean\"] = results[\"latencies\"][\"mean\"] / 1e6\n",
    "        final[\"50th\"] = results[\"latencies\"][\"50th\"] / 1e6\n",
    "        final[\"90th\"] = results[\"latencies\"][\"90th\"] / 1e6\n",
    "        final[\"95th\"] = results[\"latencies\"][\"95th\"] / 1e6\n",
    "        final[\"99th\"] = results[\"latencies\"][\"99th\"] / 1e6\n",
    "        final[\"rate\"] = results[\"throughput\"]\n",
    "        final[\"errors\"] = results[\"errors\"]\n",
    "    if print_results:    \n",
    "        print(\"Latencies:\")\n",
    "        print(\"\\tmean:\", final[\"mean\"], \"ms\")\n",
    "        print(\"\\t50th:\", final[\"50th\"], \"ms\")\n",
    "        print(\"\\t90th:\", final[\"90th\"], \"ms\")\n",
    "        print(\"\\t95th:\", final[\"95th\"], \"ms\")\n",
    "        print(\"\\t99th:\", final[\"99th\"], \"ms\")\n",
    "        print(\"\")\n",
    "        print(\"Rate:\", str(final[\"rate\"]) + \"/s\")\n",
    "        print(\"Errors:\", final[\"errors\"])\n",
    "    return final"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Benchmark Argo Workflow\n",
    "\n",
    "In order to create a benchmark, we created a simple argo workflow template so you can leverage the power of the helm charts.\n",
    "\n",
    "Before we dive into the contents of the full helm chart, let's first give it a try with some of the settings.\n",
    "\n",
    "We will run a batch job that will set up a Seldon Deployment with 1 replicas and 4 cpus (with 100 max workers) to send requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:                seldon-benchmark-process\r\n",
      "Namespace:           default\r\n",
      "ServiceAccount:      default\r\n",
      "Status:              Pending\r\n",
      "Created:             Fri Aug 07 18:09:40 +0100 (now)\r\n"
     ]
    }
   ],
   "source": [
    "!helm template seldon-benchmark-workflow helm-charts/seldon-benchmark-workflow/ \\\n",
    "    --set workflow.name=seldon-benchmark-process \\\n",
    "    --set seldonDeployment.name=sklearn \\\n",
    "    --set seldonDeployment.replicas=1 \\\n",
    "    --set seldonDeployment.serverWorkers=1 \\\n",
    "    --set seldonDeployment.serverThreads=10 \\\n",
    "    --set seldonDeployment.modelUri=\"gs://seldon-models/sklearn/iris\" \\\n",
    "    --set seldonDeployment.server=\"SKLEARN_SERVER\" \\\n",
    "    --set seldonDeployment.apiType=rest \\\n",
    "    --set benchmark.cpus=4 \\\n",
    "    --set benchmark.maxWorkers=100 \\\n",
    "    --set benchmark.duration=30s \\\n",
    "    --set benchmark.rate=0 \\\n",
    "    --set benchmark.data='\\{\"data\": {\"ndarray\": [[0\\,1\\,2\\,3]]\\}\\}' \\\n",
    "    | argo submit -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                       STATUS      AGE   DURATION   PRIORITY\r\n",
      "seldon-benchmark-process   Succeeded   2m    1m         0\r\n"
     ]
    }
   ],
   "source": [
    "!argo list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:                seldon-benchmark-process\r\n",
      "Namespace:           default\r\n",
      "ServiceAccount:      default\r\n",
      "Status:              Succeeded\r\n",
      "Created:             Fri Aug 07 18:09:40 +0100 (2 minutes ago)\r\n",
      "Started:             Fri Aug 07 18:09:40 +0100 (2 minutes ago)\r\n",
      "Finished:            Fri Aug 07 18:11:09 +0100 (51 seconds ago)\r\n",
      "Duration:            1 minute 29 seconds\r\n",
      "\r\n",
      "\u001b[39mSTEP\u001b[0m                                                             PODNAME                              DURATION  MESSAGE\r\n",
      " \u001b[32m✔\u001b[0m seldon-benchmark-process (seldon-benchmark-process)                                                          \r\n",
      " ├---\u001b[32m✔\u001b[0m create-seldon-resource (create-seldon-resource-template)  seldon-benchmark-process-3980407503  2s        \r\n",
      " ├---\u001b[32m✔\u001b[0m wait-seldon-resource (wait-seldon-resource-template)      seldon-benchmark-process-2136965893  49s       \r\n",
      " └---\u001b[32m✔\u001b[0m run-benchmark (run-benchmark-template)                    seldon-benchmark-process-780051119   32s       \r\n"
     ]
    }
   ],
   "source": [
    "!argo get seldon-benchmark-process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:41.804Z\" level=info msg=\"Starting Workflow Executor\" version=v2.9.3\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:41.809Z\" level=info msg=\"Creating a docker executor\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:41.809Z\" level=info msg=\"Executor (version: v2.9.3, build_date: 2020-07-18T19:11:19Z) initialized (pod: default/seldon-benchmark-process-3980407503) with template:\\n{\\\"name\\\":\\\"create-seldon-resource-template\\\",\\\"arguments\\\":{},\\\"inputs\\\":{},\\\"outputs\\\":{},\\\"metadata\\\":{},\\\"resource\\\":{\\\"action\\\":\\\"create\\\",\\\"manifest\\\":\\\"apiVersion: machinelearning.seldon.io/v1\\\\nkind: SeldonDeployment\\\\nmetadata:\\\\n  name: \\\\\\\"sklearn\\\\\\\"\\\\n  namespace: default\\\\n  ownerReferences:\\\\n  - apiVersion: argoproj.io/v1alpha1\\\\n    blockOwnerDeletion: true\\\\n    kind: Workflow\\\\n    name: \\\\\\\"seldon-benchmark-process\\\\\\\"\\\\n    uid: \\\\\\\"e0364966-b2c1-4ee7-a7cf-421952ba3c7a\\\\\\\"\\\\nspec:\\\\n  annotations:\\\\n    seldon.io/executor: \\\\\\\"false\\\\\\\"\\\\n  name: \\\\\\\"sklearn\\\\\\\"\\\\n  transport: rest\\\\n  predictors:\\\\n    - componentSpecs:\\\\n      - spec:\\\\n        containers:\\\\n        - name: classifier\\\\n          env:\\\\n          - name: GUNICORN_THREADS\\\\n            value: 10\\\\n          - name: GUNICORN_WORKERS\\\\n            value: 1\\\\n          resources:\\\\n            requests:\\\\n              cpu: 50m\\\\n              memory: 100Mi\\\\n            limits:\\\\n              cpu: 50m\\\\n              memory: 1000Mi\\\\n      graph:\\\\n        children: []\\\\n        implementation: SKLEARN_SERVER\\\\n        modelUri: gs://seldon-models/sklearn/iris\\\\n        name: classifier\\\\n      name: default\\\\n      replicas: 1\\\\n\\\"}}\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:41.809Z\" level=info msg=\"Loading manifest to /tmp/manifest.yaml\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:41.810Z\" level=info msg=\"kubectl create -f /tmp/manifest.yaml -o json\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:42.456Z\" level=info msg=default/SeldonDeployment.machinelearning.seldon.io/sklearn\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:09:42.457Z\" level=info msg=\"No output parameters\"\r\n",
      "\u001b[32mwait-seldon-resource\u001b[0m:\tWaiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\r\n",
      "\u001b[32mwait-seldon-resource\u001b[0m:\tdeployment \"sklearn-default-0-classifier\" successfully rolled out\r\n",
      "\u001b[35mrun-benchmark\u001b[0m:\t{\"latencies\":{\"total\":3011298973622,\"mean\":339033885,\"50th\":272840630,\"90th\":339539236,\"95th\":368299307,\"99th\":4982426813,\"max\":5597505277,\"min\":206244298},\"bytes_in\":{\"total\":3081764,\"mean\":346.9673496960144},\"bytes_out\":{\"total\":301988,\"mean\":34},\"earliest\":\"2020-08-07T17:10:37.117884325Z\",\"latest\":\"2020-08-07T17:11:07.118729145Z\",\"end\":\"2020-08-07T17:11:07.366654843Z\",\"duration\":30000844820,\"wait\":247925698,\"requests\":8882,\"rate\":296.05832946673667,\"throughput\":293.63176909007353,\"success\":1,\"status_codes\":{\"200\":8882},\"errors\":[]}\r\n"
     ]
    }
   ],
   "source": [
    "!argo logs -w seldon-benchmark-process "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Latencies:\n",
      "\tmean: 339.033885 ms\n",
      "\t50th: 272.84063 ms\n",
      "\t90th: 339.539236 ms\n",
      "\t95th: 368.299307 ms\n",
      "\t99th: 4982.426813 ms\n",
      "\n",
      "Throughput: 293.63176909007353/s\n",
      "Errors: False\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "wf_logs = !argo logs -w seldon-benchmark-process \n",
    "wf_bench = wf_logs[-1]\n",
    "wf_json_str = wf_bench[24:]\n",
    "results = json.loads(wf_json_str)\n",
    "\n",
    "print(\"Latencies:\")\n",
    "print(\"\\tmean:\", results[\"latencies\"][\"mean\"] / 1e6, \"ms\")\n",
    "print(\"\\t50th:\", results[\"latencies\"][\"50th\"] / 1e6, \"ms\")\n",
    "print(\"\\t90th:\", results[\"latencies\"][\"90th\"] / 1e6, \"ms\")\n",
    "print(\"\\t95th:\", results[\"latencies\"][\"95th\"] / 1e6, \"ms\")\n",
    "print(\"\\t99th:\", results[\"latencies\"][\"99th\"] / 1e6, \"ms\")\n",
    "print(\"\")\n",
    "print(\"Throughput:\", str(results[\"throughput\"]) + \"/s\")\n",
    "print(\"Errors:\", len(results[\"errors\"]) > 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workflow 'seldon-benchmark-process' deleted\r\n"
     ]
    }
   ],
   "source": [
    "!argo delete seldon-benchmark-process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create GRPC benchmark with GHZ and Argo Workflows "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:                seldon-benchmark-process\r\n",
      "Namespace:           default\r\n",
      "ServiceAccount:      default\r\n",
      "Status:              Pending\r\n",
      "Created:             Fri Aug 07 18:22:38 +0100 (now)\r\n"
     ]
    }
   ],
   "source": [
    "!helm template seldon-benchmark-workflow helm-charts/seldon-benchmark-workflow/ \\\n",
    "    --set workflow.name=seldon-benchmark-process \\\n",
    "    --set seldonDeployment.name=sklearn \\\n",
    "    --set seldonDeployment.replicas=1 \\\n",
    "    --set seldonDeployment.serverWorkers=1 \\\n",
    "    --set seldonDeployment.serverThreads=10 \\\n",
    "    --set seldonDeployment.modelUri=\"gs://seldon-models/sklearn/iris\" \\\n",
    "    --set seldonDeployment.server=\"SKLEARN_SERVER\" \\\n",
    "    --set seldonDeployment.apiType=grpc \\\n",
    "    --set benchmark.cpus=4 \\\n",
    "    --set benchmark.maxWorkers=100 \\\n",
    "    --set benchmark.duration=\"120s\" \\\n",
    "    --set benchmark.rate=0 \\\n",
    "    --set benchmark.data='\\{\"data\": {\"ndarray\": [[0\\,1\\,2\\,3]]\\}\\}' \\\n",
    "    | argo submit -"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME                       STATUS      AGE   DURATION   PRIORITY\r\n",
      "seldon-benchmark-process   Succeeded   4m    2m         0\r\n"
     ]
    }
   ],
   "source": [
    "!argo list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name:                seldon-benchmark-process\r\n",
      "Namespace:           default\r\n",
      "ServiceAccount:      default\r\n",
      "Status:              Succeeded\r\n",
      "Created:             Fri Aug 07 18:22:38 +0100 (4 minutes ago)\r\n",
      "Started:             Fri Aug 07 18:22:38 +0100 (4 minutes ago)\r\n",
      "Finished:            Fri Aug 07 18:25:11 +0100 (1 minute ago)\r\n",
      "Duration:            2 minutes 33 seconds\r\n",
      "\r\n",
      "\u001b[39mSTEP\u001b[0m                                                             PODNAME                              DURATION  MESSAGE\r\n",
      " \u001b[32m✔\u001b[0m seldon-benchmark-process (seldon-benchmark-process)                                                          \r\n",
      " ├---\u001b[32m✔\u001b[0m create-seldon-resource (create-seldon-resource-template)  seldon-benchmark-process-3980407503  2s        \r\n",
      " ├---\u001b[32m✔\u001b[0m wait-seldon-resource (wait-seldon-resource-template)      seldon-benchmark-process-2136965893  26s       \r\n",
      " └---\u001b[32m✔\u001b[0m run-benchmark (run-benchmark-template)                    seldon-benchmark-process-780051119   2m        \r\n"
     ]
    }
   ],
   "source": [
    "!argo get seldon-benchmark-process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:39.446Z\" level=info msg=\"Starting Workflow Executor\" version=v2.9.3\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:39.450Z\" level=info msg=\"Creating a docker executor\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:39.450Z\" level=info msg=\"Executor (version: v2.9.3, build_date: 2020-07-18T19:11:19Z) initialized (pod: default/seldon-benchmark-process-3980407503) with template:\\n{\\\"name\\\":\\\"create-seldon-resource-template\\\",\\\"arguments\\\":{},\\\"inputs\\\":{},\\\"outputs\\\":{},\\\"metadata\\\":{},\\\"resource\\\":{\\\"action\\\":\\\"create\\\",\\\"manifest\\\":\\\"apiVersion: machinelearning.seldon.io/v1\\\\nkind: SeldonDeployment\\\\nmetadata:\\\\n  name: \\\\\\\"sklearn\\\\\\\"\\\\n  namespace: default\\\\n  ownerReferences:\\\\n  - apiVersion: argoproj.io/v1alpha1\\\\n    blockOwnerDeletion: true\\\\n    kind: Workflow\\\\n    name: \\\\\\\"seldon-benchmark-process\\\\\\\"\\\\n    uid: \\\\\\\"e472d69d-44ed-4a45-86b3-d4b64146002b\\\\\\\"\\\\nspec:\\\\n  name: \\\\\\\"sklearn\\\\\\\"\\\\n  transport: grpc\\\\n  predictors:\\\\n    - componentSpecs:\\\\n      - spec:\\\\n        containers:\\\\n        - name: classifier\\\\n          env:\\\\n          - name: GUNICORN_THREADS\\\\n            value: 10\\\\n          - name: GUNICORN_WORKERS\\\\n            value: 1\\\\n      graph:\\\\n        children: []\\\\n        implementation: SKLEARN_SERVER\\\\n        modelUri: gs://seldon-models/sklearn/iris\\\\n        name: classifier\\\\n      name: default\\\\n      replicas: 1\\\\n\\\"}}\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:39.450Z\" level=info msg=\"Loading manifest to /tmp/manifest.yaml\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:39.450Z\" level=info msg=\"kubectl create -f /tmp/manifest.yaml -o json\"\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:40.060Z\" level=info msg=default/SeldonDeployment.machinelearning.seldon.io/sklearn\r\n",
      "\u001b[35mcreate-seldon-resource\u001b[0m:\ttime=\"2020-08-07T17:22:40.060Z\" level=info msg=\"No output parameters\"\r\n",
      "\u001b[32mwait-seldon-resource\u001b[0m:\tWaiting for deployment \"sklearn-default-0-classifier\" rollout to finish: 0 of 1 updated replicas are available...\r\n",
      "\u001b[32mwait-seldon-resource\u001b[0m:\tdeployment \"sklearn-default-0-classifier\" successfully rolled out\r\n",
      "\u001b[35mrun-benchmark\u001b[0m:\t{\"date\":\"2020-08-07T17:25:09Z\",\"endReason\":\"timeout\",\"options\":{\"host\":\"istio-ingressgateway.istio-system.svc.cluster.local:80\",\"proto\":\"/proto/prediction.proto\",\"import-paths\":[\"/proto\",\".\"],\"call\":\"seldon.protos.Seldon/Predict\",\"insecure\":true,\"total\":2147483647,\"concurrency\":50,\"connections\":1,\"duration\":120000000000,\"timeout\":20000000000,\"dial-timeout\":10000000000,\"data\":{\"data\":{\"ndarray\":[[0,1,2,3]]}},\"binary\":false,\"metadata\":{\"namespace\":\"default\",\"seldon\":\"sklearn\"},\"CPUs\":4},\"count\":88874,\"total\":120001033613,\"average\":67376309,\"fastest\":21863600,\"slowest\":148816057,\"rps\":740.6102874631579,\"errorDistribution\":{\"rpc error: code = Unavailable desc = transport is closing\":50},\"statusCodeDistribution\":{\"OK\":88824,\"Unavailable\":50},\"latencyDistribution\":[{\"percentage\":10,\"latency\":54583101},{\"percentage\":25,\"latency\":59326600},{\"percentage\":50,\"latency\":65257398},{\"percentage\":75,\"latency\":73167799},{\"percentage\":90,\"latency\":82939600},{\"percentage\":95,\"latency\":89598800},{\"percentage\":99,\"latency\":101463001}]}\r\n"
     ]
    }
   ],
   "source": [
    "!argo logs -w seldon-benchmark-process "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Latencies:\n",
      "\tmean: 67.376309 ms\n",
      "\t50th: 65.257398 ms\n",
      "\t90th: 82.9396 ms\n",
      "\t95th: 89.5988 ms\n",
      "\t99th: 101.463001 ms\n",
      "\n",
      "Rate: 740.6102874631579/s\n",
      "Errors: True\n",
      "Errors: {'OK': 88824, 'Unavailable': 50}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "wf_logs = !argo logs -w seldon-benchmark-process \n",
    "wf_bench = wf_logs[-1]\n",
    "wf_json_str = wf_bench[24:]\n",
    "results = json.loads(wf_json_str)\n",
    "\n",
    "print(\"Latencies:\")\n",
    "print(\"\\tmean:\", results[\"average\"] / 1e6, \"ms\")\n",
    "print(\"\\t50th:\", results[\"latencyDistribution\"][-5][\"latency\"] / 1e6, \"ms\")\n",
    "print(\"\\t90th:\", results[\"latencyDistribution\"][-3][\"latency\"] / 1e6, \"ms\")\n",
    "print(\"\\t95th:\", results[\"latencyDistribution\"][-2][\"latency\"] / 1e6, \"ms\")\n",
    "print(\"\\t99th:\", results[\"latencyDistribution\"][-1][\"latency\"] / 1e6, \"ms\")\n",
    "print(\"\")\n",
    "print(\"Rate:\", str(results[\"rps\"]) + \"/s\")\n",
    "print(\"Errors:\", results[\"statusCodeDistribution\"].get(\"Unavailable\", 0) > 0)\n",
    "print(\"Errors:\", results[\"statusCodeDistribution\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Workflow 'seldon-benchmark-process' deleted\r\n"
     ]
    }
   ],
   "source": [
    "!argo delete seldon-benchmark-process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run a set of tests\n",
    "\n",
    "We can now leverage the helm charts we created above to run a grid search on a set of parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools as it\n",
    "import json\n",
    "import time\n",
    "\n",
    "grid_opts = {\n",
    "    \"A-replicas\": [1, 3],\n",
    "    \"B-serverWorkers\": [1, 4],\n",
    "    \"C-serverThreads\": [50, 200],\n",
    "    \"D-apiType\": [\"rest\", \"grpc\"],\n",
    "    \"E-cpus\": [1, 4],\n",
    "    \"F-maxWorkers\": [100, 300],\n",
    "    \"G-useEngine\": [\"true\", \"false\"],\n",
    "}\n",
    "\n",
    "allNames = sorted(grid_opts)\n",
    "combinations = it.product(*(grid_opts[Name] for Name in allNames))\n",
    "all_results = []\n",
    "for curr_values in combinations:\n",
    "    print(\"VALUES:\", curr_values)\n",
    "    replicas, server_workers, server_threads, api_type, cpus, max_wokers, use_engine = curr_values\n",
    "\n",
    "    # For some reason python vars don't work with multiline helm charts\n",
    "    %env REPLICAS=$replicas\n",
    "    %env SERVER_WORKERS=$server_workers\n",
    "    %env SERVER_THREADS=$server_threads\n",
    "    %env API_TYPE=$api_type\n",
    "    %env CPUS=$cpus\n",
    "    %env MAX_WORKERS=$max_wokers\n",
    "    %env USE_ENGINE=$use_engine\n",
    "    \n",
    "    !helm template seldon-benchmark-workflow helm-charts/seldon-benchmark-workflow/ \\\n",
    "        --set workflow.name=seldon-benchmark-process \\\n",
    "        --set seldonDeployment.name=sklearn \\\n",
    "        --set seldonDeployment.replicas=$REPLICAS \\\n",
    "        --set seldonDeployment.serverWorkers=$SERVER_WORKERS \\\n",
    "        --set seldonDeployment.serverThreads=$SERVER_THREADS \\\n",
    "        --set seldonDeployment.apiType=$API_TYPE \\\n",
    "        --set seldonDeployment.useEngine=\\\"$USE_ENGINE\\\" \\\n",
    "        --set benchmark.cpus=$CPUS \\\n",
    "        --set benchmark.maxWorkers=$MAX_WORKERS \\\n",
    "        --set benchmark.duration=120s \\\n",
    "        --set benchmark.rate=0 \\\n",
    "        --set benchmark.data='\\{\"data\": {\"ndarray\": [[0\\,1\\,2\\,3]]\\}\\}' \\\n",
    "        | argo submit --wait -\n",
    "    \n",
    "    !argo wait seldon-benchmark-process \n",
    "    \n",
    "    wf_logs = !argo logs -w seldon-benchmark-process \n",
    "    wf_bench = wf_logs[-1]\n",
    "    wf_json_str = wf_bench[24:]\n",
    "    results = json.loads(wf_json_str)\n",
    "    \n",
    "    result = get_results(results)\n",
    "    result[\"replicas\"] = replicas\n",
    "    result[\"server_workers\"] = server_workers\n",
    "    result[\"server_threads\"] = server_threads\n",
    "    result[\"apiType\"] = api_type\n",
    "    result[\"cpus\"] = cpus\n",
    "    result[\"max_wokers\"] = max_wokers\n",
    "    result[\"use_engine\"] = use_engine\n",
    "    all_results.append(result)\n",
    "    \n",
    "    !argo delete seldon-benchmark-process\n",
    "    time.sleep(1)\n",
    "    print(\"\\n\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deeper Analysis\n",
    "Now that we have all the parameters, we can do a deeper analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>replicas</th>\n",
       "      <th>server_workers</th>\n",
       "      <th>server_threads</th>\n",
       "      <th>apiType</th>\n",
       "      <th>cpus</th>\n",
       "      <th>max_wokers</th>\n",
       "      <th>use_engine</th>\n",
       "      <th>mean</th>\n",
       "      <th>50th</th>\n",
       "      <th>90th</th>\n",
       "      <th>95th</th>\n",
       "      <th>99th</th>\n",
       "      <th>rate</th>\n",
       "      <th>errors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>489.269344</td>\n",
       "      <td>455.617128</td>\n",
       "      <td>612.294382</td>\n",
       "      <td>672.510108</td>\n",
       "      <td>832.322767</td>\n",
       "      <td>407.879172</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>529.767457</td>\n",
       "      <td>514.151876</td>\n",
       "      <td>591.278115</td>\n",
       "      <td>621.463805</td>\n",
       "      <td>749.348556</td>\n",
       "      <td>376.649458</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>547.618426</td>\n",
       "      <td>526.472215</td>\n",
       "      <td>661.947413</td>\n",
       "      <td>720.039676</td>\n",
       "      <td>863.596098</td>\n",
       "      <td>364.363839</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>593.880113</td>\n",
       "      <td>602.945695</td>\n",
       "      <td>737.993290</td>\n",
       "      <td>770.777543</td>\n",
       "      <td>1003.510371</td>\n",
       "      <td>336.075411</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>95.322943</td>\n",
       "      <td>97.896699</td>\n",
       "      <td>117.221999</td>\n",
       "      <td>125.852400</td>\n",
       "      <td>141.615501</td>\n",
       "      <td>523.628160</td>\n",
       "      <td>{'OK': 62790, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  replicas server_workers server_threads apiType cpus  max_wokers use_engine  \\\n",
       "0        1              1             50    rest    1         200       true   \n",
       "1        1              1             50    rest    1         200      false   \n",
       "2        1              1             50    rest    4         200       true   \n",
       "3        1              1             50    rest    4         200      false   \n",
       "4        1              1             50    grpc    1         200       true   \n",
       "\n",
       "         mean        50th        90th        95th         99th        rate  \\\n",
       "0  489.269344  455.617128  612.294382  672.510108   832.322767  407.879172   \n",
       "1  529.767457  514.151876  591.278115  621.463805   749.348556  376.649458   \n",
       "2  547.618426  526.472215  661.947413  720.039676   863.596098  364.363839   \n",
       "3  593.880113  602.945695  737.993290  770.777543  1003.510371  336.075411   \n",
       "4   95.322943   97.896699  117.221999  125.852400   141.615501  523.628160   \n",
       "\n",
       "                             errors  \n",
       "0                                []  \n",
       "1                                []  \n",
       "2                                []  \n",
       "3                                []  \n",
       "4  {'OK': 62790, 'Unavailable': 50}  "
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame.from_dict(results)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GRPC as expected outperforms REST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>replicas</th>\n",
       "      <th>server_workers</th>\n",
       "      <th>server_threads</th>\n",
       "      <th>apiType</th>\n",
       "      <th>cpus</th>\n",
       "      <th>max_wokers</th>\n",
       "      <th>use_engine</th>\n",
       "      <th>mean</th>\n",
       "      <th>50th</th>\n",
       "      <th>90th</th>\n",
       "      <th>95th</th>\n",
       "      <th>99th</th>\n",
       "      <th>rate</th>\n",
       "      <th>errors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>31.389861</td>\n",
       "      <td>23.769589</td>\n",
       "      <td>71.583795</td>\n",
       "      <td>78.881398</td>\n",
       "      <td>91.312797</td>\n",
       "      <td>1586.593680</td>\n",
       "      <td>{'OK': 190361, 'Unavailable': 48}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>31.398451</td>\n",
       "      <td>26.313000</td>\n",
       "      <td>64.841515</td>\n",
       "      <td>73.035800</td>\n",
       "      <td>88.744198</td>\n",
       "      <td>1586.555365</td>\n",
       "      <td>{'OK': 190333, 'Unavailable': 71}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>32.191240</td>\n",
       "      <td>30.448302</td>\n",
       "      <td>60.616301</td>\n",
       "      <td>68.724406</td>\n",
       "      <td>91.484308</td>\n",
       "      <td>1547.003054</td>\n",
       "      <td>{'OK': 185606, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>32.727674</td>\n",
       "      <td>28.483400</td>\n",
       "      <td>63.750796</td>\n",
       "      <td>72.597310</td>\n",
       "      <td>90.693812</td>\n",
       "      <td>1521.590875</td>\n",
       "      <td>{'OK': 182555, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>33.629848</td>\n",
       "      <td>29.610701</td>\n",
       "      <td>67.065895</td>\n",
       "      <td>77.773100</td>\n",
       "      <td>97.296599</td>\n",
       "      <td>1479.320474</td>\n",
       "      <td>{'OK': 177471, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>571.452398</td>\n",
       "      <td>556.699256</td>\n",
       "      <td>693.093315</td>\n",
       "      <td>751.197598</td>\n",
       "      <td>1024.233714</td>\n",
       "      <td>348.889260</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>587.900216</td>\n",
       "      <td>556.869872</td>\n",
       "      <td>723.744376</td>\n",
       "      <td>774.244702</td>\n",
       "      <td>939.994423</td>\n",
       "      <td>339.396160</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>593.880113</td>\n",
       "      <td>602.945695</td>\n",
       "      <td>737.993290</td>\n",
       "      <td>770.777543</td>\n",
       "      <td>1003.510371</td>\n",
       "      <td>336.075411</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>633.043624</td>\n",
       "      <td>617.853285</td>\n",
       "      <td>741.229073</td>\n",
       "      <td>776.560578</td>\n",
       "      <td>1846.623159</td>\n",
       "      <td>314.908167</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>641.530606</td>\n",
       "      <td>653.922529</td>\n",
       "      <td>802.558303</td>\n",
       "      <td>847.414484</td>\n",
       "      <td>1570.484029</td>\n",
       "      <td>310.839312</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>64 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   replicas server_workers server_threads apiType cpus  max_wokers use_engine  \\\n",
       "60        3              4            200    grpc    1         200       true   \n",
       "52        3              4             50    grpc    1         200       true   \n",
       "45        3              1            200    grpc    1         200      false   \n",
       "61        3              4            200    grpc    1         200      false   \n",
       "55        3              4             50    grpc    4         200      false   \n",
       "..      ...            ...            ...     ...  ...         ...        ...   \n",
       "10        1              1            200    rest    4         200       true   \n",
       "11        1              1            200    rest    4         200      false   \n",
       "3         1              1             50    rest    4         200      false   \n",
       "8         1              1            200    rest    1         200       true   \n",
       "9         1              1            200    rest    1         200      false   \n",
       "\n",
       "          mean        50th        90th        95th         99th         rate  \\\n",
       "60   31.389861   23.769589   71.583795   78.881398    91.312797  1586.593680   \n",
       "52   31.398451   26.313000   64.841515   73.035800    88.744198  1586.555365   \n",
       "45   32.191240   30.448302   60.616301   68.724406    91.484308  1547.003054   \n",
       "61   32.727674   28.483400   63.750796   72.597310    90.693812  1521.590875   \n",
       "55   33.629848   29.610701   67.065895   77.773100    97.296599  1479.320474   \n",
       "..         ...         ...         ...         ...          ...          ...   \n",
       "10  571.452398  556.699256  693.093315  751.197598  1024.233714   348.889260   \n",
       "11  587.900216  556.869872  723.744376  774.244702   939.994423   339.396160   \n",
       "3   593.880113  602.945695  737.993290  770.777543  1003.510371   336.075411   \n",
       "8   633.043624  617.853285  741.229073  776.560578  1846.623159   314.908167   \n",
       "9   641.530606  653.922529  802.558303  847.414484  1570.484029   310.839312   \n",
       "\n",
       "                               errors  \n",
       "60  {'OK': 190361, 'Unavailable': 48}  \n",
       "52  {'OK': 190333, 'Unavailable': 71}  \n",
       "45  {'OK': 185606, 'Unavailable': 49}  \n",
       "61  {'OK': 182555, 'Unavailable': 49}  \n",
       "55  {'OK': 177471, 'Unavailable': 50}  \n",
       "..                                ...  \n",
       "10                                 []  \n",
       "11                                 []  \n",
       "3                                  []  \n",
       "8                                  []  \n",
       "9                                  []  \n",
       "\n",
       "[64 rows x 14 columns]"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(\"rate\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deeper dive REST\n",
    "As expected replicas has the biggest impact. It seems the parameters on the benchmark worker don't seem to affect throughput."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>replicas</th>\n",
       "      <th>server_workers</th>\n",
       "      <th>server_threads</th>\n",
       "      <th>apiType</th>\n",
       "      <th>cpus</th>\n",
       "      <th>max_wokers</th>\n",
       "      <th>use_engine</th>\n",
       "      <th>mean</th>\n",
       "      <th>50th</th>\n",
       "      <th>90th</th>\n",
       "      <th>95th</th>\n",
       "      <th>99th</th>\n",
       "      <th>rate</th>\n",
       "      <th>errors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>201.167546</td>\n",
       "      <td>8.844305</td>\n",
       "      <td>629.250888</td>\n",
       "      <td>690.807158</td>\n",
       "      <td>809.635194</td>\n",
       "      <td>992.298652</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>208.429576</td>\n",
       "      <td>11.377699</td>\n",
       "      <td>655.466848</td>\n",
       "      <td>685.265506</td>\n",
       "      <td>758.664504</td>\n",
       "      <td>957.846772</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>211.228526</td>\n",
       "      <td>13.592301</td>\n",
       "      <td>641.484819</td>\n",
       "      <td>675.713639</td>\n",
       "      <td>795.682869</td>\n",
       "      <td>945.090980</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>214.358834</td>\n",
       "      <td>13.573121</td>\n",
       "      <td>670.449768</td>\n",
       "      <td>690.048496</td>\n",
       "      <td>722.537613</td>\n",
       "      <td>930.694079</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>216.646320</td>\n",
       "      <td>9.336961</td>\n",
       "      <td>684.733598</td>\n",
       "      <td>704.485018</td>\n",
       "      <td>733.636276</td>\n",
       "      <td>921.350903</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>217.722397</td>\n",
       "      <td>16.593757</td>\n",
       "      <td>657.144743</td>\n",
       "      <td>695.158232</td>\n",
       "      <td>745.726065</td>\n",
       "      <td>916.803160</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>218.817952</td>\n",
       "      <td>10.808913</td>\n",
       "      <td>689.809571</td>\n",
       "      <td>757.737985</td>\n",
       "      <td>867.650689</td>\n",
       "      <td>912.589694</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>221.031876</td>\n",
       "      <td>9.197338</td>\n",
       "      <td>690.217169</td>\n",
       "      <td>711.800471</td>\n",
       "      <td>742.657817</td>\n",
       "      <td>903.072311</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>221.263249</td>\n",
       "      <td>16.583482</td>\n",
       "      <td>688.637696</td>\n",
       "      <td>711.870214</td>\n",
       "      <td>781.197685</td>\n",
       "      <td>902.315850</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>221.566956</td>\n",
       "      <td>11.037262</td>\n",
       "      <td>685.417461</td>\n",
       "      <td>713.923684</td>\n",
       "      <td>771.814053</td>\n",
       "      <td>901.132352</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>225.719114</td>\n",
       "      <td>15.998348</td>\n",
       "      <td>704.701196</td>\n",
       "      <td>741.890962</td>\n",
       "      <td>852.664830</td>\n",
       "      <td>884.187996</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>229.653366</td>\n",
       "      <td>9.844413</td>\n",
       "      <td>725.066803</td>\n",
       "      <td>775.186525</td>\n",
       "      <td>857.762245</td>\n",
       "      <td>869.461119</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>231.016536</td>\n",
       "      <td>15.829218</td>\n",
       "      <td>737.382688</td>\n",
       "      <td>788.027859</td>\n",
       "      <td>885.482116</td>\n",
       "      <td>863.960992</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>231.986927</td>\n",
       "      <td>11.193407</td>\n",
       "      <td>702.083677</td>\n",
       "      <td>769.889421</td>\n",
       "      <td>901.360146</td>\n",
       "      <td>860.495277</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>239.150794</td>\n",
       "      <td>14.147647</td>\n",
       "      <td>722.982655</td>\n",
       "      <td>789.211063</td>\n",
       "      <td>929.436195</td>\n",
       "      <td>834.381347</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>240.088790</td>\n",
       "      <td>121.078205</td>\n",
       "      <td>707.862815</td>\n",
       "      <td>771.405571</td>\n",
       "      <td>965.932529</td>\n",
       "      <td>831.402721</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>413.608259</td>\n",
       "      <td>409.729690</td>\n",
       "      <td>442.576049</td>\n",
       "      <td>460.804621</td>\n",
       "      <td>502.762769</td>\n",
       "      <td>482.699096</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>429.042835</td>\n",
       "      <td>412.423403</td>\n",
       "      <td>500.170846</td>\n",
       "      <td>522.423418</td>\n",
       "      <td>586.685379</td>\n",
       "      <td>465.431891</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>432.609142</td>\n",
       "      <td>426.606234</td>\n",
       "      <td>488.443435</td>\n",
       "      <td>512.393140</td>\n",
       "      <td>556.238288</td>\n",
       "      <td>461.578501</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>463.422714</td>\n",
       "      <td>450.181537</td>\n",
       "      <td>551.644801</td>\n",
       "      <td>602.270942</td>\n",
       "      <td>670.647806</td>\n",
       "      <td>430.891782</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>475.510231</td>\n",
       "      <td>456.056479</td>\n",
       "      <td>583.716159</td>\n",
       "      <td>650.365364</td>\n",
       "      <td>746.791628</td>\n",
       "      <td>419.975983</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>481.143061</td>\n",
       "      <td>450.734477</td>\n",
       "      <td>602.026223</td>\n",
       "      <td>689.302618</td>\n",
       "      <td>863.072782</td>\n",
       "      <td>414.795159</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>488.185779</td>\n",
       "      <td>436.842244</td>\n",
       "      <td>628.922397</td>\n",
       "      <td>735.512654</td>\n",
       "      <td>1068.474298</td>\n",
       "      <td>408.992844</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>489.269344</td>\n",
       "      <td>455.617128</td>\n",
       "      <td>612.294382</td>\n",
       "      <td>672.510108</td>\n",
       "      <td>832.322767</td>\n",
       "      <td>407.879172</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>514.472545</td>\n",
       "      <td>488.358257</td>\n",
       "      <td>591.629431</td>\n",
       "      <td>631.392813</td>\n",
       "      <td>1517.062374</td>\n",
       "      <td>387.882855</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>529.767457</td>\n",
       "      <td>514.151876</td>\n",
       "      <td>591.278115</td>\n",
       "      <td>621.463805</td>\n",
       "      <td>749.348556</td>\n",
       "      <td>376.649458</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>547.618426</td>\n",
       "      <td>526.472215</td>\n",
       "      <td>661.947413</td>\n",
       "      <td>720.039676</td>\n",
       "      <td>863.596098</td>\n",
       "      <td>364.363839</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>571.452398</td>\n",
       "      <td>556.699256</td>\n",
       "      <td>693.093315</td>\n",
       "      <td>751.197598</td>\n",
       "      <td>1024.233714</td>\n",
       "      <td>348.889260</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>587.900216</td>\n",
       "      <td>556.869872</td>\n",
       "      <td>723.744376</td>\n",
       "      <td>774.244702</td>\n",
       "      <td>939.994423</td>\n",
       "      <td>339.396160</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>rest</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>593.880113</td>\n",
       "      <td>602.945695</td>\n",
       "      <td>737.993290</td>\n",
       "      <td>770.777543</td>\n",
       "      <td>1003.510371</td>\n",
       "      <td>336.075411</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>633.043624</td>\n",
       "      <td>617.853285</td>\n",
       "      <td>741.229073</td>\n",
       "      <td>776.560578</td>\n",
       "      <td>1846.623159</td>\n",
       "      <td>314.908167</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>rest</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>641.530606</td>\n",
       "      <td>653.922529</td>\n",
       "      <td>802.558303</td>\n",
       "      <td>847.414484</td>\n",
       "      <td>1570.484029</td>\n",
       "      <td>310.839312</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   replicas server_workers server_threads apiType cpus  max_wokers use_engine  \\\n",
       "41        3              1            200    rest    1         200      false   \n",
       "48        3              4             50    rest    1         200       true   \n",
       "51        3              4             50    rest    4         200      false   \n",
       "59        3              4            200    rest    4         200      false   \n",
       "57        3              4            200    rest    1         200      false   \n",
       "40        3              1            200    rest    1         200       true   \n",
       "32        3              1             50    rest    1         200       true   \n",
       "56        3              4            200    rest    1         200       true   \n",
       "50        3              4             50    rest    4         200       true   \n",
       "58        3              4            200    rest    4         200       true   \n",
       "35        3              1             50    rest    4         200      false   \n",
       "33        3              1             50    rest    1         200      false   \n",
       "42        3              1            200    rest    4         200       true   \n",
       "49        3              4             50    rest    1         200      false   \n",
       "43        3              1            200    rest    4         200      false   \n",
       "34        3              1             50    rest    4         200       true   \n",
       "26        1              4            200    rest    4         200       true   \n",
       "17        1              4             50    rest    1         200      false   \n",
       "27        1              4            200    rest    4         200      false   \n",
       "25        1              4            200    rest    1         200      false   \n",
       "16        1              4             50    rest    1         200       true   \n",
       "19        1              4             50    rest    4         200      false   \n",
       "18        1              4             50    rest    4         200       true   \n",
       "0         1              1             50    rest    1         200       true   \n",
       "24        1              4            200    rest    1         200       true   \n",
       "1         1              1             50    rest    1         200      false   \n",
       "2         1              1             50    rest    4         200       true   \n",
       "10        1              1            200    rest    4         200       true   \n",
       "11        1              1            200    rest    4         200      false   \n",
       "3         1              1             50    rest    4         200      false   \n",
       "8         1              1            200    rest    1         200       true   \n",
       "9         1              1            200    rest    1         200      false   \n",
       "\n",
       "          mean        50th        90th        95th         99th        rate  \\\n",
       "41  201.167546    8.844305  629.250888  690.807158   809.635194  992.298652   \n",
       "48  208.429576   11.377699  655.466848  685.265506   758.664504  957.846772   \n",
       "51  211.228526   13.592301  641.484819  675.713639   795.682869  945.090980   \n",
       "59  214.358834   13.573121  670.449768  690.048496   722.537613  930.694079   \n",
       "57  216.646320    9.336961  684.733598  704.485018   733.636276  921.350903   \n",
       "40  217.722397   16.593757  657.144743  695.158232   745.726065  916.803160   \n",
       "32  218.817952   10.808913  689.809571  757.737985   867.650689  912.589694   \n",
       "56  221.031876    9.197338  690.217169  711.800471   742.657817  903.072311   \n",
       "50  221.263249   16.583482  688.637696  711.870214   781.197685  902.315850   \n",
       "58  221.566956   11.037262  685.417461  713.923684   771.814053  901.132352   \n",
       "35  225.719114   15.998348  704.701196  741.890962   852.664830  884.187996   \n",
       "33  229.653366    9.844413  725.066803  775.186525   857.762245  869.461119   \n",
       "42  231.016536   15.829218  737.382688  788.027859   885.482116  863.960992   \n",
       "49  231.986927   11.193407  702.083677  769.889421   901.360146  860.495277   \n",
       "43  239.150794   14.147647  722.982655  789.211063   929.436195  834.381347   \n",
       "34  240.088790  121.078205  707.862815  771.405571   965.932529  831.402721   \n",
       "26  413.608259  409.729690  442.576049  460.804621   502.762769  482.699096   \n",
       "17  429.042835  412.423403  500.170846  522.423418   586.685379  465.431891   \n",
       "27  432.609142  426.606234  488.443435  512.393140   556.238288  461.578501   \n",
       "25  463.422714  450.181537  551.644801  602.270942   670.647806  430.891782   \n",
       "16  475.510231  456.056479  583.716159  650.365364   746.791628  419.975983   \n",
       "19  481.143061  450.734477  602.026223  689.302618   863.072782  414.795159   \n",
       "18  488.185779  436.842244  628.922397  735.512654  1068.474298  408.992844   \n",
       "0   489.269344  455.617128  612.294382  672.510108   832.322767  407.879172   \n",
       "24  514.472545  488.358257  591.629431  631.392813  1517.062374  387.882855   \n",
       "1   529.767457  514.151876  591.278115  621.463805   749.348556  376.649458   \n",
       "2   547.618426  526.472215  661.947413  720.039676   863.596098  364.363839   \n",
       "10  571.452398  556.699256  693.093315  751.197598  1024.233714  348.889260   \n",
       "11  587.900216  556.869872  723.744376  774.244702   939.994423  339.396160   \n",
       "3   593.880113  602.945695  737.993290  770.777543  1003.510371  336.075411   \n",
       "8   633.043624  617.853285  741.229073  776.560578  1846.623159  314.908167   \n",
       "9   641.530606  653.922529  802.558303  847.414484  1570.484029  310.839312   \n",
       "\n",
       "   errors  \n",
       "41     []  \n",
       "48     []  \n",
       "51     []  \n",
       "59     []  \n",
       "57     []  \n",
       "40     []  \n",
       "32     []  \n",
       "56     []  \n",
       "50     []  \n",
       "58     []  \n",
       "35     []  \n",
       "33     []  \n",
       "42     []  \n",
       "49     []  \n",
       "43     []  \n",
       "34     []  \n",
       "26     []  \n",
       "17     []  \n",
       "27     []  \n",
       "25     []  \n",
       "16     []  \n",
       "19     []  \n",
       "18     []  \n",
       "0      []  \n",
       "24     []  \n",
       "1      []  \n",
       "2      []  \n",
       "10     []  \n",
       "11     []  \n",
       "3      []  \n",
       "8      []  \n",
       "9      []  "
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df[\"apiType\"]==\"rest\"].sort_values(\"rate\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deep dive on GRPC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>replicas</th>\n",
       "      <th>server_workers</th>\n",
       "      <th>server_threads</th>\n",
       "      <th>apiType</th>\n",
       "      <th>cpus</th>\n",
       "      <th>max_wokers</th>\n",
       "      <th>use_engine</th>\n",
       "      <th>mean</th>\n",
       "      <th>50th</th>\n",
       "      <th>90th</th>\n",
       "      <th>95th</th>\n",
       "      <th>99th</th>\n",
       "      <th>rate</th>\n",
       "      <th>errors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>31.389861</td>\n",
       "      <td>23.769589</td>\n",
       "      <td>71.583795</td>\n",
       "      <td>78.881398</td>\n",
       "      <td>91.312797</td>\n",
       "      <td>1586.593680</td>\n",
       "      <td>{'OK': 190361, 'Unavailable': 48}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>31.398451</td>\n",
       "      <td>26.313000</td>\n",
       "      <td>64.841515</td>\n",
       "      <td>73.035800</td>\n",
       "      <td>88.744198</td>\n",
       "      <td>1586.555365</td>\n",
       "      <td>{'OK': 190333, 'Unavailable': 71}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>32.191240</td>\n",
       "      <td>30.448302</td>\n",
       "      <td>60.616301</td>\n",
       "      <td>68.724406</td>\n",
       "      <td>91.484308</td>\n",
       "      <td>1547.003054</td>\n",
       "      <td>{'OK': 185606, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>32.727674</td>\n",
       "      <td>28.483400</td>\n",
       "      <td>63.750796</td>\n",
       "      <td>72.597310</td>\n",
       "      <td>90.693812</td>\n",
       "      <td>1521.590875</td>\n",
       "      <td>{'OK': 182555, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>33.629848</td>\n",
       "      <td>29.610701</td>\n",
       "      <td>67.065895</td>\n",
       "      <td>77.773100</td>\n",
       "      <td>97.296599</td>\n",
       "      <td>1479.320474</td>\n",
       "      <td>{'OK': 177471, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>33.861023</td>\n",
       "      <td>30.207400</td>\n",
       "      <td>70.272698</td>\n",
       "      <td>83.485103</td>\n",
       "      <td>105.639301</td>\n",
       "      <td>1469.503585</td>\n",
       "      <td>{'OK': 176302, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>34.746801</td>\n",
       "      <td>31.896585</td>\n",
       "      <td>72.732796</td>\n",
       "      <td>84.032763</td>\n",
       "      <td>99.433090</td>\n",
       "      <td>1432.045405</td>\n",
       "      <td>{'OK': 171799, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>34.786883</td>\n",
       "      <td>32.141197</td>\n",
       "      <td>72.554313</td>\n",
       "      <td>82.649702</td>\n",
       "      <td>95.049705</td>\n",
       "      <td>1430.209225</td>\n",
       "      <td>{'OK': 171578, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>35.149376</td>\n",
       "      <td>35.153187</td>\n",
       "      <td>62.842800</td>\n",
       "      <td>72.791800</td>\n",
       "      <td>94.240299</td>\n",
       "      <td>1416.745392</td>\n",
       "      <td>{'OK': 169973, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>35.167657</td>\n",
       "      <td>31.859300</td>\n",
       "      <td>65.644895</td>\n",
       "      <td>76.240799</td>\n",
       "      <td>98.925899</td>\n",
       "      <td>1415.967279</td>\n",
       "      <td>{'OK': 169880, 'Unavailable': 48}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>35.286173</td>\n",
       "      <td>24.988500</td>\n",
       "      <td>83.079301</td>\n",
       "      <td>94.264796</td>\n",
       "      <td>111.448895</td>\n",
       "      <td>1410.595798</td>\n",
       "      <td>{'OK': 169202, 'Unavailable': 71}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>35.543940</td>\n",
       "      <td>30.528900</td>\n",
       "      <td>69.449895</td>\n",
       "      <td>82.465882</td>\n",
       "      <td>100.381195</td>\n",
       "      <td>1400.945365</td>\n",
       "      <td>{'OK': 168074, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>35.706181</td>\n",
       "      <td>30.175300</td>\n",
       "      <td>76.121701</td>\n",
       "      <td>85.842385</td>\n",
       "      <td>99.072701</td>\n",
       "      <td>1393.469861</td>\n",
       "      <td>{'OK': 167180, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>36.026804</td>\n",
       "      <td>33.541192</td>\n",
       "      <td>69.942798</td>\n",
       "      <td>81.321704</td>\n",
       "      <td>108.528901</td>\n",
       "      <td>1381.482907</td>\n",
       "      <td>{'OK': 165711, 'Unavailable': 69}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>36.325718</td>\n",
       "      <td>35.598498</td>\n",
       "      <td>73.211997</td>\n",
       "      <td>82.948302</td>\n",
       "      <td>102.248397</td>\n",
       "      <td>1369.739820</td>\n",
       "      <td>{'OK': 164333, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>37.326561</td>\n",
       "      <td>35.609388</td>\n",
       "      <td>70.522598</td>\n",
       "      <td>79.731401</td>\n",
       "      <td>101.297400</td>\n",
       "      <td>1334.058278</td>\n",
       "      <td>{'OK': 160053, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>63.240129</td>\n",
       "      <td>61.519201</td>\n",
       "      <td>72.905000</td>\n",
       "      <td>77.140700</td>\n",
       "      <td>89.520499</td>\n",
       "      <td>789.347786</td>\n",
       "      <td>{'OK': 94678, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>63.537119</td>\n",
       "      <td>61.855200</td>\n",
       "      <td>74.299100</td>\n",
       "      <td>79.876601</td>\n",
       "      <td>97.179900</td>\n",
       "      <td>785.631011</td>\n",
       "      <td>{'OK': 94233, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>65.711577</td>\n",
       "      <td>64.220500</td>\n",
       "      <td>78.085300</td>\n",
       "      <td>83.563600</td>\n",
       "      <td>94.907700</td>\n",
       "      <td>759.690398</td>\n",
       "      <td>{'OK': 91119, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>66.898143</td>\n",
       "      <td>63.420800</td>\n",
       "      <td>83.837100</td>\n",
       "      <td>92.332400</td>\n",
       "      <td>108.138499</td>\n",
       "      <td>746.209307</td>\n",
       "      <td>{'OK': 89501, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>67.211609</td>\n",
       "      <td>65.504200</td>\n",
       "      <td>79.989899</td>\n",
       "      <td>86.808200</td>\n",
       "      <td>106.460500</td>\n",
       "      <td>742.433252</td>\n",
       "      <td>{'OK': 89044, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>67.770632</td>\n",
       "      <td>62.168504</td>\n",
       "      <td>88.674303</td>\n",
       "      <td>102.537000</td>\n",
       "      <td>120.848185</td>\n",
       "      <td>736.385539</td>\n",
       "      <td>{'OK': 88318, 'Unavailable': 49}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>70.577834</td>\n",
       "      <td>68.972899</td>\n",
       "      <td>84.869200</td>\n",
       "      <td>89.875600</td>\n",
       "      <td>102.761897</td>\n",
       "      <td>707.046156</td>\n",
       "      <td>{'OK': 84796, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>70.818411</td>\n",
       "      <td>67.591600</td>\n",
       "      <td>87.914104</td>\n",
       "      <td>97.004000</td>\n",
       "      <td>115.388900</td>\n",
       "      <td>704.647865</td>\n",
       "      <td>{'OK': 84514, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>71.571627</td>\n",
       "      <td>69.348700</td>\n",
       "      <td>91.609598</td>\n",
       "      <td>98.471998</td>\n",
       "      <td>111.237797</td>\n",
       "      <td>697.252435</td>\n",
       "      <td>{'OK': 83622, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>73.853780</td>\n",
       "      <td>70.604701</td>\n",
       "      <td>91.031400</td>\n",
       "      <td>98.064600</td>\n",
       "      <td>116.658902</td>\n",
       "      <td>675.704389</td>\n",
       "      <td>{'OK': 81035, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>89.662500</td>\n",
       "      <td>87.678702</td>\n",
       "      <td>107.762199</td>\n",
       "      <td>118.226099</td>\n",
       "      <td>146.838610</td>\n",
       "      <td>556.478774</td>\n",
       "      <td>{'OK': 66728, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>4</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>90.655025</td>\n",
       "      <td>91.964500</td>\n",
       "      <td>108.453597</td>\n",
       "      <td>116.581800</td>\n",
       "      <td>148.048199</td>\n",
       "      <td>550.406903</td>\n",
       "      <td>{'OK': 66003, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>92.930400</td>\n",
       "      <td>93.020601</td>\n",
       "      <td>113.056104</td>\n",
       "      <td>122.476104</td>\n",
       "      <td>150.119004</td>\n",
       "      <td>537.076992</td>\n",
       "      <td>{'OK': 64405, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>94.695951</td>\n",
       "      <td>94.988002</td>\n",
       "      <td>111.319799</td>\n",
       "      <td>118.210000</td>\n",
       "      <td>134.270997</td>\n",
       "      <td>527.054914</td>\n",
       "      <td>{'OK': 63202, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>true</td>\n",
       "      <td>95.322943</td>\n",
       "      <td>97.896699</td>\n",
       "      <td>117.221999</td>\n",
       "      <td>125.852400</td>\n",
       "      <td>141.615501</td>\n",
       "      <td>523.628160</td>\n",
       "      <td>{'OK': 62790, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>grpc</td>\n",
       "      <td>1</td>\n",
       "      <td>200</td>\n",
       "      <td>false</td>\n",
       "      <td>96.016296</td>\n",
       "      <td>97.410200</td>\n",
       "      <td>113.779899</td>\n",
       "      <td>120.184499</td>\n",
       "      <td>136.929395</td>\n",
       "      <td>519.810588</td>\n",
       "      <td>{'OK': 62332, 'Unavailable': 50}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   replicas server_workers server_threads apiType cpus  max_wokers use_engine  \\\n",
       "60        3              4            200    grpc    1         200       true   \n",
       "52        3              4             50    grpc    1         200       true   \n",
       "45        3              1            200    grpc    1         200      false   \n",
       "61        3              4            200    grpc    1         200      false   \n",
       "55        3              4             50    grpc    4         200      false   \n",
       "47        3              1            200    grpc    4         200      false   \n",
       "62        3              4            200    grpc    4         200       true   \n",
       "54        3              4             50    grpc    4         200       true   \n",
       "37        3              1             50    grpc    1         200      false   \n",
       "36        3              1             50    grpc    1         200       true   \n",
       "46        3              1            200    grpc    4         200       true   \n",
       "53        3              4             50    grpc    1         200      false   \n",
       "63        3              4            200    grpc    4         200      false   \n",
       "39        3              1             50    grpc    4         200      false   \n",
       "38        3              1             50    grpc    4         200       true   \n",
       "44        3              1            200    grpc    1         200       true   \n",
       "29        1              4            200    grpc    1         200      false   \n",
       "28        1              4            200    grpc    1         200       true   \n",
       "20        1              4             50    grpc    1         200       true   \n",
       "21        1              4             50    grpc    1         200      false   \n",
       "30        1              4            200    grpc    4         200       true   \n",
       "7         1              1             50    grpc    4         200      false   \n",
       "31        1              4            200    grpc    4         200      false   \n",
       "22        1              4             50    grpc    4         200       true   \n",
       "15        1              1            200    grpc    4         200      false   \n",
       "23        1              4             50    grpc    4         200      false   \n",
       "14        1              1            200    grpc    4         200       true   \n",
       "6         1              1             50    grpc    4         200       true   \n",
       "5         1              1             50    grpc    1         200      false   \n",
       "12        1              1            200    grpc    1         200       true   \n",
       "4         1              1             50    grpc    1         200       true   \n",
       "13        1              1            200    grpc    1         200      false   \n",
       "\n",
       "         mean       50th        90th        95th        99th         rate  \\\n",
       "60  31.389861  23.769589   71.583795   78.881398   91.312797  1586.593680   \n",
       "52  31.398451  26.313000   64.841515   73.035800   88.744198  1586.555365   \n",
       "45  32.191240  30.448302   60.616301   68.724406   91.484308  1547.003054   \n",
       "61  32.727674  28.483400   63.750796   72.597310   90.693812  1521.590875   \n",
       "55  33.629848  29.610701   67.065895   77.773100   97.296599  1479.320474   \n",
       "47  33.861023  30.207400   70.272698   83.485103  105.639301  1469.503585   \n",
       "62  34.746801  31.896585   72.732796   84.032763   99.433090  1432.045405   \n",
       "54  34.786883  32.141197   72.554313   82.649702   95.049705  1430.209225   \n",
       "37  35.149376  35.153187   62.842800   72.791800   94.240299  1416.745392   \n",
       "36  35.167657  31.859300   65.644895   76.240799   98.925899  1415.967279   \n",
       "46  35.286173  24.988500   83.079301   94.264796  111.448895  1410.595798   \n",
       "53  35.543940  30.528900   69.449895   82.465882  100.381195  1400.945365   \n",
       "63  35.706181  30.175300   76.121701   85.842385   99.072701  1393.469861   \n",
       "39  36.026804  33.541192   69.942798   81.321704  108.528901  1381.482907   \n",
       "38  36.325718  35.598498   73.211997   82.948302  102.248397  1369.739820   \n",
       "44  37.326561  35.609388   70.522598   79.731401  101.297400  1334.058278   \n",
       "29  63.240129  61.519201   72.905000   77.140700   89.520499   789.347786   \n",
       "28  63.537119  61.855200   74.299100   79.876601   97.179900   785.631011   \n",
       "20  65.711577  64.220500   78.085300   83.563600   94.907700   759.690398   \n",
       "21  66.898143  63.420800   83.837100   92.332400  108.138499   746.209307   \n",
       "30  67.211609  65.504200   79.989899   86.808200  106.460500   742.433252   \n",
       "7   67.770632  62.168504   88.674303  102.537000  120.848185   736.385539   \n",
       "31  70.577834  68.972899   84.869200   89.875600  102.761897   707.046156   \n",
       "22  70.818411  67.591600   87.914104   97.004000  115.388900   704.647865   \n",
       "15  71.571627  69.348700   91.609598   98.471998  111.237797   697.252435   \n",
       "23  73.853780  70.604701   91.031400   98.064600  116.658902   675.704389   \n",
       "14  89.662500  87.678702  107.762199  118.226099  146.838610   556.478774   \n",
       "6   90.655025  91.964500  108.453597  116.581800  148.048199   550.406903   \n",
       "5   92.930400  93.020601  113.056104  122.476104  150.119004   537.076992   \n",
       "12  94.695951  94.988002  111.319799  118.210000  134.270997   527.054914   \n",
       "4   95.322943  97.896699  117.221999  125.852400  141.615501   523.628160   \n",
       "13  96.016296  97.410200  113.779899  120.184499  136.929395   519.810588   \n",
       "\n",
       "                               errors  \n",
       "60  {'OK': 190361, 'Unavailable': 48}  \n",
       "52  {'OK': 190333, 'Unavailable': 71}  \n",
       "45  {'OK': 185606, 'Unavailable': 49}  \n",
       "61  {'OK': 182555, 'Unavailable': 49}  \n",
       "55  {'OK': 177471, 'Unavailable': 50}  \n",
       "47  {'OK': 176302, 'Unavailable': 50}  \n",
       "62  {'OK': 171799, 'Unavailable': 50}  \n",
       "54  {'OK': 171578, 'Unavailable': 49}  \n",
       "37  {'OK': 169973, 'Unavailable': 50}  \n",
       "36  {'OK': 169880, 'Unavailable': 48}  \n",
       "46  {'OK': 169202, 'Unavailable': 71}  \n",
       "53  {'OK': 168074, 'Unavailable': 50}  \n",
       "63  {'OK': 167180, 'Unavailable': 49}  \n",
       "39  {'OK': 165711, 'Unavailable': 69}  \n",
       "38  {'OK': 164333, 'Unavailable': 49}  \n",
       "44  {'OK': 160053, 'Unavailable': 50}  \n",
       "29   {'OK': 94678, 'Unavailable': 50}  \n",
       "28   {'OK': 94233, 'Unavailable': 50}  \n",
       "20   {'OK': 91119, 'Unavailable': 50}  \n",
       "21   {'OK': 89501, 'Unavailable': 50}  \n",
       "30   {'OK': 89044, 'Unavailable': 50}  \n",
       "7    {'OK': 88318, 'Unavailable': 49}  \n",
       "31   {'OK': 84796, 'Unavailable': 50}  \n",
       "22   {'OK': 84514, 'Unavailable': 50}  \n",
       "15   {'OK': 83622, 'Unavailable': 50}  \n",
       "23   {'OK': 81035, 'Unavailable': 50}  \n",
       "14   {'OK': 66728, 'Unavailable': 50}  \n",
       "6    {'OK': 66003, 'Unavailable': 50}  \n",
       "5    {'OK': 64405, 'Unavailable': 50}  \n",
       "12   {'OK': 63202, 'Unavailable': 50}  \n",
       "4    {'OK': 62790, 'Unavailable': 50}  \n",
       "13   {'OK': 62332, 'Unavailable': 50}  "
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df[\"apiType\"]==\"grpc\"].sort_values(\"rate\", ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}