notebooks/max_grpc_msg_size.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Increasing the Maximum Message Size for gRPC\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequistes\n",
    "\n",
    "You will need\n",
    "\n",
    " - [Git clone of Seldon Core](https://github.com/SeldonIO/seldon-core)\n",
    " - [Helm](https://github.com/kubernetes/helm)\n",
    " - [Minikube](https://github.com/kubernetes/minikube) version v0.24.0 or greater\n",
    " - [seldon-core Python package](https://pypi.org/project/seldon-core/) (```pip install seldon-core```)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Cluster\n",
    "\n",
    "Start minikube and ensure custom resource validation is activated and there is 5G of memory. \n",
    "\n",
    "An example start command using the kvm2 driver would look like:\n",
    "```\n",
    "minikube start --vm-driver kvm2 --memory 4096 \n",
    "```\n",
    "\n",
    "## Running this notebook\n",
    "\n",
    "You will need to start Jupyter with settings to allow for large payloads, for example:\n",
    "\n",
    "```\n",
    "jupyter notebook --NotebookApp.iopub_data_rate_limit=1000000000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"minikube\" modified.\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io \"kube-system-cluster-admin\" already exists\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/tiller created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/tiller created\n",
      "$HELM_HOME has been configured at /home/clive/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"tiller-deploy\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"tiller-deploy\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Tue Apr 16 07:58:23 2019\n",
      "NAMESPACE: seldon-system\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/Deployment\n",
      "NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  1        0        0           0          1s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-spartakus-config  3     1s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME                                 AGE\n",
      "seldon-operator-manager-rolebinding  1s\n",
      "\n",
      "==> v1/ClusterRole\n",
      "NAME                          AGE\n",
      "seldon-operator-manager-role  1s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                                        TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE\n",
      "seldon-operator-controller-manager-service  ClusterIP  10.111.148.241  <none>       443/TCP  1s\n",
      "\n",
      "==> v1/StatefulSet\n",
      "NAME                                DESIRED  CURRENT  AGE\n",
      "seldon-operator-controller-manager  1        1        1s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                        SECRETS  AGE\n",
      "seldon-spartakus-volunteer  1        1s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                  READY  STATUS             RESTARTS  AGE\n",
      "seldon-operator-controller-manager-0  0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/Secret\n",
      "NAME                                   TYPE    DATA  AGE\n",
      "seldon-operator-webhook-server-secret  Opaque  0     1s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true --namespace seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check all services are running before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "partitioned roll out complete: 1 new pods have been updated...\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status statefulset.apps/seldon-operator-controller-manager -n seldon-system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Ingress\n",
    "There are gRPC issues with the latest Ambassador, so we rewcommend 0.40.2 until these are fixed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   ambassador\n",
      "LAST DEPLOYED: Tue Apr 16 07:59:02 2019\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/ClusterRole\n",
      "NAME        AGE\n",
      "ambassador  1s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME        AGE\n",
      "ambassador  1s\n",
      "\n",
      "==> v1/Service\n",
      "NAME               TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)                     AGE\n",
      "ambassador-admins  ClusterIP     10.100.193.126  <none>       8877/TCP                    1s\n",
      "ambassador         LoadBalancer  10.100.215.36   <pending>    80:31251/TCP,443:31599/TCP  0s\n",
      "\n",
      "==> v1/Deployment\n",
      "NAME        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "ambassador  3        3        3           0          0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                         READY  STATUS             RESTARTS  AGE\n",
      "ambassador-5b89d44544-dh8q6  0/1    ContainerCreating  0         0s\n",
      "ambassador-5b89d44544-p5c8c  0/1    ContainerCreating  0         0s\n",
      "ambassador-5b89d44544-qjtjh  0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME        SECRETS  AGE\n",
      "ambassador  1        1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Congratuations! You've successfully installed Ambassador.\n",
      "\n",
      "For help, visit our Slack at https://d6e.co/slack or view the documentation online at https://www.getambassador.io.\n",
      "\n",
      "To get the IP address of Ambassador, run the following commands:\n",
      "NOTE: It may take a few minutes for the LoadBalancer IP to be available.\n",
      "     You can watch the status of by running 'kubectl get svc -w  --namespace seldon ambassador'\n",
      "\n",
      "  On GKE/Azure:\n",
      "  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].ip}')\n",
      "\n",
      "  On AWS:\n",
      "  export SERVICE_IP=$(kubectl get svc --namespace seldon ambassador -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')\n",
      "\n",
      "  echo http://$SERVICE_IP:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install stable/ambassador --name ambassador --set crds.keep=false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"ambassador\" rollout to finish: 0 of 3 updated replicas are available...\n",
      "Waiting for deployment \"ambassador\" rollout to finish: 1 of 3 updated replicas are available...\n",
      "Waiting for deployment \"ambassador\" rollout to finish: 2 of 3 updated replicas are available...\n",
      "deployment \"ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deployment.apps/ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Port forward to Ambassador\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"model-long-timeout\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"deployment_version\"\u001b[39;49;00m: \u001b[33m\"v1\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-read-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-connection-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m,\t    \r\n",
      "\t    \u001b[34;01m\"seldon.io/grpc-read-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"long-to\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_long_timeouts.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Seldon Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy the runtime graph to kubernetes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/model-long-timeout created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_long_timeouts.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"long-to-test-7cd068f\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"long-to-test-7cd068f\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/long-to-test-7cd068f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get predictions - no grpc max message size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_core.seldon_client import SeldonClient\n",
    "sc = SeldonClient(deployment_name=\"model-long-timeout\",namespace=\"seldon\", \n",
    "                  grpc_max_send_message_length=50 * 1024 * 1024, grpc_max_receive_message_length=50 * 1024 * 1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a small request which should suceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success:True message:\n",
      "Request:\n",
      "data {\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 1\n",
      "    values: 0.7542117938738865\n",
      "  }\n",
      "}\n",
      "\n",
      "Response:\n",
      "meta {\n",
      "  puid: \"h568dh0e9mfbhfa4isnlotkkm1\"\n",
      "  requestPath {\n",
      "    key: \"classifier\"\n",
      "    value: \"seldonio/mock_classifier:1.0\"\n",
      "  }\n",
      "}\n",
      "data {\n",
      "  names: \"proba\"\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 1\n",
      "    values: 0.10317308463100552\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"ambassador\",transport=\"grpc\")\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a large request which will be above the default gRPC message size and will fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False <_Rendezvous of RPC that terminated with:\n",
      "\tstatus = StatusCode.UNAVAILABLE\n",
      "\tdetails = \"upstream connect error or disconnect/reset before headers\"\n",
      "\tdebug_error_string = \"{\"created\":\"@1555398231.521439766\",\"description\":\"Error received from peer\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1095,\"grpc_message\":\"upstream connect error or disconnect/reset before headers\",\"grpc_status\":14}\"\n",
      ">\n"
     ]
    }
   ],
   "source": [
    "r = sc.predict(gateway=\"ambassador\",transport=\"grpc\",shape=(1000000,1))\n",
    "print(r.success,r.msg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"model-long-timeout\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_long_timeouts.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Allowing larger gRPC messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we change our SeldonDeployment to include a annotation for max grpx message size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-model\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t    \u001b[34;01m\"seldon.io/grpc-max-message-size\"\u001b[39;49;00m:\u001b[33m\"10000000\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-read-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-connection-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m,\t    \r\n",
      "\t    \u001b[34;01m\"seldon.io/grpc-read-timeout\"\u001b[39;49;00m:\u001b[33m\"100000\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier_grpc:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"grpc-size\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_grpc_size.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io/seldon-model created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f resources/model_grpc_size.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"test-deployment-grpc-size-fd60a01\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"test-deployment-grpc-size-fd60a01\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/test-deployment-grpc-size-fd60a01 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a request via ambassador. This should succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True \n"
     ]
    }
   ],
   "source": [
    "sc = SeldonClient(deployment_name=\"seldon-model\",namespace=\"seldon\",\n",
    "                  grpc_max_send_message_length=50 * 1024 * 1024, grpc_max_receive_message_length=50 * 1024 * 1024)\n",
    "r = sc.predict(gateway=\"ambassador\",transport=\"grpc\",shape=(1000000,1))\n",
    "print(r.success,r.msg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment.machinelearning.seldon.io \"seldon-model\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_grpc_size.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}